https://wasmlabs.dev/articles/python-wasm32-wasi/ Wasm Labs @ VMware OCTO * Articles * Projects * About * @vmwwasm * Adding Python support to Wasm Language Runtimes By [alexandrov] Asen Alexandrov At 2023 / 01 10 mins reading We recently added Python support to Wasm Language Runtimes. This article provides an overview of how Python works in WebAssembly environments and provides a step by step guide on how to use it. At VMware OCTO WasmLabs we want to grow the WebAssembly ecosystem by helping developers adopt this new and exciting technology. Our Wasm Language Runtimes project aims to provide up-to-date, ready-to-run WebAssembly builds for the most popular language runtimes. We are happy to announce that we have a first build of Python for the wasm32-wasi target! It is based on the WASI support that is already available in CPython (the mainstream, C-based implementation of Python), augmented with additional libraries and usage examples to make it as easy to use as possible. Python joins PHP and Ruby in the list of supported languages. About the build and artifacts To build python.wasm we rely on the WASI build support that is already available in CPython. We are reusing zlib and libuuid from the singlestore-labs/python-wasi repository but are building libsqlite in-house to also enable support for the sqlite3 module. You can find the source code here. We will build and release new binaries of Python for Wasm whenever a new upstream release is available. To find the latest release look for "python" releases in webassembly-language-runtimes on github. Also, Docker+Wasm fans, we are providing a python-wasm container image. The rest of this article will deal with examples of how to run python.wasm and leverage it to run your Python apps on WASI. Hands on python.wasm All of the examples below include a short explanation along with sample output, where relevant, so it is easier to read them through. Prerequisites To give it a try, you will need to have a few tools installed in advance. Most notably, a shell that has enough Unicode support to show emojis. Yep, this is what we use as part of our examples python3 on your machine Implementing pip for python.wasm is not universally possible, because WASI still does not offer full socket support. Downloading a package from the internet may not even work on some runtimes. But that is OK for most scenarios we are interested in, as python.wasm is likely to be used as a runtime in Cloud or Edge environments rather than a generic development platform. We will start by using a native python3.11 installation to setup a sample applications. And then we will show how you can run it on python.wasm. wget and zip These tools are required to download and extract the released binary. A WASI-compatible runtime As python.wasm is built for WASI you will need to get a compatible WebAssembly runtime, such as Wasmtime. We also provide an additional binary that will run on WasmEdge, which offers extended socket support on top of a modified WASI API. Since Docker+Wasm uses WasmEdge, this is the binary you will need if you want to build a WASM container image to use with Docker, as explained later in the article. Docker+Wasm To try the examples with Docker you will need "Docker Desktop" + Wasm version 4.15 or later. Setup All of the examples below assume you are using the same working directory. Some of them build on top of each other. Where this is the case we have tried referencing the previous one that we step on. First, prepare a temporary folder and download the python.wasm binary mkdir /tmp/try-python-wasm cd /tmp/try-python-wasm wget https://github.com/vmware-labs/webassembly-language-runtimes/releases/download/python%2F3.11.1%2B20230118-f23f3f3/python-aio-3.11.1.zip unzip python-aio-3.11.1.zip rm python-aio-3.11.1.zip Taking a look at the unzipped files we see two versions of python.wasm - one that's WASI compliant and one that can run on WasmEdge with its slightly non-standard and extended socket API. Also, the usr/local/lib folder includes a zip of the standard libraries, a placeholder Lib/lib-dynload and a Lib/os.py. The last two are not strictly necessary but if omitted will cause dependency warnings whenever python runs. tree . +-- bin | +-- python-3.11.1-wasmedge.wasm | +-- python-3.11.1.wasm +-- python-aio-3.11.1.zip +-- usr +-- local +-- lib +-- python3.11 | +-- lib-dynload | +-- os.py +-- python311.zip Now, let's get the sample script and the data it will work on wget https://raw.githubusercontent.com/vmware-labs/webassembly-language-runtimes/main/python/examples/emojize_text.py wget https://raw.githubusercontent.com/vmware-labs/webassembly-language-runtimes/main/python/examples/source_text.txt First time running python.wasm The Python standard libraries are packed into usr/local/lib and the python.wasm binary is compiled to look for this path in / So, in order to run Python properly we need to pre-open the current folder as root within the sandboxed WASM environment, where python.wasm will be running. For Wasmtime this is done via --mapdir. wasmtime run \ --mapdir /::$PWD \ bin/python-3.11.1.wasm \ -- -c "import sys; from pprint import pprint as pp; \ pp(sys.path); pp(sys.platform)" ['', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload'] 'wasi' We could do the same with the WasmEdge-compliant binary (note the slight differences in the CLI arguments). wasmedge \ --dir /:$PWD \ bin/python-3.11.1-wasmedge.wasm \ -c "import sys; from pprint import pprint as pp; \ pp(sys.path); pp(sys.platform)" ['', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload'] 'wasi' Running the repl If you want, you can play with the Python repl. wasmtime run --mapdir /::$PWD bin/python-3.11.1.wasm Python 3.11.1 (tags/v3.11.1:a7a450f, Jan 18 2023, 22:43:41) ... on wasi Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> >>> sys.platform 'wasi' >>> >>> sys.version_info sys.version_info(major=3, minor=11, micro=1, releaselevel='final', serial=0) Running an app with dependencies Next, let's assume we have a Python app that has additional dependencies. For example emojize_text.py, which we downloaded as part of the setup. Installing dependencies to the pre-compiled path To set up the dependencies we will need pip3 (or python3 -m pip) on the development machine, to download and install the necessary dependencies. The most straightforward way of doing this is by running pip with --target pointing the path that is already pre-compiled into the python.wasm binary. Namely, usr/local/lib/ python3.11/ pip3 install emoji -t usr/local/lib/python3.11/ Now we can run our text emojizer. Taking a look at the sample source text. cat source_text.txt The rabbit woke up with a smile. The sunrise was shining on his face. A carrot was waiting for him on the table. He will put on his jeans and get out of the house for a walk. We get this result from emojize_text.py wasmtime run \ --mapdir /::$PWD \ bin/python-3.11.1.wasm \ -- \ emojize_text.py source_text.txt The woke up with a smile. The was shining on his face. A was waiting for him on the table. He will put on his and get out of the for a walk. Using a virtual environment Any more complex python application is likely to be using virtual environments. In that case, you will have a venv folder with all requirements pre-installed. All you need to leverage them is to: * Make sure this folder is pre-opened when running python.wasm * Add it to the PYTHONPATH environment variable Let's take a look at how to do this. We will start by creating a virtual environment within the same folder and installing 'emoji' in it. python3 -m venv venv-emoji . venv-emoji/bin/activate pip3 install emoji deactivate With what we did so far we were mapping the current folder as root anyway (for the sake of usr becoming /usr) so we only need to set the PYTHONPATH variable accordingly. wasmtime \ --env PYTHONPATH=/venv-emoji/lib/python3.11/site-packages \ --mapdir /::$PWD \ bin/python-3.11.1.wasm \ -- \ emojize_text.py source_text.txt The woke up with a smile. The was shining on his face. A was waiting for him on the table. He will put on his and get out of the for a walk. Passing an environment variable with WasmEdge is similar wasmedge \ --env PYTHONPATH=/venv-emoji/lib/python3.11/site-packages \ --dir /:$PWD \ bin/python-3.11.1-wasmedge.wasm \ emojize_text.py source_text.txt ... Running the Docker container Docker+WASM uses the WasmEdge runtime internally. To leverage it we have packaged the python-3.11.1-wasmedge.wasm binary in a container image available as ghcr.io/vmware-labs/python-wasm:3.11.1-latest. Here is an example of running the Python repl from this container image. As you can see from the output of the interactive session, the container includes only python.wasm and the standard libraries from usr. No base OS images, no extra environment variables, or any other clutter. The Dockerfile for this image is available at webassembly-language-runtimes/images/python/Dockerfile. docker run --rm \ -i --runtime=io.containerd.wasmedge.v1 \ ghcr.io/vmware-labs/python-wasm:3.11.1-latest \ -i Python 3.11.1 (tags/v3.11.1:a7a450f, Jan 27 2023, 11:37:16) ... on wasi Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> >>> sys.platform >>> 'wasi' >>> >>> import os >>> os.listdir('.') ['python.wasm', 'etc', 'usr'] >>> >>> [k for k in os.environ.keys()] ['PATH', 'HOSTNAME'] You can also run the Docker container to execute a one-liner like this. docker run --rm \ --runtime=io.containerd.wasmedge.v1 \ ghcr.io/vmware-labs/python-wasm:3.11.1-latest \ -c "import os; print([k for k in os.environ.keys()])" ['PATH', 'HOSTNAME'] Running the Docker container with dependencies The python-wasm container image comes by default just with the Python standard library, so if your project has extra dependencies you will need to take care of them. Let's reuse the venv-emoji environment in which we installed emoji in the example above. We need to do three things 1. Ensure that the emoji module installed in the venv-emoji folder is mounted in the running python-wasm container 2. Ensure that it is also on the PYTHONPATH within the running python-wasm container 3. Ensure that the python program and its data (in this case emojize_text.py and source_text.txt) are also mounted in the container A vital piece of knowledge here is that whatever you mount in the running container gets automatically pre-opened by the WasmEdge runtime. Same goes for all environment variables that you pass to the container when you run it. One way of doing what we want is to just mount site-packages from venv-emoji over the site-packages folder of the pre-compiled path in /usr/local. This is what this could look like: docker run --rm \ -v $PWD/venv-emoji/lib/python3.11/site-packages:/usr/local/lib/python3.11/site-packages \ -v $PWD/emojize_text.py:/emojize_text.py \ -v $PWD/source_text.txt:/source_text.txt \ --runtime=io.containerd.wasmedge.v1 \ ghcr.io/vmware-labs/python-wasm:3.11.1-latest \ -- \ emojize_text.py source_text.txt The woke up with a smile. The was shining on his face. A was waiting for him on the table. He will put on his and get out of the for a walk. An alternative would be to map the current folder as /opt and use PYTHONPATH like this: docker run --rm \ -v $PWD:/opt \ -e PYTHONPATH=/opt/venv-emoji/lib/python3.11/site-packages \ --runtime=io.containerd.wasmedge.v1 \ ghcr.io/vmware-labs/python-wasm:3.11.1-latest \ -- \ opt/emojize_text.py opt/source_text.txt The woke up with a smile. The was shining on his face. A was waiting for him on the table. He will put on his and get out of the for a walk. Wrapping it all in a new container image This way of running your python application with the python-wasm container is too cumbersome. Luckily OCI and Docker already offer a way to package everything nicely. Let's first create a Dockerfile that steps on python-wasm to package our emojize_text.py app and its venv into a single image. cat > Dockerfile.emojize <