It still generates a little bit of surprise to discover that there are people who use Python on a daily basis that are apparently quite unfamiliar with compiling said code. Or perhaps not; it is, after all, the world's most popular programming language, it has a syntax that's cleaner than many older languages, it has an enormous collection of extensions, and so forth. As a result, there are many people who use Python, but perhaps not so many who have the inquisitiveness and courage, to dive a little deeper. This short article is a deeper dive to understand a little more about the language.

The first common (novice) mistake is that Python is an interpreted language and can't be compiled. It most certainly can and is "compiled", but not in the same way that a compiled language (e.g., C/C++, Fortran, Pascal) is. If this sounds confusing one needs to dig a little into the architecture.

A Python program is compiled before being interpreted, but this step is hidden at the surface level. When Python is executed it generates byte code. This byte code is transformed and interpreted by the Python Virtual Machine which then converts the byte code to binary machine code that the computer processor can output.

A top-level Python source file can be compiled in the following manner:


$ python -m py_compile helloworld.py


Or, with multiple files:


$ python -m py_compile helloworld.py hellosystem.py hellogalaxy.py ...


Running this will generate a __pycache__ that will contain the byte-code (e.g., helloworld.cpython-38.pyc) version of the program. This binary byte-code can be directly invoked (e.g., python3 helloworld.cpython-38.pyc).

The distinction between byte code interpreted by the PVM and machine code is very important, as sometimes people believe (armed with this new information) that because Python can be compiled that this will lead to much faster execution. This is incorrect. A compiled Python script does not run any faster than a non-compiled script. So why would one want to do this?

The short answer is that a program doesn't just run, it must also load its environment and including any other programs, modules, packages, etc that have been imported. If the runtime of a program is quite long then the advantage of compiling a Python program is relatively less, and conversely, a Python program with a short runtime or one that imports a number of additional programs is going to have a more significant advantage. In any case, there will be some performance improvement because all Python programs will be converted to byte code and interpreted by the PVM anyway. The example helloworld.py has an improvement of over 300% in real time, for example. This is not bound by simplicity or length, by the way. For example, a complex mathematical Python program can take a very long time to load, but runtime execution is very quick. Testing, as always, is the best solution.

There are two other advantages to using compiled Python. The first is that the byte code provides some protection against unwanted changes in a shared environment, for example, a careless contributor who in their passion to fix what looks like a bug makes a modification to the source code and breaks the program. Of course, a version control system should be used but even then an additional layer of protection is often worthwhile. The second advantage is that compilation often will result in a significantly smaller file and when coupled with the faster load times, this is particularly useful for websites, embedded environments, and especially when using Python for network programming. Seriously, in such an environment when it's waiting for a trigger it is critical for a fast response; this can be achieved by having the modules like socket, stmplib, base64, etc already loaded.

All silver linings are attached to clouds, however, and compiling Python is no different. The byte code is designed for the particular system architecture it was compiled in and therefore is not exactly transportable to a new system: "different machine, different Python", as the saying goes. Thus distribution will, at the very least, require distributing all the relevant .py files for implementation.

There are, being computing, many further elaborations that one could engage in. For example, I have not discussed the deeper optimisation options for compilation and the pitfalls that could result. Nor have I raised the details of the interpreter and the evaluation loop, or the distinction between implementations at that level (e.g., CPython, Jython, IronPython, PyPy, etc). Finally, perhaps in a simpler direction, I have not discussed the use of the compile() function which converts s specified source as a code object to be executed, such as by eval() or exec(). These will be discussed at another time. In the meantime, if performance matters, you probably should compile your Python code.