Python Virtual Environments on Raspberry Pi Bookworm

Contents

What's a virtual environment?

While having to care about Python virtual environments is a bit of a bump in the road to getting started with Python they're not just a nuisance foisted upon you by curmudgeonly Linux distribution maintainers.

A virtual environment is a useful, logical way of isolating your project dependencies from each other and ensuring things don't break when there are conflicts..

eg: What happens if you create a new project using Library v2, but you have old projects that only work with Library v1? Bad things, that's what. A virtual environments mean you can just spin up a clean environment for your new project and never have to worry about breaking or updating your old ones.

They are also a great way for a user to install and use your project without affecting their other Python environments.

Why do I need one?

As of Raspberry Pi OS bookworm, you'll get a slap on the wrist in the form of an error message if you attempt to pip install any Python libraries on your Raspberry Pi.

It looks something like this:

$ pip install buildhat error: externally-managed-environment × This environment is externally managed ╰─> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install. If you wish to install a non-Debian-packaged Python package, create a virtual environment using python3 -m venv path/to/venv. Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make sure you have python3-full installed. For more information visit http://rptl.io/venv note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.

The "TLDR" (too long, don't read) of this message is that installing libraries from pypi (which is what pip does) into your system can cause all sorts of here-be-dragons weirdness and breakage. So it's recommended against.

Instead, you use a virtual environment which is just a bunch of monkeying with paths to ensure your Python project dependencies are completely self-contained, and isolated from the system and each other.

Okay, fine, how can I get started?

Virtual environments aren't half as tricky as they sound, you can set one up with a single command:

python3 -m venv --system-site-packages --prompt myenv ~/my_virtual_env

"Uh, you said it wasn't tricky?"

I did... didn't I...

Okay, let's break this down. First we have python3 -m venv, this bit is telling Python to load and run the "venv" module. Easy, right?

Next up we have --system-site-packages, this is a little trickier to understand. "System" in this case refers to your Linux Distro, and "Site Packages" are the Python packages (usually installed by "apt") available to it.

By specifying this argument we're asking "venv" to let our virtual environment see and use these system packages. This is very useful if we've got something big, complicated and scary installed by "apt" that might otherwise be tricky or unwildy to install in our virtual environment. In the case of a Raspberry Pi, this might be gpiozero and lgpio which are the default, blessed two-hit combo that make GPIO access quick and easy.

Then we have --prompt myenv, this bit is entirely optional but is useful for giving each of your virtual environments a unique, friendly name so you can tell them apart. This might be your project name, or anything memorable to you. It'll show up in your prompt, like so:

(myenv) phil@pirate:~/

Finally, and this is it, I swear, you have the location you want to put your virtual environment. This could ve anywhere but in most cases you might want it in your project directory.

In this case I've chosen ~/my_virtual_env which creates the "my_virtual_env" directory inside your home directory- you can get there with cd ~.

To use your virtual environment you must first activate it:

source ~/my_virtual_env/bin/activate (myenv) phil@pirate:~/

And now you can pip install to your heart's content!

Okay, that sounds tedious. Make it easy for me!

While a per-project virtual environment is a great idea in principle, we're used to thinking about our Pi as the project and most of the time it'll probably be doing one thing. So why bother with all this complexity?

We have a couple of answers to this conundrum:

  1. Update our Python software installers to check for a virtual environment and ask you if you'd like one created.
  2. Provide a handy, dandy script you can source from your ~/.bashrc to make activating a virtual environment automagic!

"Sauce what from my whosit!?"

Right, ~/.bashrc is a little wall-of-script that gets executed every time you open a terminal window on, or an SSH session into your Pi. By adding a little bit of virtual environment magic into this file we can create and activate a virtual environment automatically so you mostly never need to think about it.

That script might look a little something like this:

VENV_DIR=~/.virtualenvs/pimoroni if [ ! -f $VENV_DIR/bin/activate ]; then printf "Creating user Python environment in $VENV_DIR, please wait...\n" mkdir -p $VENV_DIR python3 -m venv --system-site-packages --prompt Pimoroni $VENV_DIR fi printf " ↓ ↓ ↓ ↓ Hello, we've activated a Python venv for you. To exit, type \"deactivate\".\n" source $VENV_DIR/bin/activate

This horrifying spaghetti of arcane incantations does a few simple things:

  1. Check for a venv in ~/.virtualenvs/pimoroni
  2. Create one if it doesn't exist
  3. Output a little message so you know what's happening
  4. Activate the venv!

The first time you run it, it'll create an activate a virtual environment, taking a few seconds. Subsequent runs (when you open your terminal) will instantly drop you into a usable, semi-isolated* environment for all your Python tinkering!

* note that the --system-site-packages argument exposes your environment to the system Python packages. We do this specifically for gpiozero and lgpio, the latter of which is simply broken on PyPi.

But what about Thonny?

Raspberry Pi OS runs a somewhat customised version of Thonny that starts up into a cut-down basic mode by default.

It might not be obvious at first, but this mode has no awareness of your virtual environment unless you delve into the settings and tell Thonny where it is.

You can do this in Thonny by clicking the three little bars in the bottom right corner (or via the menu if you've enabled full mode) and selecting ~/.virtualenvs/pimoroni/bin/python

Alternatively you can edit ~/.config/Thonny/configuration.ini and make sure it contains the following (adjusting for your home directory accordingly):

[LocalCPython] last_configurations = [{'run.backend_name': 'LocalCPython', 'LocalCPython.executable': '/home/pi/.virtualenvs/pimoroni/bin/python'}] executable = /home/p[/.virtualenvs/pimoroni/bin/python

Uh, I'm having a little trouble with...

Is there an easier way to manage environments?

Yes. You can install and activate virtualenvwrapper to give you some handy command shortcuts for creating and activating virtual environments.

phil@pirate: sudo apt install virtualenvwrapper python3-virtualenvwrapper

You can then activate the virtualenvwrapper helper with:

phil@pirate: source /usr/share/virtualenvwrapper/virtualenvwrapper.sh

If you add this line to the bottom of your ~/.bashrc then virtualenvwrapper will be activated when you log in.

Once activated, you can use commands like -

phil@pirate: mkvirtualenv virtual_environment_name phil@pirate: workon virtual_environment_name

Virtualenvwrapper will store environments in ~/.virtualenvs, which should be recognised by tools like VSCode.

You can read more about virtualenvwrapper here.

How do I run my Python code as superuser/sudo?

I can feel the glassy, disapproving stares on the back of my head as I write this, but: sometimes it's okay to use sudo.

Notable cases might include running Python libraries like rpi_ws281x which control WS2812 (a.k.a NeoPixels) LEDs by using low-level Linux wizardry and hardware pokery.

Trying to use sudo inside a virtual environment will normally result in sadness and bad times, but you can fix that with one weird trick:

sudo --preserve-env=PATH python my_code.py

Since virtual environments monkey with $PATH to work their magic, we want to be sure that sudo works within the context of our monkey'd $PATH.

The argument --preserve-env=PATH does just that!