Python comes preinstalled in most operating systems by default. But sometimes you may want to upgrade to a newer version of Python or a new version of a library that you depend on which is not compatible with your current version of Python.

If you upgrade your system version, you may introduce changes that are not backward compatible, and you may find yourself hours deep into the rabbit hole of upgrading your entire system.

What we should do instead is use a library like Pyenv for installing multiple versions of Python and switching between them as needed.

We'll also see how virtual environments can help us isolate changes between projects and how to keep project dependencies pinned to a specific version of Python.

Requirements

• Ubuntu 20.x / WSL
• Internet connection

Open up your terminal (if you’re using WSL, open up your Ubuntu terminal). First, we will update our package index and upgrade any installed packages.

sudo apt update && sudo apt upgrade -y

Enter your password if prompted and wait for the terminal to finish.

…

Now, we will install all the python toolchain dependencies and build tools. Even though Ubuntu ships with python installed, we still need some other libraries to build and run python packages. This is the recommended list of dependencies for sane build environments, according to pyenv instructions.

sudo apt install -y make build-essential build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev python-dev python3-dev python3-pip python3-virtualenv python3-venv git

Once that finishes, you should have all the necessary libraries installed. Now for the python specific part.

Pyenv

The recommended way to manage your python dependencies and packages is to encapsulate your builds inside a python virtual environment. You might be wondering if you really need this, the answer (unless you want to spend most of your time in package versioning hell) is definitely yes.

We’ll use pyenv to manage python installations.

git clone https://github.com/pyenv/pyenv.git ~/.pyenv

We also need to add these lines to your .bashrc (or .zshrc if you use zshell)

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"'

Finally, you will need to reload your terminal session to apply the changes, or run source ~/.bashrc.

Now, let’s go ahead and make sure we have Python installed.

python3 --version
# Python 3.8.6

So my current version is 3.8.5, but there are newer versions out there. We can install them using pyenv without messing with the default system version.

pyenv install 3.11

Once that’s done, we can list all the installed versions with:

pyenv versions
* system (...)
  2.7.18
  3.8.13
  3.11.0

You should see some output like that, which tells me I have a system version, and others installed and available.

The reason why this matters is that some legacy python code needs to run with python2 while newer code should be using python3 versions. Also, when a new version of python comes out, you can try it in a new project by installing it with pyenv without having to worry about other legacy code not being able to run on the newer version.

Python virtual environments

Let’s introduce virtual environments by looking at a problem you might encounter. You start working on some project which uses some pip package, like pandas. Without knowing about virtualenvs, you just do what one does to install pandas like pip install pandas, and just continue your work.

All is good and well… except for the fact that now you installed a version of pandas in your system dependencies, which means some code you wrote months ago that depended on pandas too is now broken because pandas changed versions, and now it’s incompatible with your old code. That can be very messy.

What we really want is to make sure whatever version of pandas we build our project with never changes behind the scenes, so our code stays working even though pandas itself might change in the future. This is known as locking your dependencies, and that’s exactly what virtualenvs are for.

Let’s install virtual environments on top of pyenv.

git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv

echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.profile

Now I’ll show you how you can use virtual environments to isolate dependencies for a new project. I recommend creating a src directory in your root dir and putting all your projects in there. Like this:

exec "$SHELL"

mkdir ~/src/new-project
cd ~/src/new-project

pyenv virtualenv 3.11 venv

This just creates a new virtualenv which will use python3.10-dev called venv. You can use any version of python installed by pyenv and any name you like, I just happen to use venv for consistency.

The last step is to actually activate the environment, so every command we run in this directory is isolated from the rest of our python installations.

pyenv activate venv

Now, you should see something like this

(venv) user@epyc02:~/src/new-project$

That first part in the path is telling us we are inside a virtual environment called venv.

BTW you can exit the virtual environment with pyenv deactivate, in case you want to switch between projects or install some packages outside your project.

Wrapping up

OK, so now you are all set with python and you can install any dependencies in your project by running pip install <package>. This will save your packages inside of your venv directory and they will be local to your project. If you’re using git it’s recommended to ignore everything under your venv directory, so go ahead and add venv to your gitignore.

Note: if you called it something other than venv, substitute your environment’s name below instead of venv.

echo 'venv' >> .gitignore

That way, if you deploy your code later or someone is pulling your code but has a different type of CPU when they install the packages, they will actually get the right build for their particular environment.

And that is all for our python environment setup. Go write some code!