Nix + python = 💖 ?

Well, it seems that quite a lot of people struggle with python on NixOS! Actually, I don't think it's that difficult so let's dive in! This post is meant to be a definitive guide to python usage in NixOs. By usage, I mainly mean 2 things:

  • make any python application work, whether it is a cli tool or has a gui
  • but also how to develop a python application, both with or without a virtualenv

I plan to update this post if other cases present themselves, so don't hesitate to reach out and give me your non-working use-case!

But why is it difficult?

This is the first question I asked myself: why do people think it's difficult? From my POV, it is often actually easier than a c++ app for instance. I think it boils down to userbase. I might be wrong, but the python userbase is typically:

  • less software engineers and more scientists
  • less knowledgeable about the native world, linking, compilation etc. than c++ devs

After all, why should python dev care about all these low-level considerations? A lot of us use python when we precisely don't want that mess :-)

But the problem is that NixOS breaks some assumptions other distros are making and that in turns breaks some applications relying on those assumptions. To use or package python application (which is more or less the same thing in NixOS), we need to know about those so that we know how to make those applications happy.

Which assumptions are not true in NixOS?

Mainly, that "some stuff" will be available globally, once you install the right packages. Usually on other distros, you can assume the following things:

  • all the installed executables would be available directly because they are in your PATH
  • all the installed libs are installed in /usr/lib or other path that is in your LD_LIBRARY_PATH
  • same things for headers.

That's usually why a simple pip install works out of the box (well, after some apt or yum commands sometimes) in other distros! Python (an executable) needed by pip is available in the path, and all the native (.so) libraries that this python program might need directly or through some dependencies will be found, either at build time, or at runtime.

NOTE: pure python applications are not compiled, but some dependencies you may use are. For instance, psycopg2 distributes its source, and you need libpq.h to compile it when doing a pip install. Numpy is actually coded in C, although they do distribute already compiled wheels for most platforms so you probably won't need anything to compile it.

One example: numpy (missing .so at runtime)

So actually, let's keep that numpy example and see what happens if you naively try to install it in NixOS. Let's say you have added python3 to your config, either through environment.systemPackages or home.packages. So python3 is in your PATH, like the binaries of all the packages you've directly added in your config. So you merrily go ahead and create a virtualenv:

/tmp via 🐍 v3.12.8
❯ python3 -m venv .venv

/tmp via 🐍 v3.12.8
❯ source .venv/bin/activate

/tmp via 🐍 v3.12.8 (.venv)
❯ pip install numpy
Collecting numpy
  Downloading numpy-2.2.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)
Downloading numpy-2.2.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.1/16.1 MB 45.0 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.2.5

Yay! Works perfectly right?

But when we try to import it:

❯ python3 -c 'import numpy as np'
Traceback (most recent call last):
  File "/tmp/.venv/lib/python3.12/site-packages/numpy/_core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/tmp/.venv/lib/python3.12/site-packages/numpy/_core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/tmp/.venv/lib/python3.12/site-packages/numpy/_core/overrides.py", line 7, in <module>
    from numpy._core._multiarray_umath import (
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

# I omitted the rest, there are many more scary messages after

Bummer! The standard c++ libraries are not found...

We can confirm that if we download a whl from this page, unzip it somewhere and check what one of the included .so needs:

❯ ldd numpy/_core/_multiarray_tests.cpython-313t-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffff5e50000)
        libstdc++.so.6 => not found
        libm.so.6 => /nix/store/p9kdj55g5l39nbrxpjyz5wc1m0s7rzsx-glibc-2.40-66/lib/libm.so.6 (0x00007f9fef8ae000)
        libgcc_s.so.1 => /nix/store/y4d9iir0yqmrcswaqfi368d8m1rkv14s-xgcc-13.3.0-libgcc/lib/libgcc_s.so.1 (0x00007f9fef889000)
        libc.so.6 => /nix/store/p9kdj55g5l39nbrxpjyz5wc1m0s7rzsx-glibc-2.40-66/lib/libc.so.6 (0x00007f9fef691000)
        /nix/store/p9kdj55g5l39nbrxpjyz5wc1m0s7rzsx-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 (0x00007f9fef9c0000)

Notice the "not found".

(Note: NixOS does provide automatically some librairies which you can't avoid if you want anything to work at all, just very few).

Second example: psycopg2 (missing executable at build time)

If you try to install psycopg2, it's even worse. It does not even install:

❯ pip install psycopg2

Collecting psycopg2
  Downloading psycopg2-2.9.10.tar.gz (385 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
# [...]
      Error: pg_config --libdir failed:
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
# [...]

The error is about a missing pg_config, even though I do have postgresql installed on my machine!

It's annoying, why NixOS does that?

This might be an evidence to some, but still let's say it out loud: being reproducible is the very "raison d'être" of nixos, and you can't be reproducible if you depend on some external contingencies. Everything you depend on must be explicitly declared, and each dependency it will be a derivation corresponding to a specific version of a source code (and a specific version of each tool involved anywhere from the build process until the runtime).

Let's solve the "python on NixOS" problem

So how will we declare all these dependencies? How will we know what to declare? Let's dive in!

To be more exhaustive, there are 2 main ways of using a python application / library:

  • as a dev, you want to have a shell that can directly run the application, so that you can hack on it
  • as a user, you want to have it installed on your system in the simplest way possible

And I have identified 3 main types of python scripts/applications/libs:

  • they can be pure python script / package without any dependency
  • they can be pure python and have dependencies that are also pure python
  • they or their dependencies may have some native dependencies both in the installation step or at runtime.

So for each of these

And flakes?

NOTES:

  • something odd, a lot of problems with python and nix
  • https://old.reddit.com/r/NixOS/comments/1k3r6lq/python_in_nixos_is_tedious/
  • https://old.reddit.com/r/NixOS/comments/1i8rj5n/python_that_just_works/
  • https://old.reddit.com/r/NixOS/comments/1i9gmsf/nix_python_hell/
  • https://old.reddit.com/r/NixOS/comments/1drkymj/python_is_a_nightmare_on_nixos/
  • https://old.reddit.com/r/NixOS/comments/1drkymj/python_is_a_nightmare_on_nixos/
  • https://old.reddit.com/r/NixOS/comments/1hcqsxa/python_on_nixos/
  • https://old.reddit.com/r/NixOS/comments/1k1ivss/getting_a_bit_frustrated_with_python_dev_flakes/
  • https://old.reddit.com/r/NixOS/comments/1k8oegb/how_to_use_python_system_wide_with_packages_in/

But python in nix is easier than a lot of c++ project for instance.

I think problems is:

  • python dev typically are not really knowledgeable in static vs dynamin linking, compilation, the native library path etc.
  • what is not a problem for a seasoned c++ dev may very well be for python dev
  • the users of python are scientists, not software engineering

This article is for them, not for people thinking "I don't see the problem".

NOTE: I think most of this is not specific to python.

Principle : in nix, you have to explicitly give the path to native librairies, you can't rely on any automatic detection.

3 main cases, by increasing order of difficulty:

  • scripts that only uses the included lib in python distribution. Easy:
    • either you have python installed globally, and you just need to use python myscript.py or to check the hashbang
    • either you don't and then very simple shell.nix
    • if you want to package it, easy. Either buildPythonApplication (TODO ex) or a writer
  • scripts that uses pypi deps, but no native dependencies anywere. Same principle as before, but you need to reference somehow the other deps
    • either from nixpkgs
    • or (if not present) use pip or uv.
    • or package the external dep (might be tedious)
    • NOTE: python packages that are only c just need a stdenv
    • the hardest: python app that uses or (transitively) depends on native libs OR native/external tools (subprocess)
      • you must use a shell.nix or a default.nix (well, it's better)
      • all the game is to identify which nix package has the deps, and add it to builInputs

Previous Post