freeze requirements with pip-compile as a best practise

Have you ever been in a situation, where you wanted to setup a project you haven’t been working on for a while and it failed with a cryptic error for no apparent reason?

Chances are that you didn’t freeze your requirements and one of the new packages you just installed got upgraded and the new code is incompatible with your codebase and this causes a random error!

This could be avoided by freezing requirements.txt after the initial development phase of a project. This solution however prevents an efficient upgrade process later on. What would an efficient dependency upgrade process look like?

  • Unfreeze all dependencies
  • run pip install –upgrade on all dependencies
  • test the project, if there are problems, find and downgrade the problematic package or upgrade the project’s code. If everything is ok, go to next step
  • freeze all dependencies again, check changes into source control and deploy.

Enters pip-tools with pip-compile!

pip-compile does exactly that. In order to be able to easily upgrade all packages, pip-compile works with two separate files.

  1. requirements.in It contains only the packages added by the developer. Typically, this is also the place where the developer adds comments next to a package, explaining why it is needed and describing any quirks or special information. The developer does NOT add any version numbers here, unless it’s required to make the project run (example: package-name>3.5 # doesnt work with a lower version)
  2. requirements.txt This file contains the compiled output from pip-compile including any dependent packages. No manual edits should be done here, they would be overwritten upon subsequent compilation. All packages are frozen in this file (= have a version number assigned). The project uses this file to install dependencies. It’s important that this file is included in source control (git).

⚠️ Warning! Pypi packages can have specific versions marked as prereleases. Those are ignored by pip-compile! If you want to use a prerelease the corresponding version has to be updated manually in requirements.txt after compilation.

Pip Compile Workflow

  • the developer manages packages in requirements.in – generally **without** versions
  • then create a new requirements.txt with docker-compose run --rm web pip-compile requirements.in > requirements.txt
  • docker-compose exec web pip install -r requirements.txt or just simply docker-compose build web
  • Testing, QA & fixes, checking changes into source control, finally deployment

Pip Compile Setup

  1. Add pip-tools to requirements.txt
  2. Copy requirements.txt to requirements.in
  3. Run pip-compile requirements.in >> requirements.txt
  4. Rebuild the project to ensure everything works
  5. You can now remove all or some of the version pinning from your requirements.in