always freeze requirements with pip-compile to avoid unpleasant surprises

Have you ever been in a situation, where you wanted to setup a project you haven’t been working on for a while and it failed with a cryptic error for no apparent reason?

Chances are that you didn’t freeze your requirements and one of the new packages you just installed got upgraded and the new code is incompatible with your codebase and this causes a random error!

This could be avoided by freezing requirements.txt after the initial development phase of a project. This solution however prevents an efficient upgrade process later on. What would an efficient dependency upgrade process look like?

  • Unfreeze all dependencies
  • run pip install –upgrade on all dependencies
  • test the project, if there are problems, find and downgrade the problematic package or upgrade the project’s code. If everything is ok, go to next step
  • freeze all dependencies again, check changes into source control and deploy.

Enters pip-tools with pip-compile!

pip-compile does exactly that. In order to be able to easily upgrade all packages, pip-compile works with two separate files.

  1. requirements.in It contains only the packages added by the developer. Typically, this is also the place where the developer adds comments next to a package, explaining why it is needed and describing any quirks or special information. The developer does NOT add any version numbers here, unless it’s required to make the project run (example: package-name>3.5 # doesnt work with a lower version)
  2. requirements.txt This file contains the compiled output from pip-compile including any dependent packages. No manual edits should be done here, they would be overwritten upon subsequent compilation. All packages are frozen in this file (= have a version number assigned). The project uses this file to install dependencies. It’s important that this file is included in source control (git).

Pip Compile Workflow

  • the developer manages packages in requirements.in – generally **without** versions
  • then create a new requirements.txt with docker-compose run --rm web pip-compile requirements.in > requirements.txt
  • docker-compose exec web pip install -r requirements.txt or just simply docker-compose build web
  • Testing, QA & fixes, checking changes into source control, finally deployment