Python poetry: Explanations and tips

For those who are used to using a “basic” virtual environment with virtualenv, poetry can be confusing. Here are some clarifications and tips on using poetry.

Check your configuration

Run poetry config --list. It will give you an idea of its configuration.

Poetry “hides” the virtual environnement

But it exists anyway. When you run poetry install, this environment is created in a place that depends on your configuration. If virtualenvs.in-project is false, it will be in the directory defined by cache-dir. If it is true, it will be created under the .venv directory from the directory where the command is executed.

Tip: Set virtualenvs.in-project to true

Personally, I prefer to keep my virtual environment on hand. If I need to delete it for any reason (especially for caching problems!), it’s directly in my project, I don’t have to look for it.

poetry config virtualenvs.in-project true

Also, if I delete the project, the environment is deleted at the same time. If it is in the common directory, I assume it stays there indefinitely.

What the run and shell commands are

• run loads the virtual environment and then executes the command that follows.
• shell loads the virtual environnement and starts a shell.

Here are the equivalent non-poetry commands.

Tip: How to work with a local dependency in editable mode

If one of your dependencies is local (so not on pypi) and it is source code (so not a .zip or a .whl, once it is installed, you have to specify manually in pyproject.toml to treat it in modifiable mode. For example:

my-package = {path = "../my/path", develop = true}

Then you need to remove your virtual environment and reinstall it. At least, this is the case at the time of writing under poetry 1.1.13. If you run poetry install or poetry update after the change is made, the change is not taken into account. If you manually inspect your .venv/lib/python/site-packages/ directory you will see that the code is still copied there rather than linked.

So, erase .venv and run poetry update.

IMPORTANT: update, not install!

Here is the reason: poetry install will simply take what is in poetry.lock and install it without considering the change in pyproject.toml. Running poetry update forces it to update poetry.lock, considering pyproject.toml before proceeding with the installation properly.

Once done, you can look at the .venv/lib/python/site-packages/ directory and you should see <your package>.<extension> where extension may vary depending on what is used to build <your package>.

Tip: How to work with a local dependency in non-editable mode

It is much easier to work with a dependency in editable mode than in non-editable mode. If for some reason you are stuck and forced to install it in non-editable mode, there is one main trick to know.

Each time the dependency changes, the version number must change.

Otherwise, poetry might not detect the change and reuse the last build that was generated. This is easily verifiable: you can compare the code copied into .venv vs. the package source. They will have diverged.

Tip: How to make sure that poetry is used to build a local dependency

Even if your dependency has a pyproject.toml file, even if you run poetry install to install it, there is no guarantee that poetry will be used to build it. You must specify this in its pyproject.toml file:

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

I didn’t track down the exact cause of the problem, but I could see that without these directives, rather than creating the “build” in the <your package>/build directory, it is generated under <your package>/<your package>.egg-info, which is a deprecated format according to setuptools documentation.

Tip: Do not install development dependencies in production

When installing production-ready code, for example in a docker image, you should use the --no-dev option during installation. (poetry install --no-dev) This will make sure that only regular dependencies are installed, which is faster, smaller and reduces the attack surface.

Be careful: If you have miscategorized dependencies that are in the [tool.poetry.dev-dependencies] group that should be in the [tool.poetry.dependencies] group, you will get a broken assembly.