Managing Dependencies in Large Projects

You inherit a Python service with 213 transitive dependencies, two of them in conflict, and a CI pipeline that passes on Tuesday and fails on Friday. “Just update the lock file” is what everyone says, and then you spend a day figuring out which lock file, in which format, generated by which tool, and whether the hash covers the wheel or just the manifest. Managing dependencies at scale is mostly about choosing your fights, and the fights are predictable. The rest of this article walks through the patterns that make those fights cheap to win.

Key takeaways

• A pyproject.toml declares intent; a lock file records resolution. Keep them in separate files.
• Float or use ~= in the manifest, then pin exact versions plus hashes in the lock file.
• PEP 621 extras are for end users; PEP 735 dependency groups are for your dev workflow. Do not conflate them.
• PEP 751 standardizes pylock.toml as the tool-independent lock file format (Final, accepted 2025-03-31).
• CI must install from the lock file, never re-resolve. Anything that re-runs the resolver in CI is a footgun.

The two layers: manifest and lock

Once a project has more than a handful of dependencies, the single biggest upgrade you can make to its reliability is separating what you ask for from what gets installed. These are two different things, and they belong in two different files.

pyproject.toml is your manifest. It records intent: “I want fastapi~=0.115 and pydantic~=2.9.” A constraint is a promise about what is acceptable, not which artifact will be picked. Constraints are written by humans, rarely, and should stay permissive enough to absorb compatible releases.

A lock file holds the resolution: the exact versions, hashes, and URLs that satisfy your constraints at a moment in time. A tool writes it, regenerates it deliberately, and CI consumes it verbatim. Skip the lock file and you do not have dependency management. You have dependency hope.

npm, cargo, and bundler have shipped this split for years. Python is catching up: PEP 751 (Final, accepted 2025-03-31) standardizes the lock file as pylock.toml.

# pyproject.toml: the manifest
[project]
name = "webapp"
version = "2.0.0"
requires-python = ">=3.11"
dependencies = [
    "fastapi~=0.115",
    "pydantic~=2.9",
    "sqlalchemy~=2.0",
]

Constraints live in the manifest. Each entry is a PEP 508 requirement string, written by a developer who has read the upstream changelog and decided on a range. The lock file, by contrast, is the resolver’s output. CI installs from it byte for byte, and reviewers should eyeball it in a pull request when a transitive bumps.

# pylock.toml: the resolution (excerpt)
lock-version = "1.0"
requires-python = ">=3.11,<3.14"
created-by = "uv 0.5.0"
environments = ["sys_platform != 'win32'"]

[[packages]]
name = "fastapi"
version = "0.115.6"

Intent changes drive manifest edits (a new feature, a tighter constraint). World changes drive lock file edits (a security patch lands, a transitive bumps). Hold onto that distinction and most of the rest of this article follows. See the pyproject.toml guide for the file schema, and the PEP 621 spec for the canonical field reference.

Managing dependencies: choosing version pins

Version specifiers in pyproject.toml come in three flavors, and most of the confusion I see in real codebases comes from mixing them up. The right default for almost every dependency is a compatible-release pin, but the alternatives have their place.

• Floats (requests>=2.31,<3): accept any compatible release in the range. Good when you want bugfixes to flow in. Risky across long-lived branches.
• Compatible release (pydantic~=2.6.0): means >=2.6.0, <2.7. Pin to a minor, accept patch updates. The right default for libraries that follow SemVer.
• Exact (requests==2.32.3): pin to one specific version. Almost never what you want in a manifest, because it freezes you out of security patches.

Float or compatible-release in the manifest, exact version plus hash in the lock file is the clean rule. Your manifest should look like this:

dependencies = [
    "requests>=2.31,<3",
    "pydantic~=2.6.0",
    "rich~=13.7.0",
]

If you’re tempted to write ==, you’re probably trying to prevent something. Move that intent into the lock file where it belongs, and let the manifest stay loose.

A subtle gotcha hides in ~=. PEP 440 defines it as accepting any version where the prefix matches. For ~=2.6, that means >=2.6, <3 (not >=2.6, <2.7!) when only one segment is given. Write the two-segment form and you are usually safe: ~=2.6.0 means >=2.6.0, <2.7.0. The single-segment form has caught a lot of teams off guard. Full grammar in PEP 440.

Extras vs. dependency groups

Python now has two mechanisms for “extra” dependencies, and they aren’t interchangeable. Picking the wrong one ships dev noise into your users’ install metadata.

project.optional-dependencies (PEP 621) defines extras, features you ship to end users:

[project.optional-dependencies]
postgres = ["psycopg[binary]~=3.2"]
redis    = ["redis~=5.0"]
all      = ["webapp[postgres,redis]"]

A user installs with pip install webapp[postgres]. Those extras get baked into the wheel’s METADATA so downstream resolvers can see them.

[dependency-groups] (PEP 735, accepted November 2023) defines groups, internal organization that stays out of the wheel metadata:

[dependency-groups]
test = ["pytest~=8.3", "httpx~=0.27", "pytest-asyncio~=0.24"]
lint = ["ruff~=0.7", "mypy~=1.13"]
dev = [
    {include-group = "test"},
    {include-group = "lint"},
    "ipython",
]

Groups don’t pollute pip install webapp. They are for your dev workflow: tests, linters, docs, typecheck. include-group is the killer feature here. You compose dev from test and lint without duplicating the lists.

A mistake I see often: putting test under optional-dependencies because it “feels” like an extra. Don’t. If a user types pip install webapp[test] and gets pytest, your METADATA is wrong, and you have shipped dev noise to production. PEP 735 groups exist precisely for this case. Full schema in the PEP 735 spec.

What is the PEP 751 lock file standard?

PEP 751 (status: Final) defines a single, tool-independent format for Python lock files. The canonical name is pylock.toml (or <project>.pylock.toml for project-specific files). Before PEP 751, every tool invented its own format; now there is one shape every tool can speak. As of mid-2026, uv and PDM are early adopters and pip support is rolling out incrementally.

Every top-level key has a job:

Key	Purpose
lock-version	Format version, currently "1.0"
requires-python	Version range, e.g. ">=3.10,<3.14"
environments	PEP 508 environment markers covered
extras	Optional-dependency groups captured
dependency-groups	PEP 735 groups captured
default-groups	Which of the above install by default
created-by	The tool that produced the file (e.g. "uv 0.5.0")
[[packages]]	One table per resolved package

A per-package [[packages]] entry has name, version, requires-python, dependencies (already resolved), a marker if applicable, and exactly one of: vcs, directory, archive, or sdist plus wheels. Hashes are mandatory: sdist.hashes and wheels[*].hashes are tables of {sha256 = "..."}. The attestation-identities field (PEP 740) carries provenance.

lock-version = "1.0"
requires-python = ">=3.10,<3.14"
created-by = "uv 0.5.0"
environments = ["sys_platform != 'win32'"]

[[packages]]
name = "requests"
version = "2.32.3"
requires-python = ">=3.8"
dependencies = [
    "charset-normalizer<4,>=2",
    "idna<4,>=2.5",
    "urllib3<3,>=1.21.1,<2",
    "certifi>=2017.4.17",
]

[[packages.wheels]]
name = "requests-2.32.3-py3-none-any.whl"
hashes = {sha256 = "e6f36a0f494e8e3b3..."}

You don’t usually hand-write these. Tools like uv and PDM emit them. But knowing the structure helps you debug “why did this version get picked” and “is this lock file covering the environment I think it’s covering.” Full field reference in the PEP 751 spec.

How does the resolver pick a version, and what trips people up?

The resolver does more than “find any version that satisfies all constraints.” Three behaviors cause most of the surprises, and each has a workaround you should know about before it bites you in production.

Transitive constraints collide. Library A requires libfoo<2 and library B requires libfoo>=2. The resolver picks one and somebody is unhappy. There is no clever default. You will either pin libfoo directly in your manifest, drop one of the two libraries, or accept that the resolver will surprise you. Run uv pip check (or pip check) after install to catch this.

Markers constrain which packages are considered. A requirement like colorama ; sys_platform == 'win32' only needs to be satisfiable on Windows. A lock file for a Linux CI run doesn’t need to resolve it. PEP 751 records markers per-package, so multi-platform repos can ship pylock.linux.toml and pylock.windows.toml from the same pyproject.toml.

Optional dependencies are resolved as-if installed. If you ship an extra like postgres that pulls in a library with a strict Python requirement, the base install can fail to resolve for users on old Python. The resolver treats all of your optional-dependencies as constraints to satisfy if the user asks for them, but if any extra has a genuinely unsatisfiable constraint, the entire project can fail to build.

You can dry-run conflict detection with stdlib and the installed env:

import subprocess

result = subprocess.run(
    ["uv", "pip", "check"],
    capture_output=True, text=True,
)
print("Conflicts found:" if result.returncode else "No conflicts.")
print(result.stdout.strip() or result.stderr.strip())

# output (example):
# No conflicts.

For large dependency graphs, the backtracking cost is real. uv and Poetry use a PubGrub-style resolver that is usually an order of magnitude faster than pip’s resolver on big projects. If your cold installs are slow, that is probably why.

What does it take to get a reproducible install?

Reproducibility is what the lock file is for. Three things make it work, and skipping any one of them means you don’t have it yet.

Hashes. PEP 751 records SHA-256 hashes for every artifact by default. If you use the legacy pip install -r requirements.lock.txt form, add --require-hashes so pip refuses to install anything whose hash doesn’t match. Hashing the lock file itself but not the artifacts is theatre. Both are needed.

Pinned Python range. requires-python in the lock file is the resolver’s view of the Python compatibility range, distinct from the manifest’s range. CI should test on every minor in the range, not just one.

Multiple environments, one manifest. When you ship for both cp39 and cp313, generate one lock file per environment. The manifest stays the source of truth; the locks are derived artifacts.

You can inspect any pylock.toml with stdlib alone. No extra dependencies required:

import tomllib
from pathlib import Path

data = tomllib.loads(Path("pylock.toml").read_text())
print(f"Lock version: {data['lock-version']}")
print(f"Pinned Python: {data['requires-python']}")
print(f"Packages: {len(data['packages'])}")

# output (example):
# Lock version: 1.0
# Pinned Python: >=3.10,<3.14
# Packages: 47

The tomllib module reference documents the parser.

Monorepo and workspace patterns

Once a repo has more than one Python package, the question is how they share dependencies. Three patterns cover most cases, and you will likely use a mix.

Path dependencies for local packages. When package B lives in ../core and you want webapp to depend on it during development:

dependencies = [
    "webapp-core @ { path = '../core', editable = true }",
]

The editable = true makes the install a symlink or .pth shim, so edits to core are picked up without reinstalling.

Workspaces for shared locks. uv, Poetry, and PDM all have workspace concepts where multiple packages share a single lock file at the repo root. Each sub-package has its own pyproject.toml with its own dependencies, and the resolver sees them as one graph. That is exactly what monorepos need: one source of truth, no version skew between siblings.

VCS dependencies for shared libraries not yet on PyPI. A pinned commit hash or tag is mandatory:

dependencies = [
    "shared-auth @ git+https://github.com/me/shared-auth@v1.2.3",
]

Without the @v1.2.3 or @abc1234, you have pinned a moving target. The lock file should record the resolved commit. If it doesn’t, you have a hole in your reproducibility story. Tool comparison in the Python pip vs poetry vs uv guide.

Security and supply chain

A lock file is also a security control, not just a reproducibility one. Three habits reduce the most common attack surface, and a fourth habit (the one most teams skip) is the one that bites you in audits.

Hash verification. With PEP 751 hashes in place, an attacker performing a PyPI MITM attack cannot swap a malicious wheel in. Install just fails. With pip --require-hashes, even a hostile local resolver cannot bypass it.

Provenance attestations. PEP 740 (referenced by PEP 751’s attestation-identities) lets publishers sign their builds. Trusted publishers via PyPI’s OIDC integration can sign on release without holding long-lived secrets. If your dependencies support attestations, your lock file records them. Check that they are present.

Vulnerability scanning. Run pip-audit (or uv pip audit) on a schedule. Pin your auditing tool’s version too. A compromised pip can rewrite a lock file at install time. pipx run pip-audit or uvx pip-audit both isolate the auditor from the env it is checking.

You can wire the audit into CI with one line:

# In a CI job, after `uv sync --frozen`
uvx pip-audit --strict
# output (example):
# No known vulnerabilities found

VCS dependencies that aren’t pinned to a commit hash are the single biggest reproducibility hole I see in audit reports. Hash them, then check the hashes.

Operational habits

Lock files don’t keep themselves honest. A few rules of thumb for keeping things clean, with the CI commands that enforce them.

• CI must install from the lock file, not re-resolve. uv sync --frozen, poetry install --no-update, pip-sync. Anything that re-runs the resolver in CI is a footgun.
• Check for lock file drift in CI. uv lock --check (or your tool’s equivalent) as a required check catches the case where two PRs both bumped a transitive dependency and only one will land cleanly.
• Review outdated reports on a schedule, not on demand. uv pip list --outdated gives you a list. A weekly review beats a fire drill.

import json, subprocess

out = subprocess.check_output(
    ["uv", "pip", "list", "--outdated", "--format", "json"]
)
data = json.loads(out)
direct = {"requests", "pydantic", "rich"}
for pkg in data:
    if pkg["name"] in direct:
        print(f"{pkg['name']}: {pkg['current_version']} -> {pkg['latest_version']}")

# output (example):
# requests: 2.31.0 -> 2.32.3

• Keep requires-python and your CI matrix aligned. If the manifest says >=3.10,<3.14, every minor in that range should be in your test matrix. Otherwise you are testing a subset of the users you are shipping to.
• Use markers correctly. A sys_platform == 'win32' marker should match an environment you actually deploy to. Stray markers from copy-pasted dependencies are a quiet source of confusion.

Frequently asked questions

Do I need a lock file in a library, or only in an application?

Both, but with different urgency. Applications (web services, CLIs, scripts) cannot operate safely if a transitive bumps a breaking change between deploys. A lock file is mandatory. Libraries ship a lock file for their own dev/CI loop, but the lock is internal. Consumers of the library use the library’s manifest constraints, not its lock file.

What is the difference between pyproject.toml and setup.py?

pyproject.toml is the modern, PEP 621 metadata format. setup.py is the legacy executable build script. New projects should use pyproject.toml for metadata. setup.py is still common for projects that need custom build logic, but the metadata belongs in pyproject.toml regardless.

Can I have more than one lock file?

Yes. PEP 751 explicitly supports per-environment locks (pylock.linux.toml, pylock.windows.toml) generated from the same pyproject.toml. Each covers a different environments marker set. This is the right pattern when the same project ships wheels for multiple Python versions or operating systems.

Should I commit uv.lock or pylock.toml?

Yes. A lock file is one of the most important files in your repo: the artifact that makes installs reproducible. CI reads from it on every run, and developers should commit their changes to it as they would any source file. See the Python pip vs poetry vs uv guide for tool-specific behavior.

Conclusion

Managing dependencies in a large Python project comes down to two habits: write the manifest for humans (loose constraints, clear intent) and let a tool write the lock file for machines (exact versions, hashes, environment coverage). Pair that with PEP 621 extras for end-user features, PEP 735 dependency groups for internal dev workflow, and PEP 751’s pylock.toml for the lock file, and you have a reproducible foundation that survives a 200-package dependency graph. If you only take one thing away from this article, make it this: install from the lock file in CI, every time, and let the resolver stay out of the build path.

See also

• pyproject.toml guide for the file structure every Python project needs.
• Python pip vs poetry vs uv for tool comparison across the patterns above.
• tomllib module reference for the stdlib parser used to inspect lock files.
• PEP 751 for the canonical pylock.toml specification.