📝 `requirements.yaml` and `pyproject.toml` structure¶

unidep allows either using a

requirements.yaml file with a specific format (similar but not the same as a Conda environment.yaml file) or
pyproject.toml file with a [tool.unidep] section.

Both files contain the following keys:

name (Optional): For documentation, not used in the output.
channels: List of conda channels for packages, such as conda-forge.
dependencies: Mix of Conda and Pip packages.
local_dependencies (Optional): List of paths to other requirements.yaml or pyproject.toml files to include.
optional_dependencies (Optional): Dictionary with lists of optional dependencies.
platforms (Optional): List of platforms that are supported (used in conda-lock).

Whether you use a requirements.yaml or pyproject.toml file, the same information can be specified in either. Choose the format that works best for your project.

Example¶

Example `requirements.yaml`¶

Example of a requirements.yaml file:

name: example_environment
channels:
  - conda-forge
dependencies:
  - numpy                   # same name on conda and pip
  - conda: python-graphviz  # When names differ between Conda and Pip
    pip: graphviz
  - pip: slurm-usage >=1.1.0,<2  # pip-only
  - conda: mumps                 # conda-only
  # Use platform selectors
  - conda: cuda-toolkit =11.8    # [linux64]
local_dependencies:
  - ../other-project-using-unidep     # include other projects that use unidep
  - ../common-requirements.yaml       # include other requirements.yaml files
  - ../project-not-managed-by-unidep  # 🚨 Skips its dependencies!
optional_dependencies:
  test:
    - pytest
  full:
    - ../other-local-dep[test]  # include its optional 'test' dependencies
platforms:  # (Optional) specify platforms that are supported (used in conda-lock)
  - linux-64
  - osx-arm64

Important

unidep can process this during pip install and create a Conda installable environment.yaml or conda-lock.yml file, and more!

Note

For a more in-depth example containing multiple installable projects, see the example directory.

Example `pyproject.toml`¶

Alternatively, one can fully configure the dependencies in the pyproject.toml file in the [tool.unidep] section:

[tool.unidep]
channels = ["conda-forge"]
dependencies = [
    "numpy",                                         # same name on conda and pip
    { conda = "python-graphviz", pip = "graphviz" }, # When names differ between Conda and Pip
    { pip = "slurm-usage >=1.1.0,<2" },              # pip-only
    { conda = "mumps" },                             # conda-only
    { conda = "cuda-toolkit =11.8:linux64" }         # Use platform selectors by appending `:linux64`
]
local_dependencies = [
    "../other-project-using-unidep",    # include other projects that use unidep
    "../common-requirements.yaml",      # include other requirements.yaml files
    "../project-not-managed-by-unidep"  # 🚨 Skips its dependencies!
]
optional_dependencies = {
    test = ["pytest"],
    full = ["../other-local-dep[test]"]  # include its optional 'test' dependencies
}
platforms = [ # (Optional) specify platforms that are supported (used in conda-lock)
    "linux-64",
    "osx-arm64"
]

This data structure is identical to the requirements.yaml format, with the exception of the name field and the platform selectors. In the requirements.yaml file, one can use e.g., # [linux64], which in the pyproject.toml file is :linux64 at the end of the package name.

See Build System Integration for more information on how to set up unidep with different build systems (Setuptools or Hatchling).

Important

In these docs, we often mention the requirements.yaml format for simplicity, but the same information can be specified in pyproject.toml as well. Everything that is possible in requirements.yaml is also possible in pyproject.toml!

Key Points¶

Standard names (e.g., - numpy) are assumed to be the same for Conda and Pip.
Use a dictionary with conda: <package> and pip: <package> to specify different names across platforms.
Use pip: to specify packages that are only available through Pip.
Use conda: to specify packages that are only available through Conda.
Use # [selector] (YAML only) or package:selector to specify platform-specific dependencies.
Use local_dependencies: to include other requirements.yaml or pyproject.toml files and merge them into one. Also allows projects that are not managed by unidep to be included, but be aware that this skips their dependencies! Can specify PyPI alternatives for monorepo setups (see PyPI Alternatives for Local Dependencies).
Use optional_dependencies: to specify optional dependencies. Can be installed like unidep install ".[test]" or pip install ".[test]".
Use platforms: to specify the platforms that are supported. If omitted, all platforms are assumed to be supported.

We use the YAML notation here, but the same information can be specified in pyproject.toml as well.

Supported Version Pinnings¶

UniDep supports a range of version pinning operators (the same as Conda):

Standard Version Constraints: Specify exact versions or ranges with standard operators like =, >, <, >=, <=.
- Example: =1.0.0, >1.0.0, <2.0.0.
Version Exclusions: Exclude specific versions using !=.
- Example: !=1.5.0.
Redundant Pinning Resolution: Automatically resolves redundant version specifications.
- Example: >1.0.0, >0.5.0 simplifies to >1.0.0.
Contradictory Version Detection: Errors are raised for contradictory pinnings to maintain dependency integrity. See the Conflict Resolution section for more information.
- Example: Specifying >2.0.0, <1.5.0 triggers a VersionConflictError.
Invalid Pinning Detection: Detects and raises errors for unrecognized or improperly formatted version specifications.
Conda Build Pinning: UniDep also supports Conda’s build pinning, allowing you to specify builds in your pinning patterns.
- Example: Conda supports pinning builds like qsimcirq * cuda* or vtk * *egl*.
- Limitation: While UniDep allows such build pinning, it requires that there be a single pin per package. UniDep cannot resolve conflicts where multiple build pinnings are specified for the same package.
  - Example: UniDep can handle qsimcirq * cuda*, but it cannot resolve a scenario with both qsimcirq * cuda* and qsimcirq * cpu*.
Other Special Cases: In addition to Conda build pins, UniDep supports all special pinning formats, such as VCS (Version Control System) URLs or local file paths. This includes formats like package @ git+https://git/repo/here or package @ file:///path/to/package. However, UniDep has a limitation: it can handle only one special pin per package. These special pins can be combined with an unpinned version specification, but not with multiple special pin formats for the same package.
- Example: UniDep can manage dependencies specified as package @ git+https://git/repo/here and package in the same requirements.yaml. However, it cannot resolve scenarios where both package @ git+https://git/repo/here and package @ file:///path/to/package are specified for the same package.

Caution

Pinning Validation and Combination: UniDep actively validates and/or combines pinnings only when multiple different pinnings are specified for the same package. This means if your requirements.yaml files include multiple pinnings for a single package, UniDep will attempt to resolve them into a single, coherent specification. However, if the pinnings are contradictory or incompatible, UniDep will raise an error to alert you of the conflict.

Conflict Resolution¶

unidep features a conflict resolution mechanism to manage version conflicts and platform-specific dependencies in requirements.yaml or pyproject.toml files.

How It Works¶

Version Pinning Priority: unidep gives priority to version-pinned packages when the same package is specified multiple times. For instance, if both foo and foo <1 are listed, foo <1 is selected due to its specific version pin.
Platform-Specific Version Pinning: unidep resolves platform-specific dependency conflicts by preferring the version with the narrowest platform scope. For instance, given foo <3 # [linux64] and foo >1, it installs foo >1,<3 exclusively on Linux-64 and foo >1 on all other platforms.
Intractable Conflicts: When conflicts are irreconcilable (e.g., foo >1 vs. foo <1), unidep raises an exception.

Platform Selectors¶

This tool supports a range of platform selectors that allow for specific handling of dependencies based on the user’s operating system and architecture. This feature is particularly useful for managing conditional dependencies in diverse environments.

Supported Selectors¶

The following selectors are supported:

linux: For all Linux-based systems.
linux64: Specifically for 64-bit Linux systems.
aarch64: For Linux systems on ARM64 architectures.
ppc64le: For Linux on PowerPC 64-bit Little Endian architectures.
osx: For all macOS systems.
osx64: Specifically for 64-bit macOS systems.
arm64: For macOS systems on ARM64 architectures (Apple Silicon).
macos: An alternative to osx for macOS systems.
unix: A general selector for all UNIX-like systems (includes Linux and macOS).
win: For all Windows systems.
win64: Specifically for 64-bit Windows systems.

Usage¶

Selectors are used in requirements.yaml files to conditionally include dependencies based on the platform:

dependencies:
  - some-package >=1  # [unix]
  - another-package   # [win]
  - special-package   # [osx64]
  - pip: cirq         # [macos win]
    conda: cirq       # [linux]

Or when using pyproject.toml instead of requirements.yaml:

[tool.unidep]
dependencies = [
    "some-package >=1:unix",
    "another-package:win",
    "special-package:osx64",
    { pip = "cirq:macos win", conda = "cirq:linux" },
]

In this example:

some-package is included only in UNIX-like environments (Linux and macOS).
another-package is specific to Windows.
special-package is included only for 64-bit macOS systems.
cirq is managed by pip on macOS and Windows, and by conda on Linux. This demonstrates how you can specify different package managers for the same package based on the platform.

Note that the package-name:unix syntax can also be used in the requirements.yaml file, but the package-name # [unix] syntax is not supported in pyproject.toml.

Implementation¶

unidep parses these selectors and filters dependencies according to the platform where it’s being installed. It is also used for creating environment and lock files that are portable across different platforms, ensuring that each environment has the appropriate dependencies installed.

`[project.dependencies]` in `pyproject.toml` handling¶

The project_dependency_handling option in [tool.unidep] (in pyproject.toml) controls how dependencies listed in the standard [project.dependencies] section of pyproject.toml are handled when processed by unidep.

Modes:

ignore (default): Dependencies in [project.dependencies] are ignored by unidep.
same-name: Dependencies in [project.dependencies] are treated as dependencies with the same name for both Conda and Pip. They will be added to the dependencies list in [tool.unidep] under the assumption that the package name is the same for both package managers.
pip-only: Dependencies in [project.dependencies] are treated as pip-only dependencies. They will be added to the dependencies list in [tool.unidep] under the pip key.

Example pyproject.toml:

[build-system]
requires = ["hatchling", "unidep"]
build-backend = "hatchling.build"

[project]
name = "my-project"
version = "0.1.0"
dependencies = [  # These will be handled according to the `project_dependency_handling` option
  "requests",
  "pandas",
]

[tool.unidep]
project_dependency_handling = "same-name"  # Or "pip-only", "ignore"
dependencies = [
    {conda = "python-graphviz", pip = "graphivz"},
]

Notes:

The project_dependency_handling option only affects how dependencies from [project.dependencies] are processed. Dependencies directly listed under [tool.unidep.dependencies] are handled as before.
This feature is helpful for projects that are already using the standard [project.dependencies] field and want to integrate unidep without duplicating their dependency list.
The project_dependency_handling feature is only available when using pyproject.toml files. It is not supported in requirements.yaml files.

📝 requirements.yaml and pyproject.toml structure¶

Example¶

Example requirements.yaml¶

Example pyproject.toml¶

Key Points¶

Supported Version Pinnings¶

Conflict Resolution¶

How It Works¶

Platform Selectors¶

Supported Selectors¶

Usage¶

Implementation¶

[project.dependencies] in pyproject.toml handling¶

📝 `requirements.yaml` and `pyproject.toml` structure¶

Example `requirements.yaml`¶

Example `pyproject.toml`¶

`[project.dependencies]` in `pyproject.toml` handling¶