Use uv for Python tasks

[halostatue]

Python packaging is in flux, but the old form of requirements (requirements.txt) is no longer recommended. The use of a more modern alternative (such as Poetry, PDM, Rye, or uv) is recommended as they also provide dependency lock files so that downstream dependencies are not updated on installation.

For a project that is not written in Python, uv looks likely to be the best choice because it can also manage the Python installation, using a suitable existing Python installation if present.

As uv does not currently support running tasks (unlike Poetry and pdm), we have added taskipy with useful shorthand tasks defined for this purpose:

name	command
build-docs	mkdocs build -f assets/chezmoi.io/mkdocs.yml
serve-docs	mkdocs serve -f assets/chezmoi.io/mkdocs.yml
deploy-docs	cd assets/chezmoi.io && mkdocs gh-deploy
lint	ruff check
format	ruff format
pycheck	task lint && task format --diff
pyfix	task lint --fix && task format
format-yaml	make format-yaml

These are run with uv run task <name>. Note that make format-yaml is the preferred way to format YAML documents, and it runs the format-yaml.py script directly.

During development of this, it was discovered that the YAML files in the repo have not been formatted recently, so this was done as a separate commit at the head of this branch.

[bradenhilton]

I actually have a local branch where I switched to Hatch, but I'm not happy with it yet. Is that something you considered?

[halostatue]

I actually have a local branch where I switched to Hatch, but I'm not happy with it yet. Is that something you considered?

My local branch (which had been intermingled with other changes, so I pulled this out) was on PDM prior to this proposal.

I’ve tried Poetry, PDM, Hatch, and now uv (I haven't tried Rye for this, but this is by the same team that does Rye). And, of course, the manual way. Poetry is the oldest and most mature, but also doesn't fully conform to the packaging PEPs, and I found Hatch nearly unusable. PDM is currently nicer than uv (because it includes task running directly, see astral-sh/uv#5903), and offers a GitHub setup action as well.

uv is the designated successor to Rye, but Rye is still being developed as uv improves (currently rye uses parts of uv for package resolution, but the original developer of rye is now working on uv as well).

There are some really interesting things with respect to uv tool where one could do something like:

$ uvx --with mkdocs-material==9.5.34 --with mkdocs-mermaid2-plugin==1.1.1 \
    --with pymdown-extensions==10.10.1 [email protected]
# or
$ uv tool install [email protected] --with mkdocs-material==9.5.34 \
    --with mkdocs-mermaid2-plugin==1.1.1 \
    --with pymdown-extensions==10.10.1
$ uvx mkdocs

That replaces pipx (which could also be used for documentation building, leaving only the format-yaml script to be handled).

The format-yaml script can also be run with uv run without pyproject.toml by adding a script dependency.

I chose not to go that direction for several reasons:

1. The tools and script dependency approach maintains versions completely independently, and cannot be tracked by Dependabot or other similar tools.
2. The tools approach is much more like pipx, intended for long-term installation and may result in people running outdated versions rather than the locked versions.
3. The mechanisms here are better for CI (IMO).

[bradenhilton]

All valid points.

I don't think there is much value in setting uv run task format-yaml to make format-yaml, as the Makefile is not platform agnostic. I think it would be better to have make format-yaml invoke format-yaml.py (via uv if needed) and add a simple arg parser to format-yaml.py so it can explicitly accept file paths, or recurse through the working tree and accumulate all YAML file paths if no args are provided.

I was going to add it to this PR, but I don't want to complicate anything yet, so I think I'll just write it here for now.

format-yaml.py

#!/usr/bin/env python3

from __future__ import annotations

import argparse
import subprocess
from pathlib import Path
from typing import Iterable

from ruamel.yaml import YAML


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        prog='format-yaml.py',
        description='Format YAML files with ruamel.yaml.',
    )
    parser.add_argument(
        'filepaths',
        nargs='*',
        type=str,
        help='Path to a YAML file',
        metavar='YAML_FILE',
    )

    args = parser.parse_args()
    args.filepaths = [Path(filepath).resolve() for filepath in args.filepaths]

    return args


def format_yaml(filepaths: Iterable[Path]) -> None:
    yaml = YAML()
    # ruamel.yaml.YAML will by default use the native line ending, which leads
    # to differences between Windows and UNIX. Force the output to use UNIX line
    # endings.
    newline = '\n'
    yaml.line_break = newline
    # ruamel.yaml.YAML will by default break long lines and introduce trailing
    # whitespace errors. Disable this behavior by setting a long line width.
    yaml.width = 1024
    for filepath in filepaths:
        with filepath.open('r') as file:
            data = yaml.load(file)
        with filepath.open('w', newline=newline) as file:
            yaml.dump(data, file)


def main() -> None:
    args = parse_args()
    filepaths = args.filepaths
    if not filepaths:
        root = Path(
            subprocess.run(
                ['git', 'rev-parse', '--show-toplevel'],
                text=True,
                check=True,
                capture_output=True,
                encoding='utf-8',
                errors='replace',
            ).stdout.strip(),
        )
        filepaths = [
            filepath
            for pattern in ('*.yml', '*.yaml')
            for filepath in root.rglob(pattern, case_sensitive=False)
        ]
    format_yaml(filepaths)


if __name__ == '__main__':
    raise SystemExit(main())

[halostatue]

I can tweak the format-yaml bits to have the dependencies go one way, and then we can revisit the format-yaml script in a different PR.

[twpayne]

This looks fantastic, thank you!

[halostatue]

I’m tracking the task management issue previously mentioned for uv; when it is implemented, I’ll transition the tasks off taskipy and into native uv tasks. (I will probably do so in stages so that uv run task … doesn't stop working immediately, but warnings will be printed for a while so that people with shell history can start using the new commands.)

[halostatue]

I’ll look at merging this tomorrow as I’m not around to watch it if something breaks on merge. Because the docs push commands have been updated, we should be aware of this for the next release but I don't think anything will break.

[twpayne]

Because the docs push commands have been updated, we should be aware of this for the next release but I don't think anything will break.

I just tagged v2.52.3 and it looks like everything worked. chezmoi.io was correctly automatically updated.

[halostatue]

I love it when everything works like it's supposed to.