Poetry 2.0 removals and other breaking changes

[radoering]

This issue tracks possible removals and other breaking changes we want to include if the next release (or the one after that) will be a major release. (That does not mean that the changes listed here cannot be included in a minor release after a decent deprecation period.)

Please add a comment if you have concerns or would like to include another pending breaking change that is not on the list.

Background & Motivation

As support for PEP 621 is slowly taking shape (see #9135 to track the current state) and this will be a big, significant change (even if the old way will still be supported), this might be the time for a major version bump. Even if we try to avoid obvious breaking changes, just not ignoring the project section anymore will be a subtle breaking change for anyone who is already using this section (for whatever reason). Further, since supporting PEP 621 requires massive changes in many areas of the code, the risk of unintentional breaking changes is higher than usual.

Long story short: If we do a major version bump anyway, we should make use of it and remove things / add breaking changes we were holding back so far.

Candidates

Disclaimer: decision is not final and might be subject to change

change	decision	details
remove dependency on poetry-plugin-export	✅	#5980, although #5740 would be nice #9547 is sufficient
remove deprecated source types secondary and default	✅	results in significantly less complicated code and less complicated documentation
remove support of scripts with extras	✅	already done in python-poetry/poetry-core#708
change poetry add --optional to require an extra	✅	required for PEP 621 support (#9135)
fail if project cannot be installed	✅	#9333
remove poetry shell	✅	Will be moved to plugin
remove poetry self	⛔	#7872
make lock --no-update the default	✅	#9327
make install --sync the default	❓	introduce --keep-untracked or similar
remove deprecated command line options	❓
remove no-setuptools config option	✅	#9331
make include behavior more consistent	✅	#9691
...	...	...

[dimbleby]

maybe an opportunity to do something about poetry shell - see eg discussion in #4216

[dimbleby]

and also to take a view on poetry self #7872

[johnthagen]

A humble 🙏 that if poetry-plugin-export is removed by default, it's accompanied by something like

• How do I make part of and auto-enable plugins in my project #5729
• poc: isolated project plugins & tools #5740

This would then be much less disruptive for users of tools such as nox-poetry.

[Evan-Zhao]

(Sorry if this is not the right place to add an Issue on this version.)
I've been trying this new version out, and it's been working great overall!

One problem I noticed is whenever Poetry tries to add a [tool.poetry] section to a PEP 621-compliant pyproject.toml file, it erases everything after the point where it inserts the new section.

For example, I'm working on the following pyproject.toml here.
When I run poetry add --group plot matplotlib from the cmdline, Poetry will fall back to adding a [tool.poetry] section:

[tool.poetry.group.plot.dependencies]
matplotlib = "^3.8.3"

which makes sense because dep group is not a thing in 621.
However, this is added after [project.optional-dependencies], and Poetry also remove everything below that in this process -- the [tool.isort] and [tool.black] sections are lost.

[radoering]

Poetry also remove everything below that in this process -- the [tool.isort] and [tool.black] sections are lost.

Thanks for reporting this. I fixed it and added a test.

(Sorry if this is not the right place to add an Issue on this version.)

Since it's an open PR that is not yet merged, you can report issues directly in #9135.

[PabloAlexis611]

I think it'd be nice to also get rid of the deprecated --dev flag for the poetry add command since Poetry support for groups is already in place

[radoering]

Yes, we should take a look at all of the deprecated command line options.

[rmorshea]

Is there any chance that some of these changes could be enabled via feature flags?

I'm particularly interested in #3248 but this could be generally useful for easing the community into most of the listed breaking changes. Perhaps something like:

[tool.poetry.future]
lock-no-update-by-default = true

Of course doing this would mean maintaining two different code paths, but that's something the maintainers would have to weigh against the benefits of early feedback and (hopefully) a smoother transition.

[Secrus]

Is there any chance that some of these changes could be enabled via feature flags?

I'm particularly interested in #3248 but this could be generally useful for easing the community into most of the listed breaking changes. Perhaps something like:

[tool.poetry.future]
lock-no-update-by-default = true

Of course doing this would mean maintaining two different code paths, but that's something the maintainers would have to weigh against the benefits of early feedback and (hopefully) a smoother transition.

No. The reason for a major version bump is for us to have an opportunity to cleanup the code. Maintaining multiple code paths goes against that goal.

[trim21]

suggest to make install --sync/--remove-untracked default

[aureliojargas]

Looking at the changelog, we've had already some breaking changes in the configuration format:

• 1.0.0: Specifying dependencies with allows-prereleases in the pyproject.toml file is deprecated for consistency with the add command. Use allow-prereleases instead.
• 1.5.0: Introduce priority key for sources and deprecate flags default and secondary
• 1.8.0: Deprecate source priority default.

Here we're discussing some additional breaking changes on the configuration.

Would it make sense having some metadata (say, in [tool.poetry]) informing which specific Poetry version(s) the current configuration is intended for? Or having some versioned configuration schema? Otherwise it can be a guess work finding the Poetry version that works with a given configuration file.

[radoering]

suggest to make install --sync/--remove-untracked default

Although I mostly use --sync myself, I am not sure about it. (Waiting for other opinions.) I suppose, the current default could be named --keep-untracked if we decided to change the default.

Would it make sense having some metadata (say, in [tool.poetry]) informing which specific Poetry version(s) the current configuration is intended for?

There is #3316. I think we are open to adding a version constraint so the user can define compatible poetry versions.

Or having some versioned configuration schema?

I think that could cause more confusion than it helps respectively increases the maintainance burden. In the end, pyproject.toml is a file that is manually edited by users so users are responsible for specifying the correct version. If I remember correctly, there are some discussions on DPO why there is no version in pyproject.toml in general.

Otherwise it can be a guess work finding the Poetry version that works with a given configuration file.

At the moment, your best chance is the first line of poetry.lock.

[johnthagen]

suggest to make install --sync/--remove-untracked default

I don't have a strong opinion, but I do myself use --sync all the time and recommend to all of my teams they do the same. I think setting it as a default is probably less surprising (e.g., someone comes back to a project months later and has old things installed in the venv, the default poetry install doesn't remove them and get them to a clean slate).

I would lean towards making it the default because it feels more "pure" (e.g., by default Poetry will always make sure that you get the same environment as others on the same platform).

The counter would be if some users install "extra" things into their venv often and are frustrated that now Poetry uninstalls them. I'd call this an anti-pattern and they should be putting these things in groups to make things sound.

Overall, I would support making --sync the default.

[alecandido]

What's the motivation to exclude poetry shell removal from 2.0?

#4216 last comment dates 2022, and in any case in that discussion there was no proposal for a better replacement, and a seemingly common feeling against it (which I personally support...).

[radoering]

What's the motivation to exclude poetry shell removal from 2.0?

No consensus among maintainers. Some would like to remove it and some think it is important core functionality. In my experience, which of course is biased, power users do not use poetry shell but many beginners do and are happy with it.

A compromise might be to outsource the command into a plugin, especially with #9547.

[emilio-martinez]

@radoering please do consider moving it to a plugin 🙏 saying it having seen first-hand how much easier it is for engineers (with varying degrees of python comfort) to switch, onboard, and upgrade projects across an org a lot easier without having to reach for yet another tool. In other words, the ability to more efficiently hit the ground running is likely a good thing to have, no matter the skill level

[alecandido]

I'm not sure I properly understood the comment by @emilio-martinez, but moving to a plugin seems a more-than-decent solution.

To me, it's a bit confusing that poetry shell is working "half the way": people using it have the impression of something like the activate from virtualenv, while it's sufficiently different.

poetry run is much more straightforward, and possibly even script-friendly. So, I have the impression that having poetry shell present by default induces newcomers to think to that as the default way of using Poetry, while advanced-enough users tend to avoid it, creating a significant (and possibly painful) gap.

I'd pretty agree with @dimbleby and @Secrus (though they may have changed their mind in 2 years)
#4216 (comment)
#4216 (comment)

At the time, the main reason seemed to be avoiding a breaking change. And that's what Poetry 2.0 is for.

Maybe this is too much biased, but moving to a plugin would definitely give the flexibility to handle separately from the rest (possibly with less maintenance, if not deemed fundamental at a later stage).
Moreover, dropping poetry shell after 2.0 would require a 3.0. While reintroducing it is perfectly backward compatible...

[emilio-martinez]

@alecandido I said

please do consider moving it to a plugin

Emphasis in DO. So I think we're saying the same thing?

[Secrus]

I am currently working on moving the shell to a plugin and adding a replacement feature in Poetry. PRs should appear in a couple of days as I need to clean up some stuff.

[radoering]

I added #9691 to the list. It is "just" a bugfix but I suppose it will be considered a breaking change by users relying on the bug. Additionally, I am not sure yet in which direction we want to change it - it is not clear what is bug and what is correct behavior - so fixing it in a major release feels better than fixing it in a minor release.