[WIP] Edit User Guide to offer more information about usability and use cases

[willingc]

Addresses #367

• Update the main index page with a note box that describes organization of the user guide
• Add brief guidance for installation only users as well as intermediate and advanced users which to create packages
• Add a Basics of Installation page which serves a similar purpose for installation as the Overview page which focuses on creating packages.
• Move Creating Documentation page from tutorials to guides since seems more related to performing a task than a tutorial on packaging concepts.
• Add details to Basics of Installation
• Edit page on Scientific installations
• Add setup.py info from @pganssle thread

FYI @dstufft

[willingc]

Folks, feel free to add notes in this Hackmd pad: https://hackmd.io/@Y6xjRiXFRUmwV7lDeM-5nQ/SkvWiBfzB

[pradyunsg]

@willingc would it be fine to provide feedback via inline comments here?

(Asking since it's marked WIP and sometimes folks prefer that specific feedback wait until WIP is dropped)

[willingc]

Completely up to you @pradyunsg. Feedback welcome. It may be best at some point to split the PR up too.

[Carreau]

I don't think write->substitute is the correct change here. You want users to keep {sys.executable}.

[willingc]

Yeah, I agree and will reword. Probably add a link to the Python docs for sys.executable.

[Carreau]

One comment I'm not sure fits here or in the hackpad is:

• If we want to replace python setup.py install|develop, do we still want to mention it in the docs, or (almost) completely remove any mentions ? Telling users "do not do <x>" has kind of the Streisand effect of making them remember <x>.

In the IPython docs we fought a long time to remove mentions of p-y-l-a-b inline, and the only effective things we did was just to completely undocument it.

[pganssle]

@Carreau

• If we want to replace python setup.py install|develop, do we still want to mention it in the docs, or (almost) completely remove any mentions ? Telling users "do not do <x>" has kind of the Streisand effect of making them remember <x>.

This is a good point, but I am not sure that we have any current documentation out there now that says you should do this, but people still somehow know to do it, so I think it is worth addressing. Maybe we can reference setup.py commands in general rather than any specific commands? Something like: "The packaging ecosystem is moving away from direct invocations of setup.py in favor of dedicated CLI tools for end users. As such, note that instructions you come across that advocate for the use of setup.py commands are likely out of date."

We can probably provide a list of potential replacements without actually mentioning the original commands by describing functions, like:

While there is not a single workflow that works best for everyone, here are some commands that are most common and best supported in modern workflows:

• package installation: pip install
• editable installation (development mode): pip install -e
• running tests: tox, pytest
• uploading packages: twine
• building packages: python -m pep517.build

I don't mean to suggest that the above be the actual text - there is actually a lot more nuance to convey that can't easily be contained in a table of "commands to use", but this is just as an example to avoid generating a "conversion guide" that ends up getting people to use the old commands.

[pganssle]

I do not particularly want to court controversy here, but I think we may want to be very careful about the degree to which pipenv is included or featured in this documentation. There has already been a lot of confusion sewn about how "official" pipenv is.

I think putting pipenv before pip in this sentence will unfortunately be seized upon by the sort of "kremlinologists" who may scrutinize documentation in some sort of legalistic manner for the "word from on high".

[willingc]

I can flip them.

[willingc]

FYI. The basics doc is still notes and not even to draft state yet. 😉

[ncoghlan]

I'd phrase this a bit differently:

• if using an environment manager like conda (with appropriate link) or pipenv (link to app dependency management tutorial), then most packages will be installed through the environment manager, regardless of the instructions given in the individual package documentation
• if not using an environment manager, then most packages will be installed directly with pip, unless the individual package documentation explicitly says otherwise (and even then, if the package documentation mentions running setup.py directly, it is likely just outdated information).

As much as I'd like to be able to tell users "trust the documentation of the packages you're using", we unfortunately can't do that:

• if using conda/pipenv/poetry, bypassing them can result in an inconsistent environment
• a lot of up to date documentation will say "call pip"
• some Linux-specific documentation will say "call pip3"
• some conda-specific documentation will say "call conda"
• some old documentation will say "call setup.py"

[willingc]

I'm inclined to have things that are more specific and perhaps not yet universal recommendations into the scientific computing doc page.

[pradyunsg]

It may be best at some point to split the PR up too.

I agree. I feel we should definitely make things like renaming guide/discussion and adding a basics of installation page, be added in separate PRs.

[Carreau]

"The packaging ecosystem is moving away from direct invocations of setup.py in favor of dedicated CLI tools for end users. As such, note that instructions you come across that advocate for the use of setup.py commands are likely out of date."

Yes that sounds great; I am doing similar things, I just want to avoid mentioning the setup.py command directly.

[willingc]

@pradyunsg I will split out into two more WIP PRs.

[pradyunsg]

Thank you @willingc! :D

[ncoghlan]

I like the direction this is going. One specific note on the challenges/risks of telling uses to trust the package documentation over this guide :(

[ncoghlan]

I'd phrase this a bit differently:

• if using an environment manager like conda (with appropriate link) or pipenv (link to app dependency management tutorial), then most packages will be installed through the environment manager, regardless of the instructions given in the individual package documentation
• if not using an environment manager, then most packages will be installed directly with pip, unless the individual package documentation explicitly says otherwise (and even then, if the package documentation mentions running setup.py directly, it is likely just outdated information).

As much as I'd like to be able to tell users "trust the documentation of the packages you're using", we unfortunately can't do that:

• if using conda/pipenv/poetry, bypassing them can result in an inconsistent environment
• a lot of up to date documentation will say "call pip"
• some Linux-specific documentation will say "call pip3"
• some conda-specific documentation will say "call conda"
• some old documentation will say "call setup.py"

[pradyunsg]

A gentle ping on this.

[willingc]

@pradyunsg Sorry about dropping the ball on this one. I have given you push access to my branch. Feel free to take over any open packaging PRs of mine. Thanks!