[RFC] Finish the backend package removal

[gadomski]

• Start Date: 2023-01-30

Summary

In Fall 2022, an effort began to remove the pgstac and sqlalchemy backends from the main stac-fastapi repository, and move them to their own repositories. This effort stalled in-progress, partially due to complexities that emerged while trying to decouple the API and backend layers. In order to complete the backend removal and restore the stac-fastapi ecosystem to a stable state, we will use a sequence of releases and refactors to move the backend packages from the core repository while ensuring minimal impact to downstream users.

Motivation

There are currently two "zombie" repositories (stac-fastapi-pgstac, stac-fastapi-sqlalchemy) in the stac-utils/stac-fastapi-* namespace. These repositories were created during the initial backend breakout efforts, but the breakout wasn't completed. Each repository has an unfinished PR that completes the backend breakout.

In the main stac-fastapi repo, there are a few draft PRs that relate to the breakout:

• Remove backend packages #450 is the main breakout PR
• Improve separation of concerns between core and backend libraries #453 and [WIP] Generate links in api layer #472 are spinoff PRs that attempted to do some decoupling between the API and the backend layers during the breakout

By finishing the reorganization in an orderly, low-impact way, we hopefully will resolve any confusion new users might have as to the state of the stac-fastapi project and its associated repos. Also, by intentionally not tackling the decoupling problem vis a vis the backend and API layers, we reduce the risk level of the shift, as we won't make many/any functional changes during the restructuring.

After the restructure, we hope to have a leaner stac-fastapi without any backend code, and two fully-functional backend repositories. This will give us a good, known state on which we can make future improvements to the API layer abstractions, etc.

Guide-level explanation

The breakout will be accomplished in the following order:

• Cut the next release (2.4.4) of stac-fastapi. This will require closing out any existing PRs and tickets in that milestone, and potentially bringing in any other bugfixes as appropriate.
• Rename default branch to main #189
• Re-create the git histories of stac-fastapi-pgstac and stac-fastapi-sqlalchemy using git filter-repo. This is a method that we used in stactools-packages with good success. A script, something like this, is run on a developer's machine: git filter-repo --path stac_fastapi/pgstac --path-rename stac_fastapi/pgstac/:/. I've created a repository to hold my scripts for doing this work: https://github.com/gadomski/stac-fastapi-breakout. This will keep all commits that relate to files in the pgstac backend, and move that subdirectory to the top level. This new git history will be pushed to the existing backend repositories as a non-default branch, e.g. new-main.
• For each backend repo, rename the default branch to old-main, then rename new-main to main. The git histories will now be reduced to only commits that relate to the that backend, but we keep the original breakout branch around just in case.
• Fixup each backend repository, using the normal pull request workflow, to get them building and testing against stac-fastapi v2.4.4 and well documented for the new layout. This should not require any functional code changes.
• Update the documentation settings on each new backend
• Create a new pull request in stac-fastapi that does the following:
  a. Removes both backend repos
  b. Updates all documentation to reference the backend repositories
  c. Closes any existing draft PRs associated with the original breakout effort via linked PRs
• Merge that PR
• Release v2.4.6 to mark the first release since the breakout. There shouldn't be any functional code changes to these packages.
• Once that PR is merged, migrate all backend-specific tickets from stac-fastapi to their new repos.

We will not be taking on any functional code changes as a part of this effort.

Drawbacks

By not doing any functional code changes, we are keeping the tight coupling between the API and backend layers. This will require close coordination between the backend repositories and the main stac-fastapi repo. While this is hard, I think it is less bad then leaving the repositories in the current undefined state.

Future possibilities

After the backend breakout, we can tackle the issues raised in #453 and elsewhere. I have another draft PR that presents an alternative approach: #515. However we do the separation of responsibilities, I think it will be a good opportunity to make breaking API changes as part of a v3.0 release.

[geospatial-jeff]

Thanks for throwing this together @gadomski! This all makes sense to me, and is pretty close to the original plan but without fixing the separation of concerns which ended up stalling out that effort as you mention. I don't have a ton of time right now to contribute but I can for sure help with code reviews, please feel free to add me as a reviewer.

[gadomski]

Cut the next release (2.4.4) of stac-fastapi. This will require closing out any existing PRs and tickets in that milestone, and potentially bringing in any other bugfixes as appropriate.

As an update, I've gone through the existing issue backlog and triages issues into the 2.4.4 milestone as appropriate. I tried to include anything that could be considered a "light to medium weight bugfix", and excluded anything that required design decisions, refactoring, etc. My hope is to fix as many bugs as possible that might touch the backend code before splitting, to reduce the number of simultaneous bugfix PRs we'll have to progress after the split.

[gadomski]

Update: DONE!