[Task / Epic]: HUGE repository cleanup ✨

[ColorfulRhino]

Task description

The Problem

Over the years, some older stuff was partially removed/not used anymore, but not fully cleaned up. Those files and leftovers in the code still live in the repository, leading to confusions (what is this? is this still used? can this be deleted?) and misleading search/grep results (e.g. for packages/extras-buildpkgs/htop or packages/extras-buildpkgs/hostapd which included changelogs and therefore lots of unrelated text).

Besides that, many blobs were added to the build repo, only some of them still remain. But even the deleted blobs still remain in the repo since Git saves all the history: The history size is huge, even though the current size of the packages/blobs folder is only 55MB. This leads to a unnecesarily bloated repository, increasing the size for everybody.
I remember one person often having to visit a local library or university to download/update their Armbian repo since the size was too large on their slow or resticted internet connection at home.

For comparison:

• Size of the whole U-Boot repository (since 2005!): ~ 228 MB
• Size of the current Armbian/build repository: ~ 636 MB

I don't believe that Armbian/build is a bigger project than U-Boot, but it is amlost triple the size in MB. Vastly reducing the repo size (TODO: calculate actual size before/after blob purge) will make contributions more inclusive overall and save time on many occasions.

The Solutions

Removing all known leftover code and moving all blobs to a separate blob repository, like already done with the Rockchip blobs in the Armbian/rkbin repo. After this is done, purge the build repository's history from all the blobs (original idea by @rpardini I believe). The goal is to have a completely blobless Arbian/build repo while blobs are only pulled from other repositories.

Task List

Leftover code:

• Remove unused packages/extras-buildpkgs/hostapd plus its Realtek part and its traces [Cleanup] Clean leftovers in packages/extras-buildpkgs #6830
• Remove unused packages/extras-buildpkgs/htop and its traces --> same PR
• Remove unused packages/extras-buildpkgs/sunxi-tools and its traces --> same PR
• Remove old compiler torrents from config/torrents/ [Cleanup] Remove old torrents #6829
• Remove 'external-toolchains' code path
• [Task / Story]: Clean up old kernels/kernel patch leftovers #6821
• Check for other leftover stuff in the code (HELP NEEDED: Please let me know or add to this task list if you know of any unused stuff 😄)

Blobs:

• Create new repository for blobs (e.g. Armbian/blobs)
• Copy all existing blobs to the new blob repo
• Change all references in the code to point to the new repo for the blobs
• Remove all existing blobs from the Armbian/build repo
• Purge all blobs from the Armbian/build Git history by rewriting history (this is where the actual size reduction happens)
• Add an Actions workflow to check new PRs for blobs and if detected, kindly auto-remind them to commit the blobs to the blob repo instead of the build repo

This task list will be extended with new findings. PRs solving specific tasks will be linked.

This task/story is open for ideas and discussions! 😄

---

Some statistics for fun and to compare the impact of this cleanup:

	Before	After	Difference
Lines of code	6 458 280	TBD
# of files	6503	TBD
Repo size	~ 636 MB	TBD

Commands used:

• Lines of code: git diff --shortstat 4b825dc642cb6eb9a060e54bf8d69288fbee4904
• # of files: git ls-files | wc -l
• Repo size: Queried on GitHub

[github-actions]

Jira ticket: AR-2391

[The-going]

• Remove unused packages/extras-buildpkgs/hostapd plus its Realtek part and its traces
• Remove unused packages/extras-buildpkgs/htop and its traces
• Remove unused packages/extras-buildpkgs/sunxi-tools and its traces

This code is not used by the build system. And can be deleted.
The last minutes of the life of the functions that did something you can see:
armbian/build> gitk -- lib/functions/extras

This functionality was designed to build packages in the native "chroot" environment
that required library dependencies from the environment.

I am currently still using this, and will be able to bring the code back if users want to use it.

[ColorfulRhino]

This code is not used by the build system. And can be deleted.

Thanks for confirming this!

The last minutes of the life of the functions that did something you can see:
armbian/build> gitk -- lib/functions/extras

My build host does not have a graphical user interface, but I get what you mean 😄

If you know of any other unused code, let me know and I'll add it to the list :)

[The-going]

The last minutes of the life of the functions that did something you can see:
armbian/build> gitk -- lib/functions/extras

My build host does not have a graphical user interface, but I get what you mean 😄

git log -p -- lib/functions/extras

If you know of any other unused code, let me know and I'll add it to the list

Unfortunately, I'm still in the old paradigm 1.5 years ago.

[rpardini]

I fully agree with the cleanup, but keep in mind git's history will be unaffected, and thus the repo size will only ever become bigger, not smaller. We'd need to rebase things out of existence (rewrite history) and force-push to actually make it smaller. See git filter-branch and git-filter-repo for possible approaches, but it would be very impacting.

[ColorfulRhino]

We'd need to rebase things out of existence (rewrite history) and force-push to actually make it smaller. See git filter-branch and git-filter-repo for possible approaches, but it would be very impacting.

Yes, I think this is what you meant when you were talking about this a long while ago. Please correct me if I'm wrong 😅

My plan is to test this in a completely separate repository (and in a second stage in the main repo but a separate branch), trying to understand its impact and getting opinions of multiple people. This will definitely have to be approved by more than one or two people 😄

The plan is to use one of those tools for this:

• bfg-repo-cleaner
• git-filter-repo

The thing is, if we want to do this, the best time is sooner rather than in 3 or 5 years.

[rpardini]

Awesome. You've full understanding.