Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

base-files: hardcoded cache of base-files version per RELEASE/ARCH, as of 20240522 #6653

Conversation

rpardini
Copy link
Member

base-files: hardcoded cache of base-files version per RELEASE/ARCH, as of 20240522

  • base-files: hardcoded cache of base-files version per RELEASE/ARCH, as of 20240522
    • packages.debian.org and packages.ubuntu.com are having a meltdown and builds are stuck
    • this is a quick & dirty fix to get builds going
    • we should urgently replace this with the 'right thing'
    • trying to maintain this manually will cause insanity

@rpardini rpardini marked this pull request as ready for review May 22, 2024 19:22
@rpardini rpardini requested a review from a team as a code owner May 22, 2024 19:22
@github-actions github-actions bot added the size/medium PR with more then 50 and less then 250 lines label May 22, 2024
@rpardini
Copy link
Member Author

not proud of 1) having done the curl version 2) replacing it with a hardcoded list 3) admins of packages.ubuntu.com 4) myself

@igorpecovnik
Copy link
Member

igorpecovnik commented May 22, 2024

Little time for proper way - mitigating the ugliness:

  • merging this only to v24.5 as this is urgent mainly for upcoming release? (and remove once done well)
  • making as extension?

@rpardini
Copy link
Member Author

rpardini commented May 22, 2024

this is too dirty to make into an extension.
the proper way will take a while, I'll be busy with smth else the next few days, so here's what we got.
merging to main should be safe; since versions are hardcoded & will be cached in OCI, it should work "forever" -- but we might miss updates published by Debian/Ubuntu.
Either way should buy us a few weeks to work proper way.

@amazingfate
Copy link
Contributor

amazingfate commented May 23, 2024

What about checking it from apt repo?

curl http://ports.ubuntu.com/ubuntu-ports/dists/noble/main/binary-arm64/Packages.xz|xz -d |grep "base-files/base-files"|grep "^Filename"|awk '{print $2}'
curl http://deb.debian.org/debian/dists/bookworm/main/binary-arm64/Packages.xz|xz -d |grep "base-files/base-files"|grep "^Filename"|awk '{print $2}'

@rpardini
Copy link
Member Author

rpardini commented May 23, 2024

What about checking it from apt repo?

curl http://ports.ubuntu.com/ubuntu-ports/dists/noble/main/binary-arm64/Packages.xz|xz -d |grep "base-files/base-files"|grep "^Filename"|awk '{print $2}'
curl http://deb.debian.org/debian/dists/bookworm/main/binary-arm64/Packages.xz|xz -d |grep "base-files/base-files"|grep "^Filename"|awk '{print $2}'

This approach I considered before I wrote the current "packages.debian.org" approach.
It works, but:

  • the Packages.xz download is significant (2mb for Ubuntu, 8mb for Debian)
  • the decompression is also a bit heavy
  • this needs to runs on every board build, which is fine for individuals, but CI runs 400+ of them

In comparison, back when it was working, the packages.xx.yy hit was small (few Kbs) and returned very fast.
Unfortunately ofc now packages.ubuntu.com is taking 10+ seconds to give a HTTP 500 error about 80% of the time, so that's down the drain.

The "proper way" I suggest is to use something similar to https://github.com/armbian/shallow -- a separate repo (with GHA action on a schedule) that gathers info for a given RELEASE + ARCH, massages it the way we need, and stores it in OCI/git/the matrix/wherever in a way that is easy for armbian/build to consume. Not only the base-files version, but also maybe simple info about every upstream package (so we can validate package lists / determine estimated size of rootfs's / generate a dependency tree etc).

@rpardini rpardini force-pushed the pr/base-files-hardcoded-cache-of-base-files-version-per-RELEASEARCH-as-of-20240522 branch 2 times, most recently from 26f1993 to f28be79 Compare June 3, 2024 06:51
@rpardini rpardini force-pushed the pr/base-files-hardcoded-cache-of-base-files-version-per-RELEASEARCH-as-of-20240522 branch from f28be79 to 84d90d8 Compare June 30, 2024 18:10
@github-actions github-actions bot added the Framework Framework components label Jun 30, 2024
@rpardini rpardini force-pushed the pr/base-files-hardcoded-cache-of-base-files-version-per-RELEASEARCH-as-of-20240522 branch from 84d90d8 to b25fa2f Compare July 7, 2024 11:08
…s of 20240522

- packages.debian.org and packages.ubuntu.com are having a meltdown and builds are stuck
- this is a quick & dirty fix to get builds going
- we should urgently replace this with the 'right thing'
- trying to maintain this manually will cause insanity
@rpardini rpardini force-pushed the pr/base-files-hardcoded-cache-of-base-files-version-per-RELEASEARCH-as-of-20240522 branch from b25fa2f to 8391938 Compare July 8, 2024 21:07
@rpardini
Copy link
Member Author

Replaced by #6978

@rpardini rpardini closed this Jul 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Framework Framework components size/medium PR with more then 50 and less then 250 lines
Development

Successfully merging this pull request may close these issues.

3 participants