Poll regarding some technical changes in linux-tkg PKGBUILD

[Tk-Glitch]

We are currently handling linux sources in linux-tkg PKGBUILD through tarballs. We are discussing with contributors on switching to using the git tree instead, similarly to what we're doing for non-pacman distros. This would allow for removing hundreds of lines from the PKGBUILD, and simplify the integrity checks handling by leaving it to git.

My main problem with this from past experience is that it's a large tree, and it takes quite a while to clone (even with shallow cloning) compared to just downloading and extracting the tarball. For people with high-bandwidth internet accesses and unlimited data plans it's not really an issue, but for the others without access to such luxuries it could be frustrating.

What do you think? Feel free to share your point of view and usecase (if it makes sense) so we can take the best decision.

[BillFleming]

slow clone bad

[AdelKS]

The first clone is slow, then every other update goes faster.

[Iglu47]

if this is only intended to solve the problem of integrity checks in the PKGBUILD, then it's probably better to keep the tarball yet

[AdelKS]

The git approach is already entirely implemented. And using git relieves a lot from the maintenance burden. Check my post below

[AdelKS]

Clarifications

So, obviously, the git and tar approaches both do the same thing. But let's get into details, with benchmark numbers

• Download compressed file(s)
  • Git: called fetching. The actual size downloaded by git is ~220MB. And this is the part that actually makes the git approach slower when downloading for the first time the kernel sources
    • There's an initial file preparation phase where the server looks for the relevant files to send (because our fetch is shallow), then compresses them before sending them. This part does not exist for tar and takes around ~10s.
    • The servers git downloads from upload at slower speeds (around 7MB/s for me). So using the git approach will actually affect more people with fast internet, which nuances/goes against what TKG said.
  • Tar: we simply download the tar.xz, takes around ~130MB. Faster than git (downloads around ~100MB/s for me). And that's what makes the Tar approach faster for people with fast internet
• Unpacking the kernel sources into a usable folder
  • Git: takes 2.5s
  • Tar: takes 5.5s
• Updating the kernel sources folder to a newer kernel version
  • Git: takes less than 1s. It only downloads the new files, then only updates/resets in-place the files that need to, in the kernel sources folder : no no need to delete the folder and this is great for SSDs. Using tmpfs is possible and is already implemented.
  • Tar: takes 5.5s. The tar approach always involves deleting the old kernel sources folder and unpacking the tar again. This is both slower and wastes writes for SSDs (if not using tmpfs, which is the case for everyone with less than 32GB of RAM)

What using only git would bring

Now with that in mind, git enables several maintenance improvements for us:

• We can entirely remove the need to version bump the latest kernel versions, as we can know that through git
  • No need to wait for bumps before users can build newer kernels (given that all the patches still apply properly): these lines become useless
  • If we force everyone to (shallow) git clone linux-tkg, the PKGBUILD can drop ENTIRELY EVERY SINGLE SHASUM LINE.
• SSD friendly for those who do not use tmpfs (anyone with less than 32GB ram)

Here's the code any of you can run to know how the two approaches work on your own computer

Click to expand

#!/bin/bash

msg2() {
 echo -e " \033[1;34m->\033[1;0m \033[1;1m$1\033[1;0m" >&2
}

# Other remotes can be chosen to test, can set "googlesource", "github" and "official"
_remote="official"

clean_git_fetch() {
    [[ -d packed-git-linux ]] && rm -rf packed-git-linux

    mkdir packed-git-linux
    pushd packed-git-linux
    git init

    git remote add googlesource https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable

    git remote add github https://github.com/gregkh/linux.git

    git remote add official https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

    git remote set-branches --add $_remote linux-5.17.y
    git fetch --depth=1 $_remote tag "v5.17.10"

    popd
}

create_git_wortree() {
    [[ -d linux-git-worktree ]] && rm -rf linux-git-worktree
    pushd packed-git-linux
    git worktree add -b linux-5.17-$_remote ../linux-git-worktree tags/v5.17.10
    popd
}

update_git_wortree() {
    pushd packed-git-linux
    git fetch --depth=1 $_remote tag "v5.17.11"
    popd

    pushd linux-git-worktree
    git checkout tags/v5.17.11
    popd
}

do_git_stuff() {
    time ( clean_git_fetch; msg2 "Git Fetching took" )
    time ( create_git_wortree; msg2 "Creating \"linux-git-worktree\" took" )
    time ( update_git_wortree; msg2 "Updating \"linux-git-worktree\" took" )
}


# ----------------- TAR CODE
download_tar() {
    [[ -f linux-5.17.tar.xz ]] && rm -rf linux-5.17.tar.xz
    wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.tar.xz
}

create_tar_worktree() {
    [[ -d linux-tar-worktree ]] && rm -rf linux-tar-worktree

    tar -xf linux-5.17.tar.xz
    mv linux-5.17 linux-tar-worktree

    [[ -f patch-5.17.11.xz ]] && rm -rf patch-5.17.11.xz
    [[ -f patch-5.17.11 ]] && rm -f patch-5.17.11

    wget https://cdn.kernel.org/pub/linux/kernel/v5.x/patch-5.17.11.xz
    unxz patch-5.17.11.xz

    pushd linux-tar-worktree
    patch --silent -Np1 -i ../patch-5.17.11

    popd
}

do_tar_stuff() {
    time ( download_tar; msg2 "Tar downloading took" )
    time ( create_tar_worktree; msg2 "Creating \"linux-tar-worktree\" took" )
}

time ( do_tar_stuff; msg2 "###############################"; msg2 "Tar total time" )

time ( do_git_stuff; msg2 "###############################"; msg2 "Git total time" )

And here's the output for me

Click to expand

``` adel@leyantoo /tmp [22:59:04] > $ ./git_vs_tar.sh --2022-05-25 22:59:12-- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.tar.xz Resolving cdn.kernel.org... 151.101.121.176, 2a04:4e42:1d::432 Connecting to cdn.kernel.org|151.101.121.176|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 128399340 (122M) [application/x-xz] Saving to: ‘linux-5.17.tar.xz’

linux-5.17.tar.xz 100%[===================>] 122.45M 99.7MB/s in 1.2s

2022-05-25 22:59:13 (99.7 MB/s) - ‘linux-5.17.tar.xz’ saved [128399340/128399340]

-> Tar downloading took

real 0m1.292s
user 0m0.048s
sys 0m0.110s
--2022-05-25 22:59:18-- https://cdn.kernel.org/pub/linux/kernel/v5.x/patch-5.17.11.xz
Resolving cdn.kernel.org... 151.101.121.176, 2a04:4e42:1d::432
Connecting to cdn.kernel.org|151.101.121.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 748028 (730K) [application/x-xz]
Saving to: ‘patch-5.17.11.xz’

patch-5.17.11.xz 100%[===================>] 730.50K --.-KB/s in 0.02s

2022-05-25 22:59:18 (40.8 MB/s) - ‘patch-5.17.11.xz’ saved [748028/748028]

/tmp/linux-tar-worktree /tmp
/tmp
-> Creating "linux-tar-worktree" took

real 0m5.571s
user 0m5.455s
sys 0m0.848s
-> ###############################
-> Tar total time

real 0m6.863s
user 0m5.504s
sys 0m0.958s
/tmp/packed-git-linux /tmp
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m
Initialized empty Git repository in /tmp/packed-git-linux/.git/
remote: Enumerating objects: 79594, done.
remote: Counting objects: 100% (79594/79594), done.
remote: Compressing objects: 100% (77637/77637), done.
remote: Total 79594 (delta 5579), reused 15983 (delta 1135), pack-reused 0
Receiving objects: 100% (79594/79594), 215.86 MiB | 7.68 MiB/s, done.
Resolving deltas: 100% (5579/5579), done.
From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux

• [new tag] v5.17.10 -> v5.17.10
  /tmp
  -> Git Fetching took

real 0m38.092s
user 0m6.744s
sys 0m1.068s
/tmp/packed-git-linux /tmp
Preparing worktree (new branch 'linux-5.17-official')
Updating files: 100% (75039/75039), done.
HEAD is now at 6c468bb6f Linux 5.17.10
/tmp
-> Creating "linux-git-worktree" took

real 0m2.587s
user 0m2.230s
sys 0m0.346s
/tmp/packed-git-linux /tmp
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 8 (delta 6), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (8/8), 1.58 KiB | 806.00 KiB/s, done.
From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux

• [new tag] v5.17.11 -> v5.17.11
  /tmp
  /tmp/linux-git-worktree /tmp
  Note: switching to 'tags/v5.17.11'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at e960d7349 Linux 5.17.11
/tmp
-> Updating "linux-git-worktree" took

real 0m0.708s
user 0m0.361s
sys 0m0.154s
-> ###############################
-> Git total time

real 0m41.387s
user 0m9.336s
sys 0m1.568s

</details>

[Iglu47]

output

--2022-05-26 00:37:31--  https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.tar.xz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.37.176, 2a04:4e42:9::432
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.37.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 128399340 (122M) [application/x-xz]
Saving to: 'linux-5.17.tar.xz'

linux-5.17.tar.xz                                           100%[==>] 122.45M  11.2MB/s    in 11s     

2022-05-26 00:37:42 (11.0 MB/s) - 'linux-5.17.tar.xz' saved [128399340/128399340]

 -> Tar downloading took

real	0m11.426s
user	0m0.227s
sys	0m0.436s
--2022-05-26 00:38:27--  https://cdn.kernel.org/pub/linux/kernel/v5.x/patch-5.17.11.xz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.37.176, 2a04:4e42:9::432
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.37.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 748028 (730K) [application/x-xz]
Saving to: 'patch-5.17.11.xz'

patch-5.17.11.xz                                            100%[==>] 730.50K  2.80MB/s    in 0.3s    

2022-05-26 00:38:27 (2.80 MB/s) - 'patch-5.17.11.xz' saved [748028/748028]

~/pr-wine/look/test/linux-tar-worktree ~/pr-wine/look/test
~/pr-wine/look/test
 -> Creating "linux-tar-worktree" took

real	0m46.326s
user	0m7.704s
sys	0m2.297s
 -> ###############################
 -> Tar total time

real	0m57.752s
user	0m7.932s
sys	0m2.734s
~/pr-wine/look/test/packed-git-linux ~/pr-wine/look/test
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /home/i47/pr-wine/look/test/packed-git-linux/.git/
remote: Enumerating objects: 79594, done.
remote: Counting objects: 100% (79594/79594), done.
remote: Compressing objects: 100% (77637/77637), done.
remote: Total 79594 (delta 5575), reused 15983 (delta 1135), pack-reused 0
Receiving objects: 100% (79594/79594), 215.87 MiB | 6.15 MiB/s, done.
Resolving deltas: 100% (5575/5575), done.
From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux
 * [new tag]             v5.17.10   -> v5.17.10
~/pr-wine/look/test
 -> Git Fetching took

real	0m45.909s
user	0m8.428s
sys	0m1.541s
~/pr-wine/look/test/packed-git-linux ~/pr-wine/look/test
Preparing worktree (new branch 'linux-5.17-official')
Updating files: 100% (75039/75039), done.
HEAD is now at 6c468bb6f Linux 5.17.10
~/pr-wine/look/test
 -> Creating "linux-git-worktree" took

real	0m39.796s
user	0m3.395s
sys	0m1.542s
~/pr-wine/look/test/packed-git-linux ~/pr-wine/look/test
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 8 (delta 6), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (8/8), 1.58 KiB | 537.00 KiB/s, done.
From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux
 * [new tag]             v5.17.11   -> v5.17.11
~/pr-wine/look/test
~/pr-wine/look/test/linux-git-worktree ~/pr-wine/look/test
Note: switching to 'tags/v5.17.11'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at e960d7349 Linux 5.17.11
~/pr-wine/look/test
 -> Updating "linux-git-worktree" took

real	0m0.979s
user	0m0.131s
sys	0m0.135s
 -> ###############################
 -> Git total time

real	1m26.685s
user	0m11.956s
sys	0m3.217s

[AdelKS]

Git makes it faster for you, as updating the work tree took only a second !

[vl-ms]

Do both. Just choose what to use on the first download (the slowest one by far) and then update with git.

[oxyzenQ]

new method, for speed🚀