Distro-specific pre-processing in 1_prune.sh can be avoided by using ostree checkout

[prydom]

I bumped in your project when I spotted the blue-build/cli#217 issue and was surprised to see someone else had come up with the same solutions to many of the problems with custom ostree overlays (e.g. replacing kernels, SELinux labeling, improve layer reuse) independently.

I've been using the same techniques since April and noticed a few differences in how we do things. The primary difference is that I use bluebuild to apply workarounds (e.g. /usr/etc merge and /etc/passwd truncation after overlay) prior to re-chunking and don't need to do any of the permissions rewrites you do due to using podman mount to access the final merged ostree image.

You can see my workarounds here: https://github.com/prydom/my-ostree-build/blob/ed2b05b29bac9dc0152862f198f166b63380c044/recipes/fedora-kinoite-laptop.yml#L199-L215. I have had no booting problems.

The key insight I think you're missing is that ostree-ext-rs does most of what the 1_prune.sh script does by applying permissions captured into the ostree metadata during the initial ostree commit. By checking it out (using hardlinks mean we don't waste space) using ostree init && ostree checkout you avoid having to do the prune permissions steps which hardcode assumptions about how the distro applies permissions. See https://github.com/prydom/my-ostree-build/blob/ed2b05b29bac9dc0152862f198f166b63380c044/.github/workflows/build.yml#L63-L157 for my full solution.

[antheas]

But the question I have for you now is do you get a space reduction if you do that? I think not.

E.g., prune cuts around 1GB of space from the bazzite-deck image and at the same time relabels it correctly.

[antheas]

I know that due to the way I do it directory quirks are lost, but that is just due to the fact ostree-rs-ext throws away permissions when exporting the files, See ostreedev/ostree-rs-ext#655

[antheas]

But you are right I did not know about ostree container pull. From my understanding ostree-rs-ext did not support pulling derived images.

[prydom]

ostree checkout does not include the /ostree directory as it is the same directory that is generated when you apply the image before booting.

For example, I replace the kernel in my image and after rechunking the space is reduced as expected.

[prydom]

But you are right I did not know about ostree container pull. From my understanding ostree-rs-ext did not support pulling derived images.

It uses the same logic as bootc and rpm-ostree so it should be safe to use instead of podman mounts and you will save disk space on your builder.

[prydom]

I know that due to the way I do it directory quirks are lost, but that is just due to the fact ostree-rs-ext throws away permissions when exporting the files, See ostreedev/ostree-rs-ext#655

Ah from that issue I see you know about bare-split-xattrs, that's what I'm talking about when I say "applying permissions captured into the ostree metadata during the initial ostree commit", I think you may find this resoves the xattr problem too but I'll test to double check later and report back.

[antheas]

ostree checkout does not include the /ostree directory as it is the same directory that is generated when you apply the image before booting.

For example, I replace the kernel in my image and after rechunking the space is reduced as expected.

Indeed.

But you are right I did not know about ostree container pull. From my understanding ostree-rs-ext did not support pulling derived images.

It uses the same logic as bootc and rpm-ostree so it should be safe to use instead of podman mounts and you will save disk space on your builder.

If you export the image to files like you do in the link you said you need to maintain the image + the ostree commit at the same time. So you need 2X the space. If you need 2X the space, you are not saving space by checking out. Also, you only save space if you are not modifying the image and pulling it into podman.

Pulling into ostree and checking out takes a lot of time. Podman mounts are instant. You have a bit of overhead because you need to nix the /sysroot/ostree dir sure.

I know that due to the way I do it directory quirks are lost, but that is just due to the fact ostree-rs-ext throws away permissions when exporting the files, See ostreedev/ostree-rs-ext#655

Ah from that issue I see you know about bare-split-xattrs, that's what I'm talking about when I say "applying permissions captured into the ostree metadata during the initial ostree commit", I think you may find this resoves the xattr problem too but I'll test to double check later and report back.

The issue is still open.

I mean it is a mixed bag. If you checkout with OSTree you maintain the original layer attrs and lose the derived ones. If you mount you maintain the derived ones and lose the originals.

As for the quantitative difference between the two approaches, rechunk allows you to use the full SELinux policy of the image so you get no systemd relabelling quirks in the end, and add xattrs to derived layers that are otherwise lost. Using ostree checkout maintains the original image attrs but loses the derived ones.

Which is better? Well for our usecase the derived quirks are a lot more so that makes sense. Kinoite ones are much fewer. If you are only adding a couple of files to the image then yeah you'd prefer the OSTree approach.

[prydom]

The issue is still open.

I have confirmed that after booting a re-chunked image that the xattrs and capabilities are preserved if you use a checkout && commit && contaner-encapsulate method to relabel and rechunk. Please try it, you don't have to take my word that it works fine.

❯ getcap /usr/bin/newuidmap /usr/bin/newgidmap /usr/bin/kwin_wayland
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep
/usr/bin/kwin_wayland cap_sys_nice=ep

If you export the image to files like you do in the link you said you need to maintain the image + the ostree commit at the same time. So you need 2X the space. If you need 2X the space, you are not saving space by checking out. Also, you only save space if you are not modifying the image and pulling it into podman.

This is true. I run in a new Github actions runner so I don't have the podman checkout anymore. The space saving I'm talking about is copying the image files out of the mount rather then hard-linking out of the ostree repo. You are correct I'm making a time-space tradeoff here.

[prydom]

lose the derived ones

Ah I did miss this. Do you know a good package I can install to test this. Otherwise I'll test this manually.
Just note I'm not saying you need to stop quirk'ing - I will be doing that in my personal overlay too. We can do both, quirk and preserve the original xattrs.

My concern is that bluebuild may look into integrating this repo and if we lose the base xattrs and other permissions or need to hardcode them then this isn't a suitable generic solution. Bluebuild works (for the most part - you need to add your own modules for it to work) with any distro compatible with ostree.

[antheas]

The issue is still open.

I have confirmed that after booting a re-chunked image that the xattrs and capabilities are preserved if you use a checkout && commit && contaner-encapsulate method to relabel and rechunk. Please try it, you don't have to take my word that it works fine.

❯ getcap /usr/bin/newuidmap /usr/bin/newgidmap /usr/bin/kwin_wayland
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep
/usr/bin/kwin_wayland cap_sys_nice=ep

You have the original attrs that are in the commit layers and the attrs that were added afterwards in the OCI layers. With checkout, you maintain the original and lose the derived. With mount, you maintain the derived and lose the original.

If you export the image to files like you do in the link you said you need to maintain the image + the ostree commit at the same time. So you need 2X the space. If you need 2X the space, you are not saving space by checking out. Also, you only save space if you are not modifying the image and pulling it into podman.

This is true. I run in a new Github actions runner so I don't have the podman checkout anymore. The space saving I'm talking about is copying the image files out of the mount rather then hard-linking out of the ostree repo. You are correct I'm making a time-space tradeoff here.

Rechunk is meant to run at the end of an OCI action. In that action, you have the image once in the image store, and you copy the files to an ostree repo. Therefore, it is the same as your example. The exception is if you do not modify the image beforehand. If you do not modify it, then yes, having to put it into podman and then ostree is an additional copy.

Again, different use cases. We use OCI workflows, not bluebuild which is ostree based. In your case since you check out into a dir and then recommit, you also get none of the xattr issues that happen

lose the derived ones

Ah I did miss this. Do you know a good package I can install to test this. Otherwise I'll test this manually. Just note I'm not saying you need to stop quirk'ing - I will be doing that in my personal overlay too. We can do both, quirk and preserve the original xattrs.

My concern is that bluebuild may look into integrating this repo and if we lose the base xattrs and other permissions or need to hardcode them then this isn't a suitable generic solution. Bluebuild works (for the most part - you need to add your own modules for it to work) with any distro compatible with ostree.

Rechunk and the algorithm behind it are not dependent on the prune script. Yes in your case it makes more sense to skip those as they are for OCI workflows.

You could instead PR me a version of the action with a var that receives a directory instead of a reference and skips the steps below.

rechunk/action.yml

Lines 122 to 140 in 496f4b8

	- name: Mount Image
	id: mount
	shell: bash
	run: |
	CREF=$(sudo podman create ${{ inputs.ref }} bash)
	MOUNT=$(sudo podman mount $CREF)
	echo "container=$CREF" >> $GITHUB_OUTPUT
	echo "mount=$MOUNT" >> $GITHUB_OUTPUT
	- name: Create OSTree Commit
	shell: bash
	run: |
	# Pruning
	echo Pruning
	sudo podman run --rm \
	-v "${{ steps.mount.outputs.mount }}":/var/tree \
	-e TREE=/var/tree \
	-u 0:0 \
	${{ inputs.rechunk }} \
	/sources/rechunk/1_prune.sh

As for a good package, gamescope has eip_nice.

[antheas]

But if you install the package on the checked out dir you will not have that issue. Only xattrs in derived container layers are lost.

[prydom]

But if you install the package on the checked out dir you will not have that issue. Only xattrs in derived container layers are lost.

I do use an OCI workflow to apply delta updates (https://github.com/prydom/my-ostree-build/blob/ed2b05b29bac9dc0152862f198f166b63380c044/recipes/Containerfile.fedora-kinoite-laptop-update-snapshot) - the re-chunk is not my final step so this behavior does concern me too. It probably just something I never noticed and now I indeed want it fixed if it reproduces.

I'll test it and confirm then look into a PR for both projects.
Thank you so much for your time and ideas!

Btw, my goal here is to cross-pollinate. I'm just just a consumer and occasional contributer to bluebuild. I'm not a regular contributor or maintainer of the project.

[antheas]

Rechunk is a tool that provides metadata to ostree-ext-cli (there is still a bit of work needed to make them compatible since the ostree-rs-ext PR merged).

If in your case you reach a point in your workflow where you have a complete ostree commit, then just run rechunk on top of it and then feed the contentmeta.json file to ostree-ext-cli to produce the image.

You do not need the action.

[prydom]

bluebuild which is ostree based

Also, please look into bluebuild. I'm not sure why you think it's not OCI based. The links I sent you were to my own downstream overlay repo using both bluebuild for OCI and OSTree for rechunking.

Bluebuild doesn't depend on OSTree, it does create OCI layers.

[prydom]

If in your case you reach a point in your workflow where you have a complete ostree commit, then just run rechunk on top of it and then feed the contentmeta.json file to ostree-ext-cli to produce the image.

Ah, so you know my current solution is entirely independant of yours. I wrote it with no knowledge of your repo and don't really intend to switch unless bluebuild ends up integrating with you.

[antheas]

If in your case you reach a point in your workflow where you have a complete ostree commit, then just run rechunk on top of it and then feed the contentmeta.json file to ostree-ext-cli to produce the image.

Ah, so you know my current solution is entirely independant of yours. I wrote it with no knowledge of your repo and don't really intend to switch unless bluebuild ends up integrating with you.

The initial version of my solution did much of what you did. Create an ostree commit, then feed it to rpm-ostree. However, I found that that did not do much in terms of bandwidth savings.

Which is why I made rechunk, which is an algorithm that rechunks the image a lot better than rpm-ostree does. Then, for distribution I wrapped it into an action that does what you see here and is tailored for OCI images.

For bazzite what this means is that the update size dropped from 5.5gb to 1.5gb average and if you update the same day (e.g., for testing), to 300mb-700mb. So the tool works and you should use it.

As for the action, that is irrelevant for you. I did not know about ostree container pull. However, even if I did for our usecase I think the way I did it makes sense because it should be faster. The whole process finishes in 6-7 minutes for a really large image.

[prydom]

Awesome! Again, thank you for your time.

Feel free to close this issue. I'll link it to the bluebuild issue about integrating this since I mentioned the quirks issue there.

In the meantime, I'll look into if I can integrate your solution into my personal repos. I do have some hard requirements as I do use rootless containers for my local builds but it sounds like your suggestion will let me sidestep that since I won't be using podman mount.

[antheas]

Yes rootful is required for mount to work correctly