Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignition tweaks to work with composefs #2404

Merged
merged 1 commit into from
Jun 18, 2023

Conversation

alexlarsson
Copy link
Contributor

I'm currently working on composefs use in ostree:
ostreedev/ostree#2640

I'm running into issues with ignition when testing it. Several places look at /sysroot and try to find the backing filesystem for it. This works because /sysroot is typically a bind-mount of some deploy directory in the real sysroot. However, in the composefs case, /sysroot is a composefs mount and doesn't have a real backing block device.

This change switches these cases to look at /sysroot/sysroot instead. This is the actual real sysroot after ostree-prepare-root did its work (and before pivot-rooting into it).

cgwalters
cgwalters previously approved these changes Apr 27, 2023
@cgwalters
Copy link
Member

This is failing on

[   60.557632] coreos-boot-edit[2475]: Error: opening /sysroot/sysroot/etc/crypttab
[   60.558229] coreos-boot-edit[2475]: Caused by:
[   60.558625] coreos-boot-edit[2475]:     No such file or directory (os error 2)
[   60.559578] EXT4-fs (md127): unmounting filesystem 0e6352a8-c202-4517-b4ca-878d6275681

Which...right. Tricky. I think we may need to instead change the rootmap logic to know how to handle this?

@alexlarsson
Copy link
Contributor Author

So, I have no idea how any of this really works, because I never looked at it before. However, it seems that rdcore rootmap really needs the actual "root" with etc and stuff, but it also assumes the root it has a block device, which is not true anymore (as it is an overlayfs mount).

I can't really think of any good way to get from the overlayfs mount back to whatever mount the erofs image in layer 1 of the overlayfs was stored on, so maybe what we need is to pass in boot the "real root" (i.e. /sysroot) and the "backing root fs dir" (i.e. /sysroot/sysroot) as arguments. The later can be optional, like:

rdcore rootmap /sysroot --root-mountdir /sysroot/sysroot --boot-mount ${bootmnt}

For the record, this is how a fully booted system looks with an overlayfs+erofs based composefs:

overlay / overlay ro,seclabel,relatime,lowerdir=/tmp/.composefs.CuUFPI:/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on 0 0
/dev/vda4 /etc xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota 0 0
/dev/vda4 /sysroot/ostree/deploy/fedora-coreos/var xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,prjquota 0 0

The erofs image is loopback mounted, initially at /tmp/.composefs.CuUFPI:

losetup 
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                                                                                                  DIO LOG-SEC
/dev/loop0         0      0         1  1 /sysroot/ostree/deploy/fedora-coreos/deploy/2758a35a494c60ae0873b52ce9364c773377c56aa4f564c72a0c48801432cb65.0/.ostree.cfs   1    4096

But the actual mount is unmounted and lives now only as a private mount inside the overlayfs mount.

Hmm, I guess if we mount the overlay at something like /tmp/.composefs.loop0.XXXXXX, then we can use the loopback data to map back to the original file. Not super clean, but workable.

@alexlarsson
Copy link
Contributor Author

Or i guess I could pass in a separate mountpoint argument for the erofs, which could be in the deploy dir, then I don't have to care about the loop part.

@alexlarsson
Copy link
Contributor Author

Ok, I updated the composefs, and now the root mount looks like this:

overlay on / type overlay (ro,relatime,seclabel,lowerdir=/sysroot/ostree/deploy/fedora-coreos/deploy/443ae0cd86a7dd4c6f5486a2283471b3c8f76fc5dcc4766cf935faa24a9e3d34.0/.ostree.mnt:/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on)

So, conceptually we could reverse map back to the deploy dir.

@alexlarsson
Copy link
Contributor Author

Note that the directory is empty, there won't be any mount there anymore when things are running, so it will map to the right block device for the deploy dir.

@jlebon
Copy link
Member

jlebon commented May 1, 2023

We should probably discuss this in coreos/fedora-coreos-tracker#1252 instead where we weigh the different possible approaches. Want to add a comment there? It sounds like you're just playing around with composefs on FCOS rather than proposing we support it, correct?

@jlebon jlebon added the hold label May 1, 2023
@alexlarsson
Copy link
Contributor Author

My goal is to support composefs in ostree because we will need it on the automotive side, I'm just looking at FCOS as an easy way to test the ostree work. However, I still think it makes sense for fcos to use it.

This is to enable ostree+composefs:
ostreedev/ostree#2867

When we care about the *physical* backing filesystem, we need to
look at /sysroot/sysroot (which in the real root is `/sysroot`)
because now `/sysroot` (aka `/` in the real root) is a composefs
(really an `overlayfs` with a transient loop-mounted erofs), which
is distinct from the physical root.

Co-authored-by: Colin Walters <[email protected]>
@cgwalters
Copy link
Member

OK, I've reduced the change here to just one for the rootfs size checking. The other bits for rdcore now require: coreos/coreos-installer#1203

I've verified that these two PRs in combination give PASS: coreos.boot-mirror.luks (87.29s) (at least with composefs, but CI here will also check without).

@cgwalters
Copy link
Member

This one is pretty trivial and safe, any takers?

@cgwalters cgwalters removed the hold label Jun 14, 2023
@alexlarsson
Copy link
Contributor Author

It looks good to me, but then I was the initial author, and my knowledge of fcos is very basic, so I don't think I should really be acking this.

@cgwalters cgwalters merged commit 617ac17 into coreos:testing-devel Jun 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants