Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

install-to-filesystem does not work when targeting LVM #175

Closed
miabbott opened this issue Nov 3, 2023 · 10 comments · Fixed by #384
Closed

install-to-filesystem does not work when targeting LVM #175

miabbott opened this issue Nov 3, 2023 · 10 comments · Fixed by #384
Labels
area/install Issues related to `bootc install` bug Something isn't working

Comments

@miabbott
Copy link
Contributor

miabbott commented Nov 3, 2023

Host system is Fedora 39 Server.

Using the tier1 Fedora ELN image, I created a custom variant that had bootc installed from Copr:

FROM registry.gitlab.com/centos/cloud/sagano/fedora-boot-tier-1:eln
RUN curl -L -o /etc/yum.repos.d/bootc.repo https://copr.fedorainfracloud.org/coprs/rhcontainerbot/bootc/repo/fedora-rawhide/rhcontainerbot-bootc-fedora-rawhide.repo
RUN sed -i 's|\$releasever|rawhide|' /etc/yum.repos.d/bootc.repo
RUN rpm-ostree install bootc
RUN ostree container commit

Built it successfully with sudo podman build -t sagano-bootc-copr:latest -f Containerfile.sagano .

Tried install-to-filesystem:

$ image=localhost/sagano-bootc-copr:latest
$ sudo podman run --rm --privileged --pid=host -v /:/target --security-opt label=type:unconfined_t "${image}" bootc install-to-filesystem --target-no-signature-verification --karg=console=ttyS0,115200n8 --replace=alongside /target
Mounting selinuxfs
ERROR No filesystem uuid found in target root

Hmm...looks like it might be SELinux as the culprit:

$ sudo ausearch -m avc
----
time->Fri Nov  3 12:03:42 2023
type=AVC msg=audit(1699027422.008:673): avc:  denied  { nosuid_transition } for  pid=3425 comm="bootc" scontext=system_u:system_r:unconfined_t:s0:c54,c375 tcontext=system_u:system_r:install_t:s0:c54,c375 tclass=process2 permissive=0
@miabbott
Copy link
Contributor Author

miabbott commented Nov 3, 2023

Similar failure using CentOS Stream 9 as the host.

$ cat Containerfile.sagano 
FROM registry.gitlab.com/centos/cloud/sagano/centos-boot-tier-1:stream9
# Not sur why I can't just use the copr repo, but it doesn't show as an
# enabled repo when I copy it to /etc/yum.repos.d ¯\_(ツ)_/¯
RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm
RUN rpm-ostree install bootc.rpm
RUN ostree container commit
$ image=localhost/sagano-cs9-bootc-copr:latest                                                                                               
$ sudo podman run --rm --privileged --pid=host -v /:/target --security-opt label=type:unconfined_t "${image}" bootc install-to-filesystem --target-no-signature-verification --karg=console=ttyS0,115200n8 --replace=alongside /target
Mounting selinuxfs                                                
ERROR Inspecting filesystem /target: findmnt /target failed: ExitStatus(unix_wait_status(256))        
$ sudo ausearch -m avc
----
time->Fri Nov  3 16:07:22 2023
type=PROCTITLE msg=audit(1699042042.505:752): proctitle=2F746D702F2E746D70376174694E4800696E7374616C6C2D746F2D66696C6573797374656D002D2D7461726765742D6E6F2D7369676E61747572652D766572696669636174696F6E002D2D6B6172673D636F6E736F6C653D74747953302C3131353230306E38002D2D7265706C6163653D616C6F6E6773696465002F74617267
type=PATH msg=audit(1699042042.505:752): item=0 name="/lib64/ld-linux-x86-64.so.2" inode=34460089 dev=00:32 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:container_file_t:s0:c264,c980 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1699042042.505:752): cwd="/"
type=EXECVE msg=audit(1699042042.505:752): argc=6 a0="/tmp/.tmp7atiNH" a1="install-to-filesystem" a2="--target-no-signature-verification" a3="--karg=console=ttyS0,115200n8" a4="--replace=alongside" a5="/target"
type=SYSCALL msg=audit(1699042042.505:752): arch=c000003e syscall=59 success=yes exit=0 a0=55bdd5e15820 a1=55bdd5ddfca0 a2=55bdd5e16ca0 a3=8 items=1 ppid=35181 pid=35183 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4 comm=".tmp7atiNH" exe="/tmp/.tmp7atiNH" subj=system_u:system_r:unconfined_t:s0:c264,c980 key=(null)
type=SELINUX_ERR msg=audit(1699042042.505:752): op=security_bounded_transition seresult=denied oldcontext=system_u:system_r:unconfined_t:s0:c264,c980 newcontext=system_u:system_r:install_t:s0:c264,c980
type=AVC msg=audit(1699042042.505:752): avc:  denied  { nosuid_transition } for  pid=35183 comm="bootc" scontext=system_u:system_r:unconfined_t:s0:c264,c980 tcontext=system_u:system_r:install_t:s0:c264,c980 tclass=process2 permissive=0

@cgwalters
Copy link
Collaborator

RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm
RUN rpm-ostree install bootc.rpm
RUN ostree container commit

(this is unrelated to the bug, but what you really want is more like:

RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm && \
  rpm-ostree install bootc.rpm && rm bootc.rpm && ostree container commit

This will avoid leaking the RPM (as a file) into the container image and ensure that unnecessary content is properly squashed.

@cgwalters
Copy link
Collaborator

Similar failure using CentOS Stream 9.

The failure is totally unrelated to the bootc-in-container (or the OS version of the container), it's about the host SELinux policy. (When you first said "Using c9s" I thought you meant for the host, not the container)

@cgwalters
Copy link
Collaborator

Hmm...looks like it might be SELinux as the culprit:

Right. We've been trying really really really hard to make this work. We have #83 which is related.

For #78 I finally just gave up and in that mode did a setenforce 0. (Background: for reasons similar to this, Anaconda has always run in setenforce 0 mode)

Can you check that setenforce 0 beforehand works? Bear in mind that that this command is effectively replacing the OS, and setenforce 0 is a transient operation for the soon-to-go-away OS.

It's really tempting to do that unconditionally but I'm trying to Do It Right as much as possible. I'm not yet sure why it works in some paths but not others.

cc @rhatdan

@cgwalters cgwalters added bug Something isn't working area/install Issues related to `bootc install` labels Nov 3, 2023
@cgwalters
Copy link
Collaborator

cgwalters commented Nov 3, 2023

Oh actually this has nothing to do with selinux...we're already detecting this situation and running setenforce 0 internally already.

The problem is we don't support LVM. One immediate issue seems to be that when executed from a container with a bind mount to the target root, the findmnt tooling gets confused and doesn't manage to introspect the UUID in this case.

I think what we need to do here is two things to fix the baseline error:

  • Add install-to-host-filesystem-alongside (as an effective alias for install-to-filesystem --replace=alongside /target) which would have improved ergonomics
  • Enter the host mountns when gathering data like from findmnt - here we know to do that instead of writing to an arbitrary filesystem as we have with /target (although I guess we could detect the case where it's the host root)

But that's just preparatory work.

I think we'll also need to pick up kargs like rd.lv* etc., and properly honor root=/dev/mapper/root.

@cgwalters cgwalters changed the title No filesystem uuid found in target root when trying install-to-filesystem using F39 Server install-to-filesystem does not work when targeting LVM Nov 3, 2023
@rhatdan
Copy link
Member

rhatdan commented Nov 4, 2023

Can't we run the tool that is attempting to install as install_t or unconfined_t? Is this bootc? We could just transition to an unconfined role. Looks like there is a type

install_exec_t

Which transition to install_t, which is allowed to do anything including put labels that don't exist yet, I believe.

@miabbott
Copy link
Contributor Author

miabbott commented Nov 5, 2023

The failure is totally unrelated to the bootc-in-container (or the OS version of the container), it's about the host SELinux policy. (When you first said "Using c9s" I thought you meant for the host, not the container)

I amended my comment to indicate I was using a c9s host, even if it is still an LVM issue.

@henrywang
Copy link
Contributor

henrywang commented Jan 31, 2024

@cgwalters Hi! I run into the same issue on ARM VM in beaker. ARM VMs in beaker are deployed as LVM when install CentOS Stream 9.

[root@ampere-mtsnow-altramax-02-vm-15 /]# df -h
Filesystem                                                Size  Used Avail Use% Mounted on
devtmpfs                                                  4.0M     0  4.0M   0% /dev
tmpfs                                                      17G  100K   17G   1% /dev/shm
tmpfs                                                     6.8G  8.8M  6.8G   1% /run
/dev/mapper/cs_ampere--mtsnow--altramax--02--vm--15-root   37G  3.7G   33G  11% /
/dev/vda2                                                 960M   39M  922M   5% /boot
/dev/vda1                                                 599M  4.0K  599M   1% /boot/efi
/dev/mapper/cs_ampere--mtsnow--altramax--02--vm--15-home   18G  159M   18G   1% /home
tmpfs                                                     3.4G     0  3.4G   0% /run/user/0

@shi2wei3
Copy link

shi2wei3 commented Mar 1, 2024

@cgwalters Do you think we need to escalate this issue a little bit? Azure RHEL on-demand images are using LVM. And I just tried to do bootc install on a RHEL9.3 public image on Azure, which lead to exactly the same problem here.

@cgwalters
Copy link
Collaborator

Ah, good find. So...I don't think we need to escalate this right now, let's just document the places that it works and the limitation on LVM.

The real fix for the "in-place" will be #78 which will default to rewriting the filesystem to not be LVM (i.e. it will default to what the container image specifies).

(We still do want to support configuring LVM in this scenario of course, but that gets into its own trickiness)

cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 8, 2024
Teach `install to-existing-root` how to gather kernel arguments
and information we need from `/proc/cmdline` (such as `rd.lvm.lv`).
In this case too, we need to adjust the args to use `-v /dev:/dev`
because we need the stuff udev generates from the real host root.

With these things together, I can do a bootc alongside install
targeting a Fedora Server instance.

Closes: containers#175
Signed-off-by: Colin Walters <[email protected]>
cgwalters added a commit to cgwalters/bootc that referenced this issue Mar 9, 2024
Teach `install to-existing-root` how to gather kernel arguments
and information we need from `/proc/cmdline` (such as `rd.lvm.lv`).
In this case too, we need to adjust the args to use `-v /dev:/dev`
because we need the stuff udev generates from the real host root.

With these things together, I can do a bootc alongside install
targeting a Fedora Server instance.

Closes: containers#175
Signed-off-by: Colin Walters <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/install Issues related to `bootc install` bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants