Unable to bind a signed image at install time

[ckyrouac]

When bootc tries to copy the signed image from the host to the install disk it fails with the following error:

ERROR Installing to disk: Pulling from host storage: registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0-13: Subprocess failed: ExitStatus(unix_wait_status(32000))
Getting image source signatures
Checking if image destination supports signatures
Error: Copying this image would require changing layer representation, which we cannot do: "Would invalidate signatures"

Adding "--remove-signatures" to the podman image push command here fixes this error, however the result is an unsigned image. I did some digging through the containers/image code, skopeo docs, and containers-storage.conf docs. It looks like the only way to copy and sign an image is to re-sign the image when copying it, I couldn't find a way to copy a signed image while preserving the signature. I might be missing something though.

[cgwalters]

Hm, only glancing at this one thing I notice is

$ skopeo inspect --raw -n docker://registry.redhat.io/amq-streams/kafka-37-rhel9:2.7.0-13 | jq -r .mediaType
application/vnd.docker.distribution.manifest.list.v2+json

It's a legacy media type, not OCI. This is a thing we should start trying to change across Red Hat images.

But indeed there may be a larger conflict here even with OCI that may need fixing.

[Odilhao]

Hey @cgwalters we just stumbled on this while tying to locally build one RHEL 9.4 image.

With a really simple config.toml:

[[blueprint.customizations.user]]
name = "someuser"
password = "somepassword"
key = ""
groups = ["wheel"]

sudo podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v ./config.toml:/root/config.toml:ro -v ./output:/root/output -v /var/lib/containers/storage:/var/lib/containers/storage registry.redhat.io/rhel9/bootc-image-builder:latest --local --type qcow2 --config /root/config.toml registry.redhat.io/rhel9/rhel-bootc:9.4

That fails with:

Checking if image destination supports signatures
time="2024-10-03T20:25:35Z" level=fatal msg="Copying this image would require changing layer representation, which we cannot do: \"Would invalidate signatures\""
Error: tmp-container-deploy-71731344553455: image not known
Traceback (most recent call last):
 File "/run/osbuild/bin/org.osbuild.container-deploy", line 50, in main
   subprocess.run(
 File "/usr/lib64/python3.9/subprocess.py", line 528, in run
   raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['skopeo', 'copy', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]1f64fbbed435d0f92755b9e6f6521e8d20c3d00f62cb869774cc6d0aab9bf897', 'containers-storage:tmp-container-deploy-71731344553455']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/run/osbuild/bin/org.osbuild.container-deploy", line 64, in <module>
   r = main(args["inputs"], args["tree"], args["options"])
 File "/run/osbuild/bin/org.osbuild.container-deploy", line 56, in main
   subprocess.run(["cp", "-a", f"{img}/.", f"{tree}/"], check=True)
 File "/usr/lib64/python3.9/contextlib.py", line 532, in __exit__
   raise exc_details[1]
 File "/usr/lib64/python3.9/contextlib.py", line 517, in __exit__
   if cb(*exc_details):
 File "/usr/lib64/python3.9/contextlib.py", line 405, in _exit_wrapper
   callback(*args, **kwds)
 File "/usr/lib64/python3.9/subprocess.py", line 528, in run
   raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'rmi', 'tmp-container-deploy-71731344553455']' returned non-zero exit status 1.

[cgwalters]

Hi @Odilhao that issue is related (in fact ultimately the same I suspect) but practically distinct because the failure there is in bootc-image-builder code.

For reasons I don't understand really (and we're debating in multiple places), bootc-image-builder ends up copying the input container's rootfs and running it via custom tooling instead of just launching it via a standard OCI mechanism (e.g. podman).

[cgwalters]

cc @mvo5 @achilleas-k re ⬆