Summary
osbuild-composer most of the time generates an osbuild manifest that skips gpg checking on RPMs from a payload repository, i.e. a repo added as a source like composer-cli sources add epel.toml, if there is another payload repository with check_gpg = false configured.
Details
I ran the same (tar) build over and over in a loop overnight. It uses RPMs from the following custom repo I configured (settings copied from epel.repo)
[root@f9c3a85f2ed5 builds]# composer-cli sources info epel
check_gpg = true
check_repogpg = false
check_ssl = true
gpgkeys = ["-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBFz3zvsBEADJOIIWllGudxnpvJnkxQz2CtoWI7godVnoclrdl83kVjqSQp+2\ndgxuG5mUiADUfYHaRQzxKw8efuQnwxzU9kZ70ngCxtmbQWGmUmfSThiapOz00018\n+eo5MFabd2vdiGo1y+51m2sRDpN8qdCaqXko65cyMuLXrojJHIuvRA/x7iqOrRfy\na8x3OxC4PEgl5pgDnP8pVK0lLYncDEQCN76D9ubhZQWhISF/zJI+e806V71hzfyL\n/Mt3mQm/li+lRKU25Usk9dWaf4NH/wZHMIPAkVJ4uD4H/uS49wqWnyiTYGT7hUbi\necF7crhLCmlRzvJR8mkRP6/4T/F3tNDPWZeDNEDVFUkTFHNU6/h2+O398MNY/fOh\nyKaNK3nnE0g6QJ1dOH31lXHARlpFOtWt3VmZU0JnWLeYdvap4Eff9qTWZJhI7Cq0\nWm8DgLUpXgNlkmquvE7P2W5EAr2E5AqKQoDbfw/GiWdRvHWKeNGMRLnGI3QuoX3U\npAlXD7v13VdZxNydvpeypbf/AfRyrHRKhkUj3cU1pYkM3DNZE77C5JUe6/0nxbt4\nETUZBTgLgYJGP8c7PbkVnO6I/KgL1jw+7MW6Az8Ox+RXZLyGMVmbW/TMc8haJfKL\nMoUo3TVk8nPiUhoOC0/kI7j9ilFrBxBU5dUtF4ITAWc8xnG6jJs/IsvRpQARAQAB\ntChGZWRvcmEgRVBFTCAoOCkgPGVwZWxAZmVkb3JhcHJvamVjdC5vcmc+iQI4BBMB\nAgAiBQJc9877AhsPBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRAh6kWrL4bW\noWagD/4xnLWws34GByVDQkjprk0fX7Iyhpm/U7BsIHKspHLL+Y46vAAGY/9vMvdE\n0fcr9Ek2Zp7zE1RWmSCzzzUgTG6BFoTG1H4Fho/7Z8BXK/jybowXSZfqXnTOfhSF\nalwDdwlSJvfYNV9MbyvbxN8qZRU1z7PEWZrIzFDDToFRk0R71zHpnPTNIJ5/YXTw\nNqU9OxII8hMQj4ufF11040AJQZ7br3rzerlyBOB+Jd1zSPVrAPpeMyJppWFHSDAI\nWK6x+am13VIInXtqB/Cz4GBHLFK5d2/IYspVw47Solj8jiFEtnAq6+1Aq5WH3iB4\nbE2e6z00DSF93frwOyWN7WmPIoc2QsNRJhgfJC+isGQAwwq8xAbHEBeuyMG8GZjz\nxohg0H4bOSEujVLTjH1xbAG4DnhWO/1VXLX+LXELycO8ZQTcjj/4AQKuo4wvMPrv\n9A169oETG+VwQlNd74VBPGCvhnzwGXNbTK/KH1+WRH0YSb+41flB3NKhMSU6dGI0\nSGtIxDSHhVVNmx2/6XiT9U/znrZsG5Kw8nIbbFz+9MGUUWgJMsd1Zl9R8gz7V9fp\nn7L7y5LhJ8HOCMsY/Z7/7HUs+t/A1MI4g7Q5g5UuSZdgi0zxukiWuCkLeAiAP4y7\nzKK4OjJ644NDcWCHa36znwVmkz3ixL8Q0auR15Oqq2BjR/fyog==\n=84m8\n-----END PGP PUBLIC KEY BLOCK-----\n"]
id = "epel"
name = "Extra Packages for Enterprise Linux 8 - $basearch"
rhsm = false
system = false
type = "yum-metalink"
url = "https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=$basearch&infra=$infra&content=$contentdir"
and here is the config for the non-gpg checked repository
[root@f9c3a85f2ed5 builds]# composer-cli sources info local
id = "local"
name = "Local RPMS"
type = "yum-baseurl"
url = "file:///rpms/"
The next day I downloaded the manifests from all 55 builds, and in 48 of the builds the RPMs from EPEL were inputs in the os pipeline org.osbuild.rpm stage WITHOUT options set (i.e. no gpgcheck set). In the remaining 7 builds the gpgcheck was enabled on these same RPMs.
Example from "good" build:
< {
< "id": "sha256:b69491a8aee0dffbd532deff530ac69691389ac0c9dfa88a169d818a25ad31b9",
< "options": {
< "metadata": {
< "rpm.check_gpg": true
< }
< }
< },
Example from "bad" build:
> }
> "id": "sha256:b69491a8aee0dffbd532deff530ac69691389ac0c9dfa88a169d818a25ad31b9"
> },
I noticed this issue when I was testing the build with check_gpg = true in the epel.toml file but without the EPEL gpgkey provided. With this configuration, the build would fail every 10-20% of the time. I discovered these manifest differences while investigating these build failures. It turned out that the build would fail with checksig errors only when the manifest was correctly generated with the check_gpg option for the EPEL rpms. All the other builds succeeded because of the bad manifest. I added the EPEL gpg key to make my builds succeed 100% of the time, but gpg checks are still skipped the majority of the time as evidenced above. Without these checks there is no guarantee that the RPM in the build is the same RPM packaged by EPEL, thus a security risk. I don't think that this is EPEL specific either. I think any payload repo would have this same problem. In every build either all 12 EPEL RPMs were verified or all 12 were unverified. This leads me to believe that the check_gpg setting on the repo itself is the culprit. Interestingly, when I remove the "local" repo the build does not fail anymore.
PoC
EDIT: The steps I provide below are simplified versions of how I reproduced, but I oversimplified and the build appears to be behaving correctly. I am currently working on fixing the below steps.
I reproduced this in a docker container based on registry.access.redhat.com/ubi8/ubi-init:8.9-3, redhat's universal base image for RHEL 8.9. I have no evidence to suggest that that is a repro prerequisite, unless this has been fixed in newer versions of osbuild-composer. The version that get's installed in that container is from the rhel 8 appstream repo, version 88-1.el8.
Once osbuild-composer is set up, add the custom repo from above
composer-cli sources add epel.toml
and add the local repo from above
composer-cli sources add local.toml
Then generate a manifest from the following blueprint
name = "build"
version = "0.0.1"
distro = "rhel-8"
[[packages]]
name = "htop"
version = "*"
I used rhel-8 distro but I don't think that is important. I use the htop package here, but I expect any EPEL package to be just as good.
Once the build manifest is available you can inspect it check if the EPEL rpms are going to be verified or not. If they are unverified then you have reproduced the problem. If they are verified, then you should try another build in the exact same way. It shouldn't take more than 2 tries.
I did a bunch of builds overnight and checked them at once using the following command
# cat *.json | jq -s 'map(.pipelines[] | select(.name == "os") | .stages[] | select(.type == "org.osbuild.rpm") | select(.options.gpgkeys | length == 3) | .inputs.packages.references | map(select(.options == null)) | length > 0) | {"unverified_builds":map(select(.)) | length, "verified_builds": map(select(. | not)) | length}'
{
"unverified_builds": 48,
"verified_builds": 7
}
Impact
I'm not a security researcher, I just wanted my builds to be less flaky and I ended up discovering this, so I don't know how to classify this vulnerability. But I do know that if without gpg signature verification the RPM that is downloaded and installed by osbuild could be corrupted or replaced with a bad version, which could then result is a compromised base image. This seems like a security vulnerability to me, but even if you disagree with that I think this is definitely a bug.
andremarianiello Reporter
Summary
osbuild-composer most of the time generates an osbuild manifest that skips gpg checking on RPMs from a payload repository, i.e. a repo added as a source like
composer-cli sources add epel.toml, if there is another payload repository with check_gpg = false configured.Details
I ran the same (
tar) build over and over in a loop overnight. It uses RPMs from the following custom repo I configured (settings copied from epel.repo)and here is the config for the non-gpg checked repository
The next day I downloaded the manifests from all 55 builds, and in 48 of the builds the RPMs from EPEL were inputs in the
ospipelineorg.osbuild.rpmstage WITHOUT options set (i.e. no gpgcheck set). In the remaining 7 builds the gpgcheck was enabled on these same RPMs.Example from "good" build:
Example from "bad" build:
I noticed this issue when I was testing the build with
check_gpg = truein the epel.toml file but without the EPEL gpgkey provided. With this configuration, the build would fail every 10-20% of the time. I discovered these manifest differences while investigating these build failures. It turned out that the build would fail withchecksigerrors only when the manifest was correctly generated with the check_gpg option for the EPEL rpms. All the other builds succeeded because of the bad manifest. I added the EPEL gpg key to make my builds succeed 100% of the time, but gpg checks are still skipped the majority of the time as evidenced above. Without these checks there is no guarantee that the RPM in the build is the same RPM packaged by EPEL, thus a security risk. I don't think that this is EPEL specific either. I think any payload repo would have this same problem. In every build either all 12 EPEL RPMs were verified or all 12 were unverified. This leads me to believe that the check_gpg setting on the repo itself is the culprit. Interestingly, when I remove the "local" repo the build does not fail anymore.PoC
EDIT: The steps I provide below are simplified versions of how I reproduced, but I oversimplified and the build appears to be behaving correctly. I am currently working on fixing the below steps.
I reproduced this in a docker container based on
registry.access.redhat.com/ubi8/ubi-init:8.9-3, redhat's universal base image for RHEL 8.9. I have no evidence to suggest that that is a repro prerequisite, unless this has been fixed in newer versions of osbuild-composer. The version that get's installed in that container is from the rhel 8 appstream repo, version88-1.el8.Once osbuild-composer is set up, add the custom repo from above
and add the local repo from above
Then generate a manifest from the following blueprint
I used
rhel-8distro but I don't think that is important. I use the htop package here, but I expect any EPEL package to be just as good.Once the build manifest is available you can inspect it check if the EPEL rpms are going to be verified or not. If they are unverified then you have reproduced the problem. If they are verified, then you should try another build in the exact same way. It shouldn't take more than 2 tries.
I did a bunch of builds overnight and checked them at once using the following command
Impact
I'm not a security researcher, I just wanted my builds to be less flaky and I ended up discovering this, so I don't know how to classify this vulnerability. But I do know that if without gpg signature verification the RPM that is downloaded and installed by osbuild could be corrupted or replaced with a bad version, which could then result is a compromised base image. This seems like a security vulnerability to me, but even if you disagree with that I think this is definitely a bug.