rpm-ostree: Setup readonly sysroot for ostree & rw karg

[poncovka]

• Enable read only sysroot in the ostree repo config.
• Add rw to the kernel arguments to keep statefull parts of the system (/var & /etc) writable.
• Update units tests to account for the new rw karg

(cherry-picked from a commit 0e00c90)

Related: RHEL-2250

[poncovka]

It looks like the ostree container support (introduced in #5399) and the related bootupd support (from #5424) don't work without this commit. The installed systems fail to boot. @cgwalters Is it ok to port this change to RHEL 9?

[poncovka]

/kickstart-test --testtype smoke

[cgwalters]

Yes.

One thing I would note here is with bootc, I've been trying hard to have the container image own its own installation! So if we did #5197 this would automatically happen because bootc install to-filesystem handles this stuff; xref https://github.com/containers/bootc/blob/cbe606265bccda19f6ce01a0816498c5263cc33e/lib/src/install.rs#L597

It's always basically been a bug that this stuff lives in the thing doing the installation...really an ostree design flaw in the end. We've actually fixed that recently with ostreedev/ostree#2930

However the thing that is currently definitely required is that rw end up on the initial kernel commandline.

[cgwalters]

In theory I guess, ostree could just try to force on the rw thing if it detects it's needed...it's just messy and a layering violation.

[jkonecny12]

Looks good to me.