Produce live PXE images

[bgilbert]

Container Linux supports PXE-booting to a live system, which is often used to run Container Linux both on bare metal and in certain virtualization environments (e.g. VMware). Users migrating from Container Linux, as well as users who could otherwise benefit from this deployment model, will need a way to netboot a live Fedora CoreOS system.

Container Linux's PXE support has the following properties:

1. initramfs
   1. The initramfs contains a squashfs with the /usr filesystem image (i.e., OS code). There's a dracut module that detects the squashfs and mounts it instead of a permanent disk partition.
   2. The "normal" Container Linux initramfs is compiled into the kernel image, so from the kernel's perspective there are two initramfsen, but from the user's perspective there's only one.
2. Filesystems
   1. The root filesystem (i.e., configuration and user data) is a tmpfs by default, but can be a persistent filesystem if the user creates one with the proper label.
   2. Regardless of the existence of a persistent root, there is no /boot filesystem.
3. Lifecycle
   1. Ignition runs iff the PXE server requests it via a kernel command-line option. If the machine has a persistent root filesystem, this means that Ignition may or may not run on every boot, depending on the configuration of the PXE server.
   2. Automatic updates are disabled in PXE mode, and thus reboot coordination is disabled as well. To apply updates, users must update the OS image on their PXE server and then reboot nodes into the new image.
   3. The live PXE environment may be used to install Container Linux to disk using coreos-install, either manually or via a systemd oneshot service installed via an Ignition config. A CL system installed to disk in this way is an ordinary system with no further dependence on network booting.

As a starting point, I'd propose continuing essentially the same model for Fedora CoreOS. The main initramfs will not be compiled into the kernel image, but since PXE loaders (PXELINUX and iPXE) support passing multiple initramfsen to the kernel, it shouldn't be a problem to continue storing the root filesystem in a separate initramfs. (It may provide confusing UX, however.) We'll want to provide binaries and docs for PXE-booting on both BIOS and UEFI systems, including UEFI with Secure Boot enabled.

One other point: on Container Linux, PXE-booted systems are assigned the OEM ID diskless [sic] for some purposes. (It's not passed on the kernel command line, but is used by update_engine_stub when checking into CoreUpdate.) PXE booting is orthogonal to the platform ID (e.g., it's possible to netboot a VMware VM) so we should avoid conflating them in Fedora CoreOS.

[vielmetti]

This is great, we'd love to have PXE support at @packethost and we have infrastructure to help test this out - just let me know how we can help once you get to the point of wanting to try this.

A relevant KB article for reference is https://support.packet.com/kb/articles/custom-ipxe

[lucab]

The initramfs contains a squashfs with the /usr filesystem image (i.e., OS code)

If somebody wants to take it over, I started coreos/coreos-assembler#261 exactly for this reason, but there are a few things to polish that I'm not doing at the moment.

[dustymabe]

The initramfs contains a squashfs with the /usr filesystem image (i.e., OS code)

If somebody wants to take it over, I started coreos/coreos-assembler#261 exactly for this reason, but there are a few things to polish that I'm not doing at the moment.

yep. i'm also looking at PXE as part of #91

[vielmetti]

One note to drop in re PXE, specifically iPXE - we are tracking this issue ipxe/ipxe#90 which yields intermittent failures with iPXE as a result of a bug in OCSP certificate validation. Any eyes on this especially for test and validation cycles would be helpful.

[bgilbert]

Discussed OOB with @ajeddeloh and @jlebon. If we allow both 1) a persistent /etc and 2) upgrading the OS by simply PXEing a new image, updates will bypass the rpm-ostree three-way merge of /etc. There are ways to fix that, but since we don't know how many CL users even use PXE with a persistent root, we think we should defer support for persistent /etc.

We think live PXE on FCOS should initially:

• Allow persistent data filesystems, including persistent /var.
• Not allow persistent root or persistent /etc. The root filesystem will be a r/o squashfs.
• Assume that the PXE config runs Ignition on every boot. (Strictly speaking that's not a requirement, but there won't be a way to customize /etc without it.)
• Refuse to install updates.

[ajeddeloh]

Not allow persistent root or persistent /etc. The root filesystem will be a r/o squashfs.

In the initramfs or when running? I'm assuming it's rw in the initramfs at least.

[bgilbert]

zstd compression would make sense for the squashfs. It can compress well and it decompresses faster than xz. The Fedora kernel includes support; squashfs-tools doesn't yet.

In the initramfs or when running? I'm assuming it's rw in the initramfs at least.

squashfs is always r/o, so you're right that this won't work as stated.

[bgilbert]

Disabling Zincati in live PXE images: coreos/zincati#63

[dustymabe]

@bgilbert - I know there has been a lot of work here. Where do we stand?

[vielmetti]

As point of interest, we are at Packet also looking at netboot.xyz for an Arm boot effort.

…

On Wed, Oct 2, 2019 at 11:05 AM Dusty Mabe ***@***.***> wrote: @bgilbert <https://github.com/bgilbert> - I know there has been a lot of work here. Where do we stand? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#105?email_source=notifications&email_token=AAAIRZYXR75DU6CJJ5GGYK3QMS2FBA5CNFSM4GO33SFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAFCUUQ#issuecomment-537537106>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAIRZYC6XTF3ELTIPAKBF3QMS2FBANCNFSM4GO33SFA> .

[vtolstov]

For workaround can i get full ostree system create squashfs and boot via dracut?

[cgwalters]

For workaround can i get full ostree system create squashfs and boot via dracut?

We landed buildextend-live in coreos-assembler.

[vtolstov]

@cgwalters can you provide some draft howto to build live pxe image with ostree?
Two years ago i simply to like i say - install via anaconda ostree system in qemu and create squashfs from it.

[dustymabe]

If you can build using COSA (see README) then cosa fetch && cosa build && cosa buildextend-live should give you a live-kernel and live-initramfs that you can then use to LIVE PXE boot.

[bgilbert]

The latest release shipped live PXE images. There are no docs yet, and #281 should be fixed, but other than that it's done.

[dustymabe]

The latest release shipped live PXE images. There are no docs yet, and #281 should be fixed, but other than that it's done.

Should we close this and let remaining pieces be covered by #281 and maybe #159 (or create a separate docs ticket)

[bgilbert]

Yup, sounds good. Closing.