Support curl'ing to a URL during install to a provisioning system

[dcritch]

I'm installing via PXE (with Foreman/Red Hat Satellite) and its working great!

The typical flow with Foreman is:

1. click 'build'
2. the pxelinux.cfg for the host gets modified to point to the install initrd/kernel/etc
3. system pxe boots and installs the OS
4. system executes a curl against a specific URL to tell Foreman that the system is built
5. Foreman modifies the pxelinux.cfg so that the system will boot from local disk
6. system reboots in to installed system

The issue I have is that there is no way to do step 4 with CoreOS. As a result, the host permanently stays in build mode and re-installs on every reboot. Manually cancelling the build in Foreman works, but must be done at the right time and ruins the automation of it all.

It would be great if there was a way to curl an arbitrary URL to inform Foreman or any PXE based build system to reset the config to properly complete the installation.

In the case of Foreman, the URL is unique and generated for each build. The pxelinux.cfg files are generated from templates, so I'm including that URL in the hopes of one day being able to pick it up (e.g. from /proc/cmdline) during install and curl'ing it.

Here's my PXE config for reference:

TIMEOUT 10
DEFAULT Fedora CoreOS default PXELinux

LABEL Fedora CoreOS default PXELinux
  KERNEL boot/fcos-installer-kernel
  APPEND ip=dhcp rd.neednet=1 initrd=boot/fcos-installer-initramfs.img console=tty0 console=ttyS0 coreos.autologin=tty1 coreos.autologin=ttyS0 coreos.inst=yes coreos.inst.install_dev=vda coreos.first_boot=1 coreos.inst.image_url=http://foreman.example.com/pub/fcos/fcos-metal-bios.raw.gz coreos.inst.ignition_url=http://foreman.example.com/pub/ignition.ign coreos.autologin=1 forman_url=http://foreman.example.com/unattended/built?token=fc697858-ecb2-4e49-bb22-a641d3fc22d3
  IPAPPEND 2

I'd like to curl that forman_url=http://foreman.example.com/unattended/built?token=fc697858-ecb2-4e49-bb22-a641d3fc22d3 before it is too late.

[lzap]

Implemented this in #25 have fun.

[dustymabe]

Thanks @lzap - This seems like a reasonable request. We'll consider it along with other similar requests and try to have a solution for this problem in the next iteration of the installer.

[StykMartin]

@dustymabe
Any update on this issue? Do you have any plan for incorporation? Unattended installation is getting priority for QA groups which are testing CoreOS/OpenShift

[bgilbert]

From #25 (comment):

For Fedora CoreOS we're going to prioritize live PXE (coreos/fedora-coreos-tracker#105), which will allow running the installer from a full system instead of the initramfs. This will enable arbitrary phone-home functionality via an Ignition config that specifies arbitrary systemd service units to run before/after the installer. That approach seems better than hardcoding a specific callback which runs in initramfs context. RHEL CoreOS should eventually pick up these changes as well.

[StykMartin]

coreos/fedora-coreos-tracker#105 closed.
What is the status of this issue then?

[bgilbert]

The separate Fedora CoreOS installer images are no longer shipped, and the live ISO and live PXE images can now run the installer. It's possible to pass them an Ignition config which adds additional systemd units. This unit (untested) should report success to Foreman after the install is complete:

[Unit]
Description=Report success to Foreman
After=coreos-installer.service
Before=coreos-installer.target

[Service]
Type=oneshot
ExecStart=/usr/bin/curl http://foreman.example.com/unattended/built?token=fc697858-ecb2-4e49-bb22-a641d3fc22d3

[Install]
WantedBy=coreos-installer.target

This should work today. We're still working out some ergonomics, though. Ideally the Ignition config could also configure the installer directly, but today that's not straightforward; you'd need to pass both coreos.inst kernel arguments and an Ignition config.

[dustymabe]

@bgilbert - from that I read that the outstanding items are:

• some small ergonomics improvements
• documentation

[Elyytscha]

The separate Fedora CoreOS installer images are no longer shipped, and the live ISO and live PXE images can now run the installer. It's possible to pass them an Ignition config which adds additional systemd units. This unit (untested) should report success to Foreman after the install is complete:

[Unit]
Description=Report success to Foreman
After=coreos-installer.service
Before=coreos-installer.target

[Service]
Type=oneshot
ExecStart=/usr/bin/curl http://foreman.example.com/unattended/built?token=fc697858-ecb2-4e49-bb22-a641d3fc22d3

[Install]
WantedBy=coreos-installer.target

This should work today. We're still working out some ergonomics, though. Ideally the Ignition config could also configure the installer directly, but today that's not straightforward; you'd need to pass both coreos.inst kernel arguments and an Ignition config.

so how should this work? i create the ignition files manually with the openshift installer, then i have to hack something like this into the right place in the bootstrap.ign // master.ign // worker.ign

{"contents":"[Unit]\nDescription=Report success to Foreman\nAfter=coreos-installer.service\nBefore=coreos-installer.target\n\n[Service]\nType=oneshot\nExecStart=/usr/bin/curl -k http://foreman.example.com/unattended/built\n[Install]\nWantedBy=coreos-installer.target","enabled":true,"name":"foreman-callback.service"}

ok but a vm needs to report a build token back to foreman, how i get the dynamically created build token for the vm into the systemd service which calls back to foreman?

can't hardcode it because multiple vm's (ex. 3 worker nodes) use the same ignition config
can't use foreman snippets (and so foreman variables) to template out the systemd service because all what i can do is specifying a pxe template for fedora // redhat coreos, so this won't work ..

just in my opinion its a really big hoax that there is still no viable way to provision openshiftv4 cluster out of foreman // satellite

[bgilbert]

so how should this work? i create the ignition files manually with the openshift installer, then i have to hack something like this into the right place in the bootstrap.ign // master.ign // worker.ign

Note that there are two separate Ignition configs. The one created by the OpenShift installer is for the installed nodes. The one with the Foreman unit above is for the single boot of the live system that runs coreos-installer.

We're working on better integration of the new installer with RHCOS. Until then, while it should technically be possible to install RHCOS with the FCOS coreos-installer, I wouldn't recommend doing that in production workflows.

[lzap]

Thanks @bgilbert. Is there any CoreOS build I can use to PXE boot and test this feature available?

[StykMartin]

@lzap latest FCOS?
https://builds.coreos.fedoraproject.org/browser
Live Kernel, Live Initrd + Bare Metal img

[dustymabe]

WIth the new rust based installer that runs in the real root (not intramfs) of our Live ISO/PXE artifacts you can do a custom install and curl URLs.. Here is a tutorial for how to do that: https://dustymabe.com/2020/04/04/automating-a-custom-install-of-fedora-coreos/

We need to get this added to our documentation, but I think we can probably close this issue now.