Readonly RootFS with an OverlayFS

[h0l0gr4m]

Hi,

First of all thank you for providing this layer!

So basically what i want/need is a jetson device that is using swupdate with an ab partitioning scheme. I see that there is already a flag for a redundant partitioning scheme and i also see that there is already an example in the demo distro that uses the swupdate mechanism.
Sadly i need some data (configuration data, certificates or logs) to be persistent between system updates. Therefore i want to have another partition that can be used as an upperlayer in a overlayfs. So basically i would have a readonly rootfs (a or b) as lower layer and an another partition as upper layer for the persistent configuration data .

My state of knowledge right now is as follows:

• there is a xml-file that describes the partionioning scheme
• probably i would have to copy the existing one for ab scheme and adjust it to create another partition
• there is an overlay.bbclass and a overlay-etc.bbclass

What i don't know:

• How to know which device the "newly" create partition?
• Overlay-etc.bbclass seems to only overlay the etc folder, however i want the overlayfs cover the whole root.
• Is this whole idea a reasonable approach?

Is there already something in this direction that i missed? Or is there another approach more suitable ? Any hints, documentation or help in this direction would be appreciated.

[dwalkes]

there is a xml-file that describes the partionioning scheme
probably i would have to copy the existing one for ab scheme and adjust it to create another partition

Yes, correct, and you can find some information scattered about on the wiki and discussions about this. We could use some better documentation for these steps.

How to know which device the "newly" create partition?

You'll specify the device in the XML file, and a partition name. You can use the /dev/disk sysfs paths to find the partition by name.

Sadly i need some data (configuration data, certificates or logs) to be persistent between system updates.
Overlay-etc.bbclass seems to only overlay the etc folder, however i want the overlayfs cover the whole root.

The main reason to overlay /etc is that systemd gets unhappy if /etc isn't writeable. You can find some threads about this here or elsewhere.

You might reconsider whether you actually need the entire rootfs overlayed. In my experience it's better to let things which try to write to places in the filesystem you don't expect fail, hopefully loudly, so you can figure out why they think they need to write there and redirect to a more suitable location which either is persistent if needed or tmpfs if not necessary to persist across reboots.

One way to do the overlaying is through the initrdscripts - see https://github.com/OE4T/meta-tegra/tree/master/recipes-core/initrdscripts/tegra-minimal-init - this allows you to do overlay mounts before the rest of the system starts up. I think there are some public examples of this floating around but I couldn't find any in a quick search.

[yrodriguez31]

Hello ,
I working on the same issues , do you managed to mount the overlayfs with init-boot.sh script ?
I'm also trying to make it work with the initrd ,but i don't know if it the best way to do it .

[h0l0gr4m]

Sadly i didn't have much time to work further at this specific problem. But one other aproach we are considering now is that we won't use an overlayfs but rather have all config files on the seperate partition and rewrite the application in such a way that the config files pathes can be easily adjusted. Seems easier and more flexible.

[lfdmn]

Here's an example on the dunfell branch.

You'll need to customize the flash layout i.e: flash_l4t_custom.xml derived from flash_l4t_t186.xml.

Depending if you want to use the same overlay partition or one for each a and b slots, add something like this to your custom layout:

        <partition name="overlay" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> xxxxxxxxxx </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
            <filename> overlay.img </filename>
        </partition>
        <partition name="overlay_b" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> xxxxxxxxxx </size>
            <file_system_attribute> 0 </file_system_attribute>
            <allocation_attribute> 0x8 </allocation_attribute>
            <percent_reserved> 0 </percent_reserved>
            <filename> overlay.img </filename>
        </partition>

create tegra-binaries_32.%.bbappend with:

FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
SRC_URI += " \
    file://flash_l4t_custom.xml \
"

do_preconfigure:append() {
    mkdir -p ${S}/bootloader/${NVIDIA_BOARD}/cfg/
    cp ${WORKDIR}/flash_l4t_custom.xml ${S}/bootloader/${NVIDIA_BOARD}/cfg/
}

PARTITION_LAYOUT_TEMPLATE must be set to the new flash layout file name for the MACHINE config. In a layer distro you could add this to set the variables without modifying the meta-tegra MACHINE definitions

include conf/machine-extra/${MACHINE}-extra.conf

Then in a layer conf/machine-extra/jetson-tx2-devkit-extra.conf add

PARTITION_LAYOUT_TEMPLATE = "flash_l4t_custom.xml"
TEGRA_BUPGEN_STRIP_IMG_NAMES = "overlay.img"

Adding overlay.img to TEGRA_BUPGEN_STRIP_IMG_NAMES strips the name form the flash xml definition during the signing step of the image build. Without it, the image creation will fail.

The overlay.img image file needs to be created during the image build. For example add this to your image definition i.e: my-image.bb

tegraflash_custom_pre() {
    dd if=/dev/zero of=overlay.img bs=1M count=10
    /sbin/mkfs.ext4 overlay.img
}

With the overlay image ready, you can now bbappend tegra-minimal-init_%.bbappend

FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
RDEPENDS_${PN} += " \
    keypressed \
    findutils \
    util-linux-lsblk \
    util-linux-blkid \
"

and customize your init-boot.sh

#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
mount -t proc proc /proc
mount -t devtmpfs none /dev
mount -t sysfs sysfs /sys

slotsfx=""
$factory_reset=""

if [ -z "$rootdev_ro" ]; then
    for bootarg in `cat /proc/cmdline`; do
    echo "${bootarg}"
    arg=`expr "x$bootarg" : 'x[^=]*=\(.*\)' || echo ''`

        case "$bootarg" in
            boot.slot_suffix=*) slotsfx="${arg}" ;;
            factory_reset=*) factory_reset="${arg}";;
        esac
    done
fi

# Resize partitions to full volume size
echo "Resizing overlay partitions"
overlay_p1=`blkid -l -t PARTLABEL=overlay | cut -d: -f1`
overlay_p1_tmp="/tmp${overlay_p1}"
echo "overlay partition a: $overlay_p1 - $overlay_p1_tmp"

mkdir -p "${overlay_p1_tmp}"
mount -t ext4 -o "rw" "${overlay_p1}" "${overlay_p1_tmp}"
resize2fs ${overlay_p1}
umount "${overlay_p1_tmp}"
rm -fr "${overlay_p1_tmp}"

overlay_p2=`blkid -l -t PARTLABEL=overlay_b | cut -d: -f1`
overlay_p2_tmp="/tmp${overlay_p2}"
echo "overlay partition b: $overlay_p2 - $overlay_p2_tmp"
mkdir -p "${overlay_p2_tmp}"
mount -t ext4 -o "rw" "${overlay_p2}" "${overlay_p2_tmp}"
resize2fs ${overlay_p2}
umount "${overlay_p2_tmp}"
rm -fr "${overlay_p2_tmp}"

mkdir -p /mnt/lowerdev
mkdir -p /mnt/overlaydev
mkdir -p /mnt/overlay

rootfs=`blkid -l -t PARTLABEL=APP$slotsfx | cut -d: -f1`
overlay=`blkid -l -t PARTLABEL=overlay$slotsfx | cut -d: -f1`
echo "rootfs: $rootfs"
echo "overlay: $overlay"

# Mount image and overlay partitions
echo "Mouting partitions for slot_suffix ${slotsfx}"
mount -t ext4 -o "ro" "${rootfs}" /mnt/lowerdev
mount -t ext4 -o "rw" "${overlay}" /mnt/overlaydev

# Factory reset
if [ "$factory_reset" == "1" ] || [ -f "/mnt/overlaydev/upper/boot/factoryreset" ]; then
    echo "Applying factory defaults..."  
    rm -fr /mnt/overlaydev/*
fi

# Upper and work folders must be on the same mount
mkdir -p /mnt/overlaydev/upper
mkdir -p /mnt/overlaydev/work

mount -t overlay overlay -o lowerdir=/mnt/lowerdev,upperdir=/mnt/overlaydev/upper,workdir=/mnt/overlaydev/work /mnt/overlay

echo "Switching to rootfs on overlay..."
mount --move /sys  /mnt/overlay/sys
mount --move /proc /mnt/overlay/proc
mount --move /dev  /mnt/overlay/dev

exec switch_root /mnt/overlay /sbin/init

Finally, enable the overlay kernel module in the kernel config for the linux-tegra recipe:
files/enable_overlay.cfg

CONFIG_OVERLAY_FS=y

linux-tegra_4.9.bbappend

FILESEXTRAPATHS:prepend := "${THISDIR}/files:"

# Fragments
###############################################################################
SRC_URI += " \
    file://enable_overlay.cfg \
"

[yrodriguez31]

Thanks for you'r help . In my case i want the overlay into ram so i don't think i have to change the xml . But you're example it's gonne be very helpfull

[dwalkes]

I just noticed/remembered yesterday there is a recipe for data overlay at https://github.com/OE4T/tegra-demo-distro/tree/master/layers/meta-tegrademo/recipes-demo/data-overlay-setup as well. I haven't used it myself.

[madisongh]

There's also a somewhat more advanced example in my test distro that implements systemd.volatile=overlay to provide the full overlay for the rootfs.

[albertbu]

Sorry for reopening this thread...
I am building now a system with overlay FS.
I wasn't able to add a custom layout, so I am using UDA to hold the overlay FS.
Now, how can I add the overlay image file to the final tegraflash.tar.gz?
I understand I can copy i manually, but it would be nice to bring it in with all the tegra stuff
I am using initrd-flash of course
Thanks!