Support UUID= (filesystem UUIDs), Armbian and partially Ubuntu #140
Conversation
Previously, e2label was used, but that only works for ext partitions. Using lsblk allows showing the filesystem labels for all filesystems supported by the kernel.
Now labels are retrieved for all destination partitions, there is no longer any point in separately retrieving the destination root label, just use the previously retrieved label.
Instead of only setting FS labels on ext partitions when specified with the --label-partitions option and leave all other partitions unlabeled, this tries to copy the source filesystem labels to the destination where possible. Setting labels requires filesystem-specific commands or mkfs options, so not all filesystems are supported. For changing labels on existing partitions, only ext and fat partitions are supported. For mkfs a few more are supported, though these are probably not used in practice. This also refactors some of the code, introducing a `mkfs_label()` and `change_label()` function to prevent having to duplicate the filesystem-type checking code. This fixes billw2#100.
This makes it non-interactive and prevents the script from silently
hanging when parted finds something weird, e.g.:
$ sudo rpi-clone /dev/sda
Warning: The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes.
This just hangs, because parted is waiting for an answer:
$ sudo parted -m /dev/sda unit s print
Warning: The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes.
Ignore/Cancel?
It seems that parted can force a partition scan (observed on a removable USB disk), which temporaly clears the filesystem labels from lsblk output. To prevent this, call udevadm settle to wait until all udev events are processed before continuing.
This supports systems where the filesystem's UUID is specified in fstab and on the kernel commandline. This is the first step in supporting Armbian, which uses UUID rather than PARTUUID.
matthijskooijman
force-pushed
the
support-fsuuid-and-armbian
branch
from
79071e9
to
e5ad6bf
Compare
matthijskooijman
changed the title
|
I just pushed one more commit that enables modifying @ioogithub, Could you test this? |
|
To confirm, can I test this PR with the following: |
|
Yup, that sounds like the correct approach. |
|
I have tested the code, at first it appeared that it had worked because I got this result at the end of the clone: however when I inspected I ran the test again with the modified code (above) and I confirmed that it did work. Im not sure why this change did not work, from looking at your code it looks like it should and it even reported that it made the change so perhaps there is something else that overwrites it back afterwards? |
Hm, weird. Did you check that the actual PARTUUIDs were different (or alternatively, that the PARTUUID show in the log output really did not match cmdline.txt)?
You mean you ran the test a second time without changing anything? Or is your "above" referring to the other issue where you posted a manual change of the cmdline.txt file? |
I did notice another unusual event, several hours later I noticed that after a restart I was booted from the /dev/sdb which was was completely unexpected. Is it possible that the code changed the cmdline.txt on the source card and not the destination? I will do another test, before the test I will check both fstab and cmdline.txt then I will check then again after and report back. |
Yup, that is exactly what happened, thanks for catching that. Turns out the UUID version prepends I'll push a correction later, maybe tomorrow. |
I've spent a lot of time wrestling with trying to get this super useful script which used to work easily with Raspberry Pi OS to use instead with Ubuntu 22.04 on a Raspberry Pi 4 (nothing I tried would get it to boot after cloning with the original script), I finally found your branch and all is now working as expected. I can clone from |
I've just pushed a fixup commit to fix this issue. I haven't tested it, @ioogithub maybe you can check if it works for you now? @diagonali, good to hear this branch has been useful for you. I'm assuming you have change |
|
@matthijskooijman To be honest, despite tinkering with linux for a quite a while now, I haven't fully understood I simply installed a fresh Ubuntu 22.04 onto a Raspberry Pi 4 using USB boot and ran your branch of this script to clone to an SD card. Then I removed the USB drive and rebooted and the system boots successfully. Then I plugged in the USB drive and cloned back to that and removed the SD card and again it boots successfully. As far as I can tell there is some new messaging (relating to labels I think?) during runtime which I assume is due to your edits. Also I noticed that running the script with -f flag doesn't seem to do the same thing(s) as running without this flag in terms of making the cloned drive bootable? I didn't get the same issue as the #140 comment as far as I can see.
I'm using the following commands to use your branch - use this again to update?: |
Ah, you're only keeping either the USB drive or SD card plugged in, that should work even without manual
You could probably also get away with a simple |
@matthijskooijman Interesting, I definitely want to be able to keep the SD card inserted when using USB boot (and vice versa) as the SD card operates in my case as a "live" bootable failsafe "backup" in case of issues with the USB drive/installation. It makes sense from what you say that it could boot one but use the filesystem of the other - not ideal obviously! Question for me now is how to as you say "convert fstab and cmdline.txt to UUIDs". I'll start looking into (Googling) how to do this in Ubuntu on the Raspberry Pi and I'm sure there are guides online. I'm wondering if you could give any simple instructions if this is a simple process? I'll be doing this on a live system so want to make sure I do this correctly. I use my Raspberry Pi as a home automation server so the setup I have needs to be as robust as possible and I avoid the boot/filesystem overlap you mention. Also I'm wondering if this process could be incorporated into the script itself? Is switching over to using UUIDs too tricky to process in an automated way? |
|
@diagonali See the discussion starting at #100 (comment)
It could probably be done (there is already an option and code to switch from device names (e.g. |
Ah ok no problem, thanks very much for what you've done so far. |
This removes all hardcoded references to cmdline.txt and cmdline.boot by removing the `$clone` prefix from cmdline_txt and introducing a cmdline_txt variable. In addition, this adds some quoting to any lines modified to be a bit more robust when spaces are involved. This should not change behavior (apart from maybe in cases with spaces that were previously broken), the commands ran should be identical.
On Armbian, the kernel commandline (or at least the root device to use) is stored in armbianEnv.txt rather than cmdline.txt, so update that if it exists and cmdline.txt does not. The actual operation on this file is just replacing UUIDs inside the file, so it works even though the syntax of the file is different.
On Ubuntu, cmdline.txt is stored in a subdirectory, so if the file does not exist in the usual location, fall back to the Ubuntu location (and fall back to the Armbian version if the Ubuntu version does not exist either). Note that this does not support Ubuntu completely out of the box, since it uses filesystem labels instead of UUIDs for mounting, but if you manualy change them to UUIDs, this change allows cloning to work.
This mostly moves code into functions, but also makes minor changes to how the code is called to prepare for more reuse of this code. Moving these functions up in the file with the other functions is left for the next commit to make review easier. Diff best viewed with --ignore-all-space.
This puts them together with the other function. This commit only moves code, without making any changes.
This applied largely the same operations to cmdline.txt as fixup_device_references_in_file did to fstab, so this can just call the latter. This should work the same, except that the PARTUUID is not replaced with the g flag (all occurences instead of just one), all filesystem UUIDs are replaced (instead of just the root partition), and progress output messages are slightly different. Also, fstab UUID= replacements are now done anywhere in the file, instead of just at the start of a line.
This modifies fixup_device_references_in_file to accept the mountpoint and filename separately to prevent printing the destination mountpoint (which might be ugly and meaningless). fixup_boot_partition already had this. Then, a partition label is passed which is prefixed to this plain filename to tell the user which partition is being modified. This could just have used src_mount, but passing a separate label prepares for using this code for non-mounted partitions too. Finally, the UUID output is slightly reworded and some indentation is added (preparing for this output to also be intermixed with mounting and rsync calls later).
This facilitates the case where some of the non-mounted partitions contain a second operating system (possibly for a different system/board), and updates these so that system also stays working.
Since this is called for fstab on unmounted partitions now, without checking if the file exists, just silently do nothing if the file does not exist.
This adds some consistency with fixup_boot_partition, making code easier to read, and removes duplication of the list of files to update (currently only fstab).
matthijskooijman
force-pushed
the
support-fsuuid-and-armbian
branch
from
628e542
to
94b2bcc
Compare
|
I did a bit more work on this PR to make rpi-clone update UUIDs in non-mounted partitions as well (plus some refactoring leading up to that). I needed this to work with a dual-boot image/SD-card I have been working with (single image that contains boot+root for both a raspberry pi and an orange pi), but it might be useful for other cases as well. I did a force push to squash the fixup commit I previously pushed. Edit: And one more commit to fix permission warnings on FAT filesystems mounted with the uid option, which is really unrelated but I just wanted to publish it somewhere without adding yet another PR |
|
Ik ben tot en met vrijdag 5 augustus 2022 afwezig en zal mijn email niet of beperkt kunnen lezen en beantwoorden. Neem voor dringende zaken contact op met mijn collega André de Jonge, email ***@***.***, tel. 0224-297819.
Met vriendelijke groet,
FSE Turnstiles b.v.
Jos de Vries
|
This can potentially cause rsync error messages, especially if the filesystem is already mounted with a uid= option (which will not be copied to the destination mount, leading to different owners and thus rsync trying and failing to fix the owner).
|
I know it's been some time since someone commented on this but I'd really like to thank you. It took me a lot of time this evening and your fork was a solution to this. Thank you :) |
|
I have an Orangepi 3LTS running Armbian 23.8.1 Bullseye with Linux 5.15.93-sunxi64. sys-clone works perfect as long as I have the system on a SD card. I moved the system to the EMMC. The initial sync works, but sys-clone fails to do a subsequent sync complaining that it can not mount the clone disk: `sudo sys-clone sdc Destination disk partition /dev/sdc1 is mounted on /mnt/clone. Booted disk: mmcblk2 7,8GB Destination disk: sdc 31,3GBPart Size FS Label Part Size FS Label
|
| Verbose mode : no. |
|---|
Ok to proceed with the clone? (yes/no): yes
Syncing file systems (can take a long time)
Syncing mounted partitions:
e2label /dev/sdc1 --
Mounting /dev/sdc1 on /mnt/clone
mount: /mnt/clone: wrong fs type, bad option, bad superblock on /dev/sdc1, missing codepage or helper program, or other error.
Mount failure of /dev/sdc1 on /mnt/clone.
Aborting!`
I have tested this with two separate SD cards, and same error occurs. Is this a bug or user error?
|
Hm, so it fails to mount Btw, I think this might be unrelated to this particular pullrequest - Yes, you're using Armbian, but the changes in this PR are mostly about updating filesystem references for Armbian, and your problem seems to occur way earlier before even starting to copy files. So maybe it would be better to open up a separate issue for this? Maybe comment with a link to that issue here? |
|
This still does not work after updating to Armbian 23.8.3. Anyone else seeing this behavior, or is this a problem with my setup? |
This makes two changes:
With these changes, I was able to run rpi-clone on an Orange Pi PC running Armbian (but I expect it to run on most if not all other Armbian-supported boards as well). I tried both a stock image with a single partition, as well as a self-compiled image with 2 partitions (/boot and /). I also tried imaging SD to USB with
-l, which then boots from USB succesfully, and then swapping out the SD card and image back from USB to SD (without-l), which produces a working SD again.One thing I did not implement is changing the FS UUID for partitions that are copied using
dd, so for these partitions you might end up with duplicate UUIDS. I'm not exactly sure when this happens (I think only for partitions that are not mounted, so maybe the duplicate UUIDs are not problematic directly), but this might need a better looking at.Note that this PR includes the commits of #101, since I needed both so I developed this on top of that and because they touch nearby code, there's merge conflicts when trying to untangle them (which is possible, but would only result in more conflicts when trying to merge both, so I thought it better to just leave them entangled). Effectively this means only the last three commits should be reviewed here, the first 5 are subject of #101.