Prefetch fails when v2.1.4 nydusd prefetches data against images built by 2.2 nydusify

[changweige]

https://github.com/containerd/nydus-snapshotter/actions/runs/4549441536/jobs/8021516689?pr=434

Not sure about whether v2.2 nydusd also has this problem

index-sha256:67cdf74618bf51208f7a603b120c1dc8d0967f1f3ab6fd4d6e661886395d618c:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:de225a68da5c5ab33a704fb2f51abef302f45fc8982843d5669b53ff27c8d9c1: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:8694b779040bfa51dd267866bb82d12039c689721be4731397e22c103614cd6b:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.1 s                                                                    total:  2.2 Ki (730.0 B/s)                                       
time="2023-03-30T01:52:04.163671209Z" level=info msg="[View] snapshot with key default/43//tmp/initialC3454911934 parent default/38/sha256:d40e028a61a55d02904a711d651027e8977a2b53d9817d76ed77849cfb9ffdf5"
time="2023-03-30T01:52:04.169756410Z" level=info msg="Prepares active snapshot default/44/b89d801d07d7232e2e93790b85231811ac4e760bbce7e0d31b8ced429796283c, nydusd should start afterwards" key=default/44/b89d801d07d7232e2e93790b85231811ac4e760bbce7e0d31b8ced429796283c parent="default/38/sha256:d40e028a61a55d02904a711d651027e8977a2b53d9817d76ed77849cfb9ffdf5"
time="2023-03-30T01:52:04.169892212Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/41/work upperdir=/var/lib/containerd-nydus/snapshots/41/fs lowerdir=/var/lib/containerd-nydus/mnt/35]"
time="2023-03-30T01:52:04.177340936Z" level=info msg="[Mounts] snapshot default/44/b89d801d07d7232e2e93790b85231811ac4e760bbce7e0d31b8ced429796283c ID 41 Kind Active"
time="2023-03-30T01:52:04.178025648Z" level=info msg="Nydus remote snapshot 35 is ready"
time="2023-03-30T01:52:04.178505356Z" level=info msg="remote mount options [workdir=/var/lib/containerd-nydus/snapshots/41/work upperdir=/var/lib/containerd-nydus/snapshots/41/fs lowerdir=/var/lib/containerd-nydus/mnt/35]"
b89d801d07d7232e2e93790b85231811ac4e760bbce7e0d31b8ced429796283c
testing pull_remove_multiple_images
time="2023-03-30T01:52:04.223324102Z" level=info msg="[Cleanup] orphan directories [/var/lib/containerd-nydus/snapshots/40]"
[2023-03-30 01:52:04.269928 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob f415e61a43940212dc48d47f9b77ba8065a0fb1c4b229768c79dcbfc6a026375, offset 0, size 5827, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.270164 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob c0fe5f937e550126e01094a1b230e9492f361f6c90bfeee549df9da143be6889, offset 16777216, size 2000045, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.321113 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 1df7b6a788d93fbfa8526bbcff4c4b8422359d0a34504899cf693f6e1c7be0e5, offset 10485760, size 605722, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.321336 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 3880baf688d4d54302d9c370763723c3993841589cf011eed31ccd1c7102a717, offset 0, size 6017, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.362314 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 57222fda899cf4a19e3b0b4ac821319406978051d0c580903c4ff106eb4ac544, offset 0, size 5697, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.362682 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob bf8c53980e517d38d8704f8f667acaf57ecfd7d751804c2e1ff2a75a1870e621, offset 10485760, size 138518, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.362724 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 75e610b064b03b2d5030a0f352adbcc2e81b73dda90dac3da015cc389f5c9573, offset 0, size 6424, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.362761 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob 4188b2a65f5fe271081d125564026fb5fa42f259feafaae2eab2ae388fb489ea, offset 0, size 9415, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.480641 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob d21e899e83f0172a888d95973b18ddf4ec1ce5c6ce6ac5274f97497c1501c2d8, offset 18874368, size 915909, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
[2023-03-30 01:52:04.560149 +00:00] WARN [storage/src/cache/worker.rs:364] storage: failed to prefetch data from blob e3dce768d747ebaf07f6bf0c9599212678db02e48a500407159f36bac2d36e0c, offset 10485760, size 948502, Invalid argument (os error 22): storage/src/meta/mod.rs:568, will try resend
Deleted Containers:
10bccd0afb0682a8fa5e4f338930ca4195a2d71f12f0d3d35a8dca5c65d3945c
3933d0668586e81cbc722129c79579799f22b0c26ae0f179babdba060de87875
b89d801d07d7232e2e93790b85231811ac4e760bbce7e0d31b8ced429796283c