PoC Mavic 3M RGB unwarping

[pierotofy]

Opening early to encourage testing.

Since the Mavic 3M's RGB sensor differs significantly from the multispectral sensor, image alignment fails. We actually currently drop the RGB band to allow the program to at least process the rest of the bands.

This PR unwarps the RGB images before processing using the EXIF dewarp data field which is provided by DJI and rescale the images to match the focal lengths between the two sensors.

The images this way are pretty close, although on very small datasets (5 images) this still yields some alignment issues.

I will test with larger datasets in the upcoming days.

[danieldobosi]

good day, what can I do to help closing this issue?

[pfxuan]

Previously, I tested this PR with a series of DJI M3M datasets (50 ~ 400 acres). Everything works perfectly so far. However, one aspect I'm uncertain about is whether we need to execute the downstream calibration/unwarping from OpenSfM after applying unwarping on the source images.

[pierotofy]

Thanks for testing @pfxuan ! The unwarping from OpenSfM is needed because the reference images are unwarped also (and they need to align).

This PR needs more testing, in my (small) experiments this did not completely solve the alignment issue and since this PR introduces some complex logic, I'm hesitant to merge it unless it really shows some good improvements over baseline.

[pfxuan]

Thanks for explaining the OpenSfM unwarping part! From my previous tests, I found the following setup (Incremental + 2.5D mesh) is working particularly good in a high density of vegetation area. It can produce a better orthos than pix4d and metashape does. However, I was not able to produce a similar result using Triangulation and Plannar SfM-algorithm.

In addition, I found the names of DJI GPS tags are inconsistent across different DJI models and probably firmwares. e.g. in M3M, EXIF meta shows @drone-dji:GpsLongitude rather than @drone-dji:Longitude. I did some tests with the updated code in photo.py. But it seems like there is no improvement for Triangulation SfM-algorithm.

/code/run.sh
                --cog \
                --orthophoto-resolution 1.0 \
                --max-concurrency 20 \
                --use-hybrid-bundle-adjustment \
                --fast-orthophoto \
                --sfm-algorithm incremental \
                --feature-type dspsift \
                --feature-quality high \
                --min-num-features 45000 \
                --gps-accuracy 1 \
                --build-overviews \
                --auto-boundary

[Saijin-Naib]

In addition, I found the names of DJI GPS tags are inconsistent across different DJI models and probably firmwares

This is a never-ending nightmare that impacts practically every tag, including altitude, offsets, etc. Sorry this is impacting your work on this -_-

[smathermather]

Ok, digging deeper and still well out of my depth, I think the missing step is maybe using the Dewarp HMatrix, which is also in exif:

[balthazarneveu]

From what I see in yours screenshot , looks like the calibration homography (close to identity = just a bit of misalignment) is the one you'll need to apply in the warp. This looks good if DJI provides this.. 95% shall be fixed ... The rest will be tiny parallax and unperfect Calibration ✌️✌️✌️🚀🚀🚀

[smathermather]

From what little I can discern for the process, I assume we'd apply a cv2.warpPerspective after cv2.initUndistortRectifyMap, or would we somehow bundle those transformations together?

[lewispb]

I've been playing with getting a good alignment locally and came up with this, based on the DJI image processing guide. I'm a newbie when it comes to CV and Python though so there's likely issues.

https://gist.github.com/lewispb/6c7dd6022ba3266cfd4f1310d4d51f5f

One thing I haven't attempted is using the sun sensor data for calibration. Seems to only be present in the MS images XMP, not RGB.

I've been testing this branch locally but was intermittently running into several stack overflows at the final orthophoto stage. I can share a dataset if it's helpful, or help with testing.

[smathermather]

I've been playing with getting a good alignment locally and came up with this, based on the DJI image processing guide. I'm a newbie when it comes to CV and Python though so there's likely issues.

https://gist.github.com/lewispb/6c7dd6022ba3266cfd4f1310d4d51f5f

This is great! Reading through (as a non-expert) this looks pretty solid. Would you be willing to do a pull request to integrate? As far as I can tell, just integrating the parse_homography_matrix and then use that padded warp_matrix in the align_image function prior to writing out would do the trick. But I'm no expert either, so happy to be corrected on approach.

Testing data would be great, and I can do some testing too. Are your overflows memory overflows, or a race condition, or something else? I can run a few different docker images with different memory allocated up to 1.5TB, if we need to do any testing on memory needs.

[smathermather]

I'm getting segmentation faults when I run your code snippet headless. I'm not sure if that's an opencv module that requires a gui or an issue with my configuration, but I thought I'd mention it.

Crunching though some test data now.

[balthazarneveu]

@lewispb you seem to be on the right track.
One point whereI hope I can help:

you can do both warping operations at once:

https://github.com/wisescootering/infrareddrone/blob/master/registration/warp_flow.py#L95-L138
cv2.initUndistortRectifyMap & cv2.remap allows warping with a generic flow ... so perform homography + distortion at once (which avoid 2 geometric interpolations).

[smathermather]

@lewispb -- do you have some test data to share? Perhaps unsurprisingly, my test data doesn't match your tags. It'd be good to cross walk them.

[smathermather]

Disregard: I have a matching dataset. And it works great.

[smathermather]

This looks quite good:

I had to disable multiprocessing, and we might have datasets that don't have the appropriate tags, in which case, we may have to fall back to DJI's published expected values.

@lewispb -- are you ok with licensing this GNU Affero v3 ala the repo: https://github.com/OpenDroneMap/ODM/blob/master/LICENSE?

Alternatively a pull request integrating it would be even better. :) Thanks for sharing this!

[lewispb]

@lewispb -- are you ok with licensing this GNU Affero v3 ala the repo.

Of course, that's fine.

I'm not sure I if I'd be able to get a PR ready - I already attempted an integration with the work started here but struggled to get past / understand seg faults in the final orthophoto stage. If I get some time this week I'll share more on those.

This is what I managed BTW - I just used the MS bands, not the RGB:

[smathermather]

I need to get back to testing, but traveling is slowing me down. I did just see this on the community forum, so I'll drop it here in the interest of keeping the conversation in one place:

https://gitlab.com/Yario/image_registration_dji_mavic_3m/-/blob/main/coregistration_mavic_3m_images_with_CV2.py?ref_type=heads

https://community.opendronemap.org/t/pre-processing-script-for-multispectral-images-from-the-dji-mavic-3m/20743

[MarioTro]

Hi there, I posted that comment and code smathermather linked above.
I was not aware of the discussion here and it looks like I did some double work.
What my script currently does is take the green image as reference and then apply cv2.MOTION_HOMOGRAPHY .
It is quite slow but adding some parallelization will hopefully speed things up.

You can tune the paramters number_of_iterations and termination_eps, which determine the "loop that aligns the images". But when you lower these numers you will run into issues when your images are quite homogeneous (grassland in my case). Increasing the number on the other side will slow things down even more.

I ran it on two datasets and it seems to work fine.

[smathermather]

I ran it on two datasets and it seems to work fine.

Cool! Diving in now.

@MarioTro: Am I reading it correctly that it is calculating the homography and then applying it? If so, have you compared the results to using the homography published in the exif?

[pierotofy]

Progress! I'm closing this in favor of #1771.