multiprocessing gt_masks and gt_info calculation

[njsymanz]

Describe your feature request

Currently, the calc_gt_masks and calc_gt_info functions (and presumably also calc_gt_coco) in BopWriterUtility are exceptionally slow for scenes with many (100+) objects. I noticed some improvement with the switch to pyrender, but it is still taking several days to generate 5000 images of a custom dataset, and most of the time is spent on gt mask generation. My current workaround is to set "calc_mask_info_coco" to False when writing data, split the dataset into several folders, then run a bunch of instances of calc_gt_masks from the bop_toolkit at the same time. Ideally, though, I would prefer the multiprocessing be done automatically while generating the dataset so the CPU isn't underutilized.

Describe a possible solution

No response

[cornerfarmer]

Hey @njsymanz,

thanks for the hint. Can you give a bit more infos about your scene? How long does the bop writer take for one image? In the bop_object_on_surface_sampling example, the bop writer takes ~2.8s per frame, so rendering 5000 images shouldnt take multiple days.
Could you maybe give a small code snippet which reproduces the slow speed or is the only reason the number of objects?

[njsymanz]

The scenes I am generating generally have up to 300 visible objects to be labelled. I currently render 10 frames at a time, then write bop data, which takes 1-2 minutes to render all 10 at 1024x1024px resolution, then 10-15 minutes to get through bop_writer. I have delta set to 1mm rather than 15, and I am writing with PNG filetype. Besides those two and the number of objects, I am using the write_bop function just like in the example.

[cornerfarmer]

@njsymanz I added an implementation of the multiprocessing bop writer in #996. For 300 objects it should bring down the execution time from 60s to 12s with 8 cores. Could you test whether you can reproduce this?

This speedup is also caused by an improvement to the bop toolkit itself (thodan/bop_toolkit#105). So make sure to remove your current bop_toolkit_lib installation first, such that the newest version will be installed on the next blenderproc run:

blenderproc pip uninstall bop_toolkit_lib

[njsymanz]

Fantastic. On an i9, gt_masks and gt_info alone are about 3x faster. I had the coco annotations function commented out before, but the improvement in the toolkit was enough that I can leave it be now (<10s for 5000 annotations as opposed to 10 min)! Factoring that in, it is running about 5x faster for me as well. Thank you so much!