DDP problem when migrate TSD to mmdet2.0

[zhaoxin111]

I am trying to migrate TSD to mmdet2.0, anything is ok when training faster_rcnn_TSD only on sigle GPU.
When I run TSD with DDP, some error happened. Similar error #2153

I have tried set find_unused_parameters=True in DDP, this makes the error not happen, but makes the program stuck.
Does anyone have any suggestions？

Traceback (most recent call last):
File "./tools/train.py", line 178, in
main()
File "./tools/train.py", line 167, in main
train_detector(
File "/home/zhaoxin/workspace/mmdetection/mmdet/apis/train.py", line 150, in train_detector
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/home/zhaoxin/tools/miniconda3/envs/torch1.6/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/zhaoxin/tools/miniconda3/envs/torch1.6/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
self.run_iter(data_batch, train_mode=True)
File "/home/zhaoxin/tools/miniconda3/envs/torch1.6/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 29, in run_iter
outputs = self.model.train_step(data_batch, self.optimizer,
File "/home/zhaoxin/tools/miniconda3/envs/torch1.6/lib/python3.8/site-packages/mmcv/parallel/distributed.py", line 49, in train_step
self.reducer.prepare_for_backward([])
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel; (2) making sure all forward function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).