This project is heavliy based on MaxSquareLoss and Adaptation Seg, pytorch-deeplab-xception. Thanks a lot!
- "Paper: Domain Adaptation for Semantic Segmentation with Maximum Squares Loss"
- "Paper: Learning to Adapt Structured Output Space for Semantic Segmentation"
- "Paper: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation"
A PyTorch implementation for our NCSU GEARS summer research program experiment. The segmentation model is based on DeeplabV2 and DeeplabV3+ with ResNet-101 backbone. DeeplabV3+ models achieves competitive results on SYNTHIA-to-CITYSCAPES adversarial domain adaptation task. We also split the cityscapes dataset into two parts and apply output-level adversarial domain adaptation, hoping to see if the performance can be improved through our used domain adaptation method even two domains are quite similar. However, experimental results show that this adaptation hurts the performance. Due to time constraints
Due to time constraints, we didn't tune the model to its best performance and improve the visual results. This may be left as our future work.
Poster of this project(vertical)
Poster of this project(horizontal)
Report of this project(note: some of the words and formats are borrowed from the listed papers)
Evaluation letter from Prof.Chi
If you use this code in your research, please cite aforementioned three git repos according to their requirements. Our code are free to use.
These instructions below will get you a copy of the project up and running on your local machine for development and testing purposes.
Install the following
- Python >= 3.6
- Pytorch >= 1.0
Then, install python packages:
pip install -r requirements.txt
Those datasets involved in this project are:
- SYNTHIA-RAND-CITYSCAPES . It consists of 9,400 1280 x 760 synthetic images. We resize images to 1280x760.
- Cityscapes. It contains 5,000 annotated images with 2048 x 1024 resolution taken from real urban street scenes. We resize Cityscapes images to 1024x512.
Make sure to download them and move to datasets/ folder.
Those datasets retain its orginal organization.
-
Many pretrained models on SYNTHIA and CITYSCAPES datasets. Coming soon.
Make sure to move the Deeplabv2_resnet101pretrained model (DeepLab_resnet_pretrained_init-f81d91e8.pth) to pretrained_model folder.
Pretraining on CITYSCAPES
- single + DeeplabV2
python3 tools/train_source.py --gpu "0" --dataset 'cityscapes' --checkpoint_dir "/nfs/byf/log/city_oracle_sv2" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 19 --multi "False" 2. multi + DeeplabV2
python3 tools/train_source.py --gpu "0" --dataset 'cityscapes' --checkpoint_dir "/nfs/byf/log/city_oracle_mv2" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 19 --multi "True"- single + DeeplabV3
python3 tools/train_source.py --gpu "0" --dataset 'cityscapes' --checkpoint_dir "/nfs/byf/log/city_oracle_sv3" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 19 --multi "False" --backbone "deeplabv3_plus" --batch_size_per_gpu 2NOTE: usually checkpoint_dir is "./log/your_dir_name"; the num_classes can be 19 or 16 depending on your purpose.
Pretraining on SYNTHIA
- single + DeeplabV2
python3 tools/train_source.py --gpu "0" --dataset 'synthia' --checkpoint_dir "/nfs/byf/log/synthia_pretrained_sv2" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 16 --multi "False"- multi + DeeplabV2
python3 tools/train_source.py --gpu "0" --dataset 'synthia' --checkpoint_dir "/nfs/byf/log/synthia_pretrained_mv2" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 16 --multi "True"- single + DeeplabV3+
python3 tools/train_source.py --gpu "0" --dataset 'synthia' --checkpoint_dir "/nfs/byf/log/synthia_pretrained_mv2" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 16 --multi "True"Adversarial domain adaptation from SYNTHIA to Cityscapes
- single + DeeplabV2
python3 tools/solve_synthia_adv.py --gpu "0" --dataset "cityscapes" --checkpoint_dir "/nfs/byf/log/synthia_city_v2_single" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --backbone 'deeplabv2_multi' --num_classes 16 --multi False --pretrained_ckpt_file "/nfs/byf/log/synthia_pretrained_sv2" - multi + DeeplabV2
python3 tools/solve_synthia_adv.py --gpu "0" --dataset "cityscapes" --checkpoint_dir "/nfs/byf/log/synthia_city_v2_multi" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --backbone 'deeplabv2_multi' --num_classes 16 --multi True --pretrained_ckpt_file "/nfs/byf/log/synthia_pretrained_mv2" - single + DeeplabV3+
python3 tools/solve_synthia_adv.py --gpu "0" --dataset "cityscapes" --checkpoint_dir "/nfs/byf/log/synthia_city_v3_single" --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --backbone 'deeplabv3_plus' --num_classes 16 --multi False --pretrained_ckpt_file "/nfs/byf/log/synthia_pretrained_sv3" --batch_size_per_gpu 2 NOTE: please add like --round_num 5 --epoch_each_round 10 to extend traning. Usually, this does not need too much training. More information on args please see the code.
Here we evaluate our results.
- Source only + DeeplabV2
python3 tools/evaluate.py --gpu "0" --checkpoint_dir "/nfs/byf/log/synthia_pretrained_sv2/eval" --source_dataset 'synthia' --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 16 --multi "False" --pretrained_ckpt_file "/nfs/byf/log/synthia_pretrained_sv2/synthiabest.pth"- Source only + DeeplabV3+
python3 tools/evaluate.py --gpu "0" --checkpoint_dir "/nfs/byf/log/synthia_pretrained_sv3/eval" --source_dataset 'synthia' --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 16 --multi "False" --pretrained_ckpt_file "/nfs/byf/log/synthia_pretrained_sv3/synthiabest.pth" --backbone "deeplabv3_plus"- Adaptation + DeeplabV2
python3 tools/evaluate.py --gpu "0" --checkpoint_dir "/nfs/byf/log/synthia_city_v2_single/eval" --source_dataset 'synthia' --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 16 --multi "False" --pretrained_ckpt_file "/nfs/byf/log/synthia_city_v2_single/synthia2cityscapesbest.pth " --backbone "deeplabv2_multi"- Adaptation + DeeplabV3+
python3 tools/evaluate.py --gpu "0" --checkpoint_dir "/nfs/byf/log/synthia_city_v3_single/eval" --source_dataset 'synthia' --iter_max 200000 --iter_stop 80000 --freeze_bn False --weight_decay 5e-4 --lr 2.5e-4 --crop_size "1280,720" --num_classes 16 --multi "False" --pretrained_ckpt_file "/nfs/byf/log/synthia_city_v3_single/synthia2cityscapesbest.pth " --backbone "deeplabv3_plus"- We can do sensitivy analysis by changing "--lambda-adv-target1" arg. Here's our results for different
$\lambda_{adv}$
- We also provide between-cityscapes domain adaptation(i.e., we split Cityscapes dataset into two parts, one as source domain and the other as target domain). Our assume that small shift can also be addressed by our method. However, results turn out to be negative.
-
Release our pretrained models.
-
Tune the model to get a better performance.
We thank Prof. Chi for insightful dicussions and ideas. We also thank North Carolina State University's GEARS program for providing us with such fulfilling research experience.
If you have any question, please feel free to contact
Yifan Bao, [email protected]
OR open an issue.