gandlf_collectStats is not giving expected output

[smcch]

GaNDLF Version
Version: 0.0.16.

Desktop (please complete the following information):

• OS: Ubuntu (WSL)

How did you install GaNDLF
I followed the installation guide

Dataset description
MRI scans, 4 modalities, 240 x 240 x 155

Describe your question/problem
I got this error after run training:

UserWarning: Using a target size (torch.Size([1, 1])) that is different to the input size (torch.Size([1, 2, 64, 64, 64])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
loss = F.mse_loss(prediction, target, reduction=reduction)
Looping over training data: 0%| | 0/125 [00:03<?, ?it/s]
ERROR: shape '[1, 1]' is invalid for input of size 262144

My config file:

# affix version
version:
  {
    minimum: 0.0.16,
    maximum: 0.0.16 # this should NOT be made a variable, but should be tested after every tag is created
  }
## Choose the model parameters here
model:
  {
    dimension: 3, # the dimension of the model and dataset: defines dimensionality of computations
    base_filters: 30, # Set base filters: number of filters present in the initial module of the U-Net convolution; for IncU-Net, keep this divisible by 4
    architecture: resunet, # options: unet, resunet, deep_resunet, deep_unet, light_resunet, light_unet, fcn, uinc, vgg, densenet
    norm_type: batch, # options: batch, instance, or none (only for VGG); used for all networks
    final_layer: softmax, # can be either sigmoid, softmax or none (none == regression/logits)
    class_list: ['0','1'], # Set the list of labels the model should train on and predict
    amp: False, # Set if you want to use Automatic Mixed Precision for your operations or not - options: True, False
    num_channels: 4, # set the input channels - useful when reading RGB or images that have vectored pixel types from the CSV
    print_summary: True, # prints the summary of the model before training; defaults to True


## metrics to evaluate the validation performance
metrics:
    - accuracy # classification
    - classification_accuracy # classification
    - per_label_accuracy # used for classification
    - f1 # classification/segmentation
    - precision # classification/segmentation ## more details https://torchmetrics.readthedocs.io/en/latest/references/modules.html#id3
    - recall # classification/segmentation ## more details 
inference_mechanism: {
  grid_aggregator_overlap: crop, # this option provides the option to strategize the grid aggregation output; should be either 'crop' or 'average' - https://torchio.readthedocs.io/patches/patch_inference.html#grid-aggregator
  patch_overlap: 0, # amount of overlap of patches during inference, defaults to 0; see https://torchio.readthedocs.io/patches/patch_inference.html#gridsampler
}
# this is to enable or disable lazy loading - setting to true reads all data once during data loading, resulting in improvements
# in I/O at the expense of memory consumption
in_memory: False
# this will save the generated masks for validation and testing data for qualitative analysis
save_output: False
# this will save the patches used during training for qualitative analysis
save_training: False
# Set the Modality : rad for radiology, path for histopathology
modality: rad
## Patch size during training - 2D patch for breast images since third dimension is not patched
patch_size: [64,64,64]
# uniform: UniformSampler or label: LabelSampler
patch_sampler: uniform
enable_padding: False
# Number of epochs
num_epochs: 10
# Set the patience - measured in number of epochs after which, if the performance metric does not improve, exit the training loop - defaults to the number of epochs
patience: 2
# Set the batch size
batch_size: 1
# gradient clip : norm, value, agc
clip_mode: norm
# clip_gradient value
clip_grad: 0.1
## Set the initial learning rate
learning_rate: 0.001
# Learning rate scheduler - options:"triangle", "triangle_modified", "exp", "step", "reduce-on-plateau", "cosineannealing", "triangular", "triangular2", "exp_range"
# triangle/triangle_modified use LambdaLR but triangular/triangular2/exp_range uses CyclicLR
scheduler:
  {
    type: triangle,
    min_lr: 0.00001,
    max_lr: 1,
  }
# Set which loss function you want to use - options : 'dc' - for dice only, 'dcce' - for sum of dice and CE and you can guess the next (only lower-case please)
# options: dc (dice only), dc_log (-log of dice), ce (), dcce (sum of dice and ce), mse () ...
# mse is the MSE defined by torch and can define a variable 'reduction'; see https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss
# use mse_torch for regression/classification problems and dice for segmentation
loss_function: mse
# this parameter weights the loss to handle imbalanced losses better
weighted_loss: True

optimizer: adam

nested_training:
  {
    testing: 5, # this controls the testing data splits for final model evaluation; use '1' if this is to be disabled
    validation: 5 # this controls the validation data splits for model training
  }

data_augmentation:
  {
    'flip':{
      'axis': [0,1,2] # one or more axes can be put here. if this isn't defined, all axes are considered
    },
    'rotate_180', # explicitly rotate image by 180; if 'axis' isn't defined, default is [1,2,3]

q_max_length: 40
# this determines the number of patches to extract from each volume. A small number of patches ensures a large variability in the queue, but training will be slower
q_samples_per_volume: 5
# this determines the number subprocesses to use for data loading; '0' means main process is used
q_num_workers: 2 # scale this according to available CPU resources
# used for debugging
q_verbose: False

[sarthakpati]

Hi,

Thank you for your interest in GaNDLF. ☺️

It seems like you are combining a segmentation task with classification loss and metrics. Perhaps you can try to replicate the following tutorial, which describes solving a segmentation problem for 3D radiology scans?

https://mlcommons.github.io/GaNDLF/getting_started/#segmentation-using-3d-radiology-images

[smcch]

Dear Sarthak,

Thank you very much for your promt response.

I'm working on a 'Classification' task. I'm using as input 3D MRI volumes, 4 modalities per subject.

The classes to predict are: '0' and '1'

Here is my data structure:

SubjectID,Channel_0,Channel_1,Channel_2,Channel_3,ValueToPredict
0001,001_0_t1.nii.gz,0001_0_t1ce.nii.gz,0001_0_t2.nii.gz,0001_0_flair.nii.gz,0
0002,0002_0_t1.nii.gz,0002_0_t1ce.nii.gz,0002_0_t2.nii.gz,0002_0_flair.nii.gz,0
0003,0003_0_t1.nii.gz,0003_0_t1ce.nii.gz,0003_0_t2.nii.gz,0003_0_flair.nii.gz,1

I follow this section: Classification using 3D Radiology Images of the tutorial: https://mlcommons.github.io/GaNDLF/getting_started/#segmentation-using-3d-radiology-images

I'm using a modified version of the config file : https://github.com/mlcommons/GaNDLF/blob/master/samples/config_getting_started_regression_rad3d.yaml

batch_size: 1
clip_grad: null
clip_mode: null
data_augmentation: {}
data_postprocessing: {}
data_postprocessing_after_reverse_one_hot_encoding: {}
data_preprocessing:
  normalize: null
enable_padding: false
grid_aggregator_overlap: crop
in_memory: false
inference_mechanism:
  grid_aggregator_overlap: crop
  patch_overlap: 0
learning_rate: 0.001
loss_function: mse
medcam_enabled: false
metrics:
  - accuracy
  - f1
  - precision
  - recall
modality: rad
model:
  amp: false
  architecture: densenet
  base_filters: 64
  class_list:
  - 0
  - 1
  data_type: FP32
  dimension: 3
  final_layer: softmax
  ignore_label_validation: null
  norm_type: batch
  num_channels: 4
  num_classes: 2
  onnx_export: false
  print_summary: false
  save_at_every_epoch: false
  type: torch
nested_training:
  testing: -5
  validation: -5
num_epochs: 1
optimizer:
  type: adam
parallel_compute_command: ''
patch_sampler: uniform
patch_size:
- 64
- 64
- 64
patience: 1
pin_memory_dataloader: false
print_rgb_label_warning: true
problem_type: classification
q_max_length: 1
q_num_workers: 0
q_samples_per_volume: 1
q_verbose: false
save_output: false
save_training: false
scaling_factor: 1
scheduler:
  step_size: 0.0002
  type: triangle
track_memory_usage: false
verbose: false
version:
  maximum: 0.0.17
  minimum: 0.0.14
weighted_loss: true

I got this warnings:

my_path/anaconda3/envs/venv_gandlf/lib/python3.8/site-packages/GANDLF/losses/regression.py:201: UserWarning: Using a target size (torch.Size([1, 1])) that is different to the input size (torch.Size([1, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
loss = F.mse_loss(prediction, target, reduction=reduction)

my_path/envs/venv_gandlf/lib/python3.8/site-packages/torchio/data/image.py:131: UserWarning: Not specifying the image type is deprecated and will be mandatory in the future. You can probably use tio.ScalarImage or tio.LabelMap instead
warnings.warn(

Finally, after completing the training, validation, and test, when executing gandlf_collectStats I got this message: Segmentation task detected, generating dice and loss plots, and the plots generated are empty.

I appreciate your help!

[sarthakpati]

Hi,

Ah, your previous config had resunet as the model, so I thought you were trying to do segmentation. Anyway, I think you mean to use this as your starting point : https://mlcommons.github.io/GaNDLF/getting_started/#classification-using-3d-radiology-images

There is a configuration specific for classification there: https://github.com/mlcommons/GaNDLF/blob/master/samples/config_getting_started_classification_rad3d.yaml

[smcch]

Yes, it's my fault. I was referring to https://mlcommons.github.io/GaNDLF/getting_started/#classification-using-3d-radiology-images. I have actually used that configuration, making minor changes since I have 2 classes instead of 3, and my channels are 4 instead of 1. Despite that, I still get the aforementioned errors.

[sarthakpati]

Ah okay, I understand.

What are the contents of your output directory? Also, did you delete the contents of the output directory from your segmentation run [ref] before trying this?

[smcch]

Yes, I deleted the contents before.

output_dir

Subfolders 'output_testing' and 'output_validation' are empty.

[Geeks-Sid]

Please delete the entire output directory or use the reset flag available through gandlf_run. Also, If you could share the config file one more time so that we could verify the error, that would be great!

[sarthakpati]

It seems like the training has actually worked, but the collectStats function is somehow not picking it up correctly. Can you send the logs_* files as well?

[smcch]

config_getting_started_classification_rad3d.zip
model_dir.zip

[sarthakpati]

Nothing seems to be an issue here. Seems like the model has successfully trained for 1 epoch.

[smcch]

This is the output of collectStats

data.zip

[sarthakpati]

Opened an issue for this: #680

In any case, the issue is not that the training isn't working, it is the fact that the gandlf_collectStats script is not functioning as expected. You can proceed with training as you see fit, BTW.

[smcch]

Thanks !

[sarthakpati]

Hey @smcch,

Thanks for reporting this issue.

There are 2 PRs that should address this: #687

And another coming from @Geeks-Sid.

[sarthakpati]

This is now merged: #691

Can you please take a look once again? After you have trained your model for a few epochs, of course.

[smcch]

I got this error:

Traceback (most recent call last):
File "********/anaconda3/bin/gandlf_collectStats", line 13, in
from GANDLF.utils.plot_utils import plot_all
ModuleNotFoundError: No module named 'GANDLF.utils.plot_utils'

[sarthakpati]

Have you done git pull on the GaNDLF repo (when installing from sources)? Otherwise, you need to install the nightly build using pip install --pre gandlf.

[smcch]

I installed the nightly build version

[sarthakpati]

Ah, sorry about that. Can you try commenting out the line 13 in gandlf_collectStats (from GANDLF.utils.plot_utils import plot_all)?

[sarthakpati]

Update: should be fixed with #694

[smcch]

Now, It works perfect! Thanks!