Error while using 'gandlf_run'

[anishshriram]

GaNDLF Version

Version information of the GaNDLF package in the virtual environment.

0.0.13-dev

Desktop (please complete the following information):

• macOS
• Version 12.1

How did you install GaNDLF
I followed the steps on the documentation website, ie the following commands:

git clone https://github.com/CBICA/GaNDLF.git
cd GaNDLF
conda create -n venv_gandlf python=3.7 -y
conda activate venv_gandlf
conda install -c conda-forge mamba -y 
mamba install -c pytorch pytorch torchvision -y
mamba install -c sdvillal openslide
pip install -e .

Dataset description
The dataset is titled RSNA_ASNR_MICCAI_BraTS2021_TrainingData_16July2021, referring to the 2021 BraTS training data.

Description from website:
Routine clinically-acquired multi-parametric MRI (mpMRI) scans of glioma, with pathologically confirmed diagnosis and available MGMT promoter methylation status (for the glioblastoma cases with such associated data). All BraTS mpMRI scans are available as NIfTI files (.nii.gz) for Segmentation, which is the use in this issue. These mpMRI scans describe a) native (T1) and b) post-contrast T1-weighted (T1Gd), c) T2-weighted (T2), and d) T2 Fluid Attenuated Inversion Recovery (T2-FLAIR) volumes.

Describe your question/problem
A clear and concise description of what issue you are facing.

I first edited the sample configuration file available in the gandlf repository titled ['config_segmentation_brats.yaml']:

patch size:
[128,128,128] to [64,64,64]

loss_function:
dcce to dc

optimizer
adam to sgd

I then executed:

python gandlf_run \
-c pathToConfigFile \
-I pathToDataCSV \
-m pathToEmptyModelDirectory \
-t True \
-d cuda

When run, it gave me the following error:

ValueError: The parameters are not the same as the ones stored in the previous run, please re-check.

I am not sure what this means and how to fix it. What should I do?

Error with surrounding context:

Traceback (most recent call last):
  File "gandlf_run", line 87, in <module>
    args.inputdata, args.config, args.modeldir, args.train, args.device, args.resetprev
  File "/gpfs/fs001/cbica/comp_space/shrirama/project/meningioma/GaNDLF/GANDLF/cli/main_run.py", line 38, in main_run
    "The parameters are not the same as the ones stored in the previous run, please re-check."
ValueError: The parameters are not the same as the ones stored in the previous run, please re-check.

[sarthakpati]

The error is saying that the parameters saved in the previous run do not match with those you are trying to run now (I guess you made a change to one of the parameters). You can mitigate this by doing the following:

python gandlf_run \
-c pathToConfigFile \
-I pathToDataCSV \
-m pathToEmptyModelDirectory \
-t True \
-d cuda \
-r True # this resets the previous run by deleting everything saved before

[anishshriram]

Thank you so much for the advice so far.

I changed the class list and the final layer and commented out the parallel training.

How may I run the latest version of gandlf? Is 0.0.13 not the latest?

[sarthakpati]

You would need to do a git clone of the repo and install from there.

[anishshriram]

I will try that and get back to you. Thank you!

[anishshriram]

So it started to work, and then it ran into another error.

RuntimeError: Exception thrown in SimpleITK ImageFileReader_ReadImageInformation: /tmp/SimpleITK/Code/IO/src/sitkImageReaderBase.cxx:105:
sitk::ERROR: Unable to determine ImageIO reader for "/cbica/home/shrirama/comp_space/project/meningioma/BraTS2021_01165/BraTS2021_01165_3.nii.gz"

I initially thought that there was a problem with the file, so I did:

test -f /cbica/home/shrirama/comp_space/project/meningioma/BraTS2021_01165/BraTS2021_01165_3.nii.gz && echo "$FILE exists."

and it definitely exists, so I do not know exactly what the problem is. I took the exact same steps to train, with the same configuration file.

(The version now is: 0.0.14-dev)

[sarthakpati]

@anishshriram - did you have any further updates on this error?

[xupengfei1009]

This is my .yaml file, but the following error occurred
RuntimeError: 0D or 1D target tensor expected, multi-target not supported

batch_size: 1
clip_grad: null
clip_mode: null
data_augmentation: {}
data_postprocessing: {}
data_postprocessing_after_reverse_one_hot_encoding: {}
data_preprocessing:
normalize: null
enable_padding: false
grid_aggregator_overlap: crop
in_memory: false
inference_mechanism:
grid_aggregator_overlap: crop
patch_overlap: 0
learning_rate: 0.001
loss_function: cel
medcam_enabled: false
metrics:
accuracy:
average: weighted
mdmc_average: samplewise
multi_class: true
subset_accuracy: false
threshold: 0.5
balanced_accuracy: null
cel: null
classification_accuracy: null
f1:
average: weighted
f1:
average: weighted
mdmc_average: samplewise
multi_class: true
threshold: 0.5
iou:
iou:
reduction: sum
reduction: elementwise_mean
threshold: 0.5
per_label_one_hot_accuracy: null
precision:
average: weighted
mdmc_average: samplewise
multi_class: true
precision:
average: weighted
threshold: 0.5
recall:
average: weighted
mdmc_average: samplewise
multi_class: true
threshold: 0.5
modality: rad
model:
amp: false
architecture: vgg16
base_filters: 32
class_list:

• 0
• 1
  data_type: FP32
  dimension: 3
  final_layer: softmax
  ignore_label_validation: null
  norm_type: batch
  num_channels: 1
  num_classes: 2
  onnx_export: false
  print_summary: false
  save_at_every_epoch: false
  type: torch
  nested_training:
  testing: 5
  validation: 2
  num_epochs: 10
  optimizer:
  type: adam
  parallel_compute_command: ''
  patch_sampler: uniform
  patch_size:
• 32
• 32
• 32
  patience: 1
  pin_memory_dataloader: false
  print_rgb_label_warning: true
  problem_type: classification
  q_max_length: 1
  q_num_workers: 0
  q_samples_per_volume: 1
  q_verbose: false
  save_output: false
  save_training: false
  scaling_factor: 1
  scheduler:
  step_size: 0.0002
  type: triangle
  track_memory_usage: false
  verbose: false
  version:
  maximum: 0.0.18
  minimum: 0.0.14
  weighted_loss: true

[sarthakpati]

Thank you for your interest in GaNDLF!

If you don't mind, can you please post your config file, the command you used, and all traceback of the error (preferably in a new discussion thread so that we can keep it independent)?

[xupengfei1009]

What loss functions are included in this tool?

[sarthakpati]

You can find all loss functions in the documentation, here: https://mlcommons.github.io/GaNDLF/customize/#loss-function

[xupengfei1009]

yaml.pdf

When I use the above configuration file for training, I encounter the following error. What could be the reason? Thank you.

File "E:\MRI\GaNDLF-master\gandlf_run", line 128, in
main_run(
File "E:\MRI\GaNDLF-master\GANDLF\cli\main_run.py", line 100, in main_run
TrainingManager(
File "E:\MRI\GaNDLF-master\GANDLF\training_manager.py", line 252, in TrainingManager
training_loop(
File "E:\MRI\GaNDLF-master\GANDLF\compute\training_loop.py", line 347, in training_loop
overall_metrics = overall_stats(
File "E:\MRI\GaNDLF-master\GANDLF\metrics_init_.py", line 120, in overall_stats
return classification.overall_stats(predictions, ground_truth, params)
File "E:\MRI\GaNDLF-master\GANDLF\metrics\classification.py", line 76, in overall_stats
output_metrics[metric_name] = get_output_from_calculator(
File "E:\MRI\GaNDLF-master\GANDLF\utils\tensor.py", line 514, in get_output_from_calculator
temp_output = calculator(predictions, ground_truth)
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torchmetrics\metric.py", line 298, in forward
self._forward_cache = self._forward_reduce_state_update(*args, **kwargs)
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torchmetrics\metric.py", line 368, in _forward_reduce_state_update
batch_val = self.compute()
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torchmetrics\metric.py", line 607, in wrapped_func
value = _squeeze_if_scalar(compute(*args, **kwargs))
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torchmetrics\classification\auroc.py", line 123, in compute
return _binary_auroc_compute(state, self.thresholds, self.max_fpr)
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torchmetrics\functional\classification\auroc.py", line 88, in _binary_auroc_compute
fpr, tpr, _ = _binary_roc_compute(state, thresholds, pos_label)
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torchmetrics\functional\classification\roc.py", line 54, in _binary_roc_compute
fps, tps, thres = _binary_clf_curve(preds=state[0], target=state[1], pos_label=pos_label)
File "C:\ProgramData\Anaconda3\envs\venv_gandlf\lib\site-packages\torchmetrics\functional\classification\precision_recall_curve.py", line 61, in _binary_clf_curve
target = target[desc_score_indices]
IndexError: index 7 is out of bounds for dimension 0 with size 5

[sarthakpati]

Hmm, it seems like roc is problematic. Have you tried running this without precision and recall, just so that we can confirm it runs?