multi-gpu selection

[mhoibo]

When having multiple gpus, FastPathology seems to automatically choose the first gpu, even when it is not available (in use). It is possible to make it so that is will automatically choose the first available gpu?

[andreped]

I guess this happens because on the AICAN server we have 3 identical GPUs, and FAST might just take the first one, as all are of equal quality. Might be that FAST does not take into account GPU memory usage at the point of selection. Any comments, @smistad?

[smistad]

Since you are using NVIDIA GPUs, the simplest answer to this question is to use the CUDA_VISIBLE_DEVICES environment variable.
Let say you want to use the first GPU, you can run FastPathology like so:
CUDA_VISIBLE_DEVICES="0" /opt/fastpathology/bin/fastpathology

If you want use the second GPU instead you can do so:
CUDA_VISIBLE_DEVICES="1" /opt/fastpathology/bin/fastpathology

This environment variable will simply hide the GPUs that are not listed.

You can check which GPU is busy and idle on the server with the nvidia-smi command.

[andreped]

Yes, we did that as a workaround and it resolved the issue, but IMO it is not a suitable solution for non-technical personell.

It can also be an issue for long-running tasks. Suddenly, when working, someone might occupy the same GPU after FP has been launched, and then to avoid OOM issues, the user would need to chance the env and restart FP.

[smistad]

Most non-technical personnel don't have multiple GPUs.
There are no good methods for cross-platform/cross-vendor checking memory and processor usage on GPUs.
There are some CUDA methods for doing it (cudaGetMemInfo), but this will not work for Intel, Apple Silicon, and AMD GPUs. We could add that to the TensorRT inference engine though.
But as you say you will always have the possibility that you or another user will start a process on the same GPU after you have started it.
I don't think there is any way to guard completely against this, except using schedulers, queuing and containers/virtualization, and then you are way over what even technical personnel can handle.

[andreped]

Thanks for the clarification. I agree with @smistad then.

At least it was nice that this was reported and that a solution was added here. Will likely help someone in the future who experience a similar issue.