Assistance troubleshooting CVMFS - long timeout on startup

[timrosenow]

Hey.

I was wondering if you could provide some insight on how best to troubleshoot CVMFS on startup? I'm running neurodesktop locally on my Fedora desktop and it takes around 15-20 mins to attempt to mount CVMFS on container startup (hangs at "Mounting CVMFS"), then again when testing for fastest server, then eventually connects after 30-40 mins. It isn't purely a network latency issue as here is the output of server testing after it does connect:

    ==================================================================
    Testing which CVMFS server is fastest
OK
  [0] http://cvmfs.neurodesk.org/cvmfs/neurodesk.ardc.edu.au (48 ms)
  [1] http://152.67.114.42/cvmfs/neurodesk.ardc.edu.au (48 ms)
  [2] http://203.101.231.144/cvmfs/neurodesk.ardc.edu.au (65 ms)
  [3] http://140.238.170.185/cvmfs/neurodesk.ardc.edu.au (240 ms)
  [4] http://150.136.239.221/cvmfs/neurodesk.ardc.edu.au (261 ms)
  [5] http://132.145.96.34/cvmfs/neurodesk.ardc.edu.au (289 ms)
  [6] http://130.61.74.69/cvmfs/neurodesk.ardc.edu.au (291 ms)
Active host 0: http://cvmfs.neurodesk.org/cvmfs/neurodesk.ardc.edu.au
==================================================================

However, if I SSH into the container and run 'sudo mount -t cvmfs neurodesk.ardc.edu.au /cvmfs/neurodesk.ardc.edu.au' (even while the original connection is hanging or if I kill the process manually and re-mount), it works instantaneously. My workaround is to disable CVMFS on startup and then manually run it the first time I log in, but I would like to get to the bottom of it!

Do you have advice on how best I would troubleshoot this issue? I can't find any CVMFS or autofs logs anywhere on the system, and dmesg doesn't say anything, but my knowledge of CVMFS is basically nil so shrug.

Here is the docker command I use:

sudo docker run \
        --shm-size=1gb -it --privileged --name neurodesktop \
        -v /data/vm/neurodesktop-storage:/neurodesktop-storage \
        -e HOST_UID="$(id -u)" -e HOST_GID="$(id -g)" \
        -e NVIDIA_VISIBLE_DEVICES=all \
        -e NVIDIA_DISABLE_REQUIRE=1 \
        -p 8080:8080 -h neurodesktop-latest \
        vnmd/neurodesktop:latest

[stebo85]

Dear Tim, great to hear you are using Neurodesk :)

Your problem is really interesting and I think we might have to look at this in an interactive session. Can you book a meeting with me here: https://calendly.com/s-bollmann/meeting

I am suspecting a firewall/deep packet inspection issue on your side and would like to try a few things to narrow down the problem.

Cheers
Steffen

[stebo85]

The problem was a bug in docker in combination with fedora 36 firewall. Solution: Reinstall operating system and docker -> worked

[timrosenow]

Sure that sounds good, thanks for looking at it! And yes Neurodesk is excellent and I am moving all of my pipelines to it. Dr Tim Rosenow NIF Facility Fellow | CMCA Platform Leader - Preclinical MRI National Imaging Facility<http://www.anif.org.au/> @ Centre for Microscopy Characterisation and Analysis<http://cmca.uwa.edu.au/> The University of Western Australia • M519 35 Stirling Hwy Perth WA 6009 Australia T +61 8 6151 1023 • M +61 423 266 880 • E ***@***.******@***.***> [cid:3b2c6b6b-44b4-4c81-b1c4-218395f27ed3] [cid:37776cb7-25f6-42df-8e72-b48ceca953cd]

…

________________________________ From: Steffen Bollmann ***@***.***> Sent: Friday, 8 July 2022 1:52 PM To: NeuroDesk/neurodesk.github.io ***@***.***> Cc: Tim Rosenow ***@***.***>; Author ***@***.***> Subject: Re: [NeuroDesk/neurodesk.github.io] Assistance troubleshooting CVMFS - long timeout on startup (Discussion #175) Dear Tim, great to hear you are using Neurodesk :) Your problem is really interesting and I think we might have to look at this in an interactive session. Can you book a meeting with me here: https://calendly.com/s-bollmann/meeting<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcalendly.com%2Fs-bollmann%2Fmeeting&data=05%7C01%7CTim.Rosenow%40uwa.edu.au%7C4af84b9a62364f64aa8208da60a610a9%7C05894af0cb2846d8871674cdb46e2226%7C0%7C0%7C637928563620117632%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pZz6KnxJKQTNLUSR0dxvVCcWLSMe3h%2BX9w9IcRNxuOU%3D&reserved=0> I am suspecting a firewall/deep packet inspection issue on your side and would like to try a few things to narrow down the problem. Cheers Steffen — Reply to this email directly, view it on GitHub<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNeuroDesk%2Fneurodesk.github.io%2Fdiscussions%2F175%23discussioncomment-3105087&data=05%7C01%7CTim.Rosenow%40uwa.edu.au%7C4af84b9a62364f64aa8208da60a610a9%7C05894af0cb2846d8871674cdb46e2226%7C0%7C0%7C637928563620117632%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7uEIHwOMwJnBCMGfRlAj5jE1qjcviNIftjvSySHof8Q%3D&reserved=0>, or unsubscribe<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALJO6WBKCAXIMY6HT5G4MALVS662JANCNFSM527T6B3Q&data=05%7C01%7CTim.Rosenow%40uwa.edu.au%7C4af84b9a62364f64aa8208da60a610a9%7C05894af0cb2846d8871674cdb46e2226%7C0%7C0%7C637928563620117632%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wYmUktcF79yFv3b2EjVcEJvGa%2FAf8X4Qt9sIJA8lLsI%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>