Support taking unpacked install from CVMFS

[chrisburr]

It would be nice for the Pilot to take an existing DIRACOS installation from CVMFS to reduce the load on the worker node's filesystem.

I've put an example on /cvmfs/lhcbdev.cern.ch which would probably be hosted on /cvmfs/dirac.egi.eu instead (Don't use it, I may delete it at any time!). The idea would then be to make a Python venv on top of it like so:

$ source /cvmfs/lhcbdev.cern.ch/experimental/dirac-pilot-ideas/v1/DIRACOS/v2.29/Linux-x86_64/diracosrc
$ python -m venv --system-site-packages /tmp/diracos-venv
$ cp $DIRACOS/diracosrc /tmp/diracos-venv/diracosrc
$ echo "source /tmp/diracos-venv/bin/activate" >> /tmp/diracos-venv/diracosrc
# Some other modifications should probably also be done to the diracosrc
# From now onwards "source /tmp/diracos-venv/diracosrc" can be used to activate the CVMFS based DIRACOS
$ source /tmp/diracos-venv/diracosrc
$ pip install 'DIRAC==7.3.32'

This avoids creating 62108 files/links/directories (1.6GB of data) and would make pilots much faster while still letting people control the DIRAC version and install extensions if desired.

My suggestion to implement this would be:

• Add a script to DIRACOS which creates this virtual environment
• Add support to the pilot to do this instead of installing DIRACOS2 (might also need a little SiteDirector work)
• Setup automatic installation of versions DIRACOS environments on /cvmfs/dirac.egi.eu

cc @IgorPelevanyuk who expressed interest in this before the BiLD meeting

[IgorPelevanyuk]

Great! Thank you!
I will try to look into this.

[fstagni]

PR #205 is for this. There are anyway few points:

• If we would say that the policy is to have all releases on CVMFS, then, by default, we would not need to set any of the --preinstalledEnv flags
• the mechanism could be valid also for extensions, if such would have the possibility to deploy on cvmfs.dirac.egi.eu
• In general, I think that releases should be deployed as soon as they are done (we had something related to this in Always upload releases to CVMFS management#11 already few years ago)

[fstagni]

Achieved. For the moment in devel branch.