Running r21/main xAODAnaHelpers on el9 machines

[jmsardain]

Hello,

I am finishing an analysis in r21, I am using the r21/main branch of xAODAnaHelpers and AnalysisBase,21.3.234
My code is running normally locally, the issue comes when I use condor.
Here’s what I do:

cd src/
asetup --restore
cd ../build 
source x86_64-centos7-gcc8-opt/setup.sh
cd ../run/ 
xAH_run.py --config ….. --inputList --files … -f --treeName .. condor

However, it crashes as there’s a default ATLAS setup that is done inside the run executable that is created.
You can see an example in the attached file, and here’s a snippet:

export AtlasSetupSite=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/AtlasSetup/.config/.asetup.site
export AtlasSetup=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/AtlasSetup/V03-00-06/AtlasSetup
export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
source $ATLAS_LOCAL_ROOT_BASE/user/atlasLocalSetup.sh --quiet
export AtlasProject=AnalysisBase
export AtlasVersion=21.2.234
export AtlasBuildStamp=2022-10-29T0401
export AtlasBuildBranch=21.2
export AtlasReleaseType=stable
if [ "${AtlasReleaseType}" == "stable" ]; then
     source ${AtlasSetup}/scripts/asetup.sh ${AtlasProject},${AtlasVersion} || abortJob
else
     source ${AtlasSetup}/scripts/asetup.sh ${AtlasProject},${AtlasBuildBranch},${AtlasBuildStamp} || abortJob
fi
echo "Using default setup command" || abortJob
source /data/jmsardain/charge/build/x86_64-centos7-gcc8-opt/setup.sh || abortJob

I tried adding a setupATLAS -c centos7 before this initialization happens using --optBatchShellInit

But this comes before the default initialization that comes with the driver and there’s no way to deactivate it:
https://gitlab.cern.ch/atlas/athena/-/blob/main/PhysicsAnalysis/D3PDTools/EventLoop/Root/BatchDriver.cxx?ref_type=heads#L688
Even if I do driver.batchSkipReleaseSetup = True in the xAH_run.py script in xAODAnaHelpers..

Any help would be appreciated!
Jad
run.txt

[mdhank]

Hi @jmsardain ,

Could you provide the log showing how it crashes?

As another option, would it work to run via the grid? Switching to this may be quicker than debugging, if this is time-sensitive.

Best,
Michael

[jmsardain]

Hello @mdhank

i am attaching the run files with and without the addition of
--optBatchShellInit=\"export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase\nexport ALRB_localConfigDir=$HOME/localConfig\nsource /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/user/atlasLocalSetup.sh -c centos7 -m /data -m /home \" "

log-0_withoutShellInit.txt
run_without_optShellInit.txt
log-0_withShellInit.txt
run_with_optShellInit.txt

I would like to avoid running on the grid if i can find a temporary solution that i can implement on my end that would be great, this is why i was trying to add driver.batchSkipReleaseSetup = True but it's just not able to put this flag to false:
https://gitlab.cern.ch/atlas/athena/-/blob/main/PhysicsAnalysis/D3PDTools/EventLoop/Root/BatchDriver.cxx?ref_type=heads#L688

I also tried to just remove by hand the default setup and just do my own setupATLAS and asetup .. but asetup and eventloop_batch_worker aren't recognized anymore..
run_removeByHand.txt
log-0.txt

[mdhank]

Hi @jmsardain ,

I noticed that in run_with_optShellInit.txt you have setupATLAS twice- once with -c centos7 and once without, while in run_without_optShellInit.txt it only occurs once without. (source /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/user/atlasLocalSetup.sh -c centos7 -m /data -m /home and
source $ATLAS_LOCAL_ROOT_BASE/user/atlasLocalSetup.sh --quiet). Is there a reason for doing this twice? I'm not sure how it interacts when you run both the centos7 version and the default version.

In particular, what happens if you use run_without_optShellInit.txt but just add -c centos7 -m /data -m /home to the atlasLocalSetup line, without adding any new lines?