Specify conda environment for test run phase?

[samsrabin]

In CTSM, we have a couple of SystemTests—FSURDATMODIFYCTSM and RXCROPMATURITY—that call non-CIME Python scripts to do things like manipulate input files. These scripts need access to certain modules that we access using our ctsm_pylib conda environment. This is currently being handled by calling them with conda run -n ctsm_pylib python3 script ... or conda activate ctsm_pylib && python3 script ... in subprocess.run(), but that doesn't work for some users. (We think it's something to do with their Cheyenne environments, but it's pretty tricky to track down exactly why.)

To fix the issue for those users and improve robustness generally, I decided to call the external scripts directly rather than using subprocess.run(). This works for FSURDATMODIFYCTSM, which calls its external script in the SHAREDLIB_BUILD phase, as long as the user does conda activate ctsm_pylib before calling create_test. However, it doesn't work for RXCROPMATURITY, which calls its external scripts in the run phase. It seems that the conda environment isn't active in the run phase.

Is there any way—without using subprocess.run()—to make the run phase take place within the same conda environment as create_test was called from?

[jedwards4b]

I have an issue with adding tests that cannot be run outside of the ncar environment. In fact we had a forum user just yesterday who was very upset that one of the tests in the ctsm provided test suite was failing for them. But after examining the test I concluded that it had not been tested outside of the ncar systems.

[ekluzek]

I'm not sure exactly what you are saying here @jedwards4b. The reason we are setting up conda environments for our tests and tools, is precisely that, with the hope that they can be run outside of the NCAR environment. If we rely on particular NCAR conda environments we know it's tied to NCAR. But, having the ability to setup a conda environment that the user can replicate we figure it should be able to work anywhere.

[samsrabin]

Well said, @ekluzek. I'll also add that our tests don't rely on anything too exotic, mostly just numpy and xarray. But somehow even those aren't present in the standard Cheyenne or Derecho environments.

[jedwards4b]

The test I was referring to above was SMS_D.f09_g16.I1850Clm50BgcSpinup.*_intel.clm-cplhist it relies on data stored outside of inputdata and so not available to users not at NCAR.

[ekluzek]

Ahh, yes, that is a test that relies on local NCAR data. We haven't been sure if we want to support a bunch of CPLHIST data. We do need at least some data for testing though. I'll make an issue about that. It does look like we have some testmods that are built around assuming a path to /glade. There's some offline tools that are only assumed to work at NCAR. But, it looks like there are paths that we should clean up to use inputdata.

[ekluzek]

I made an issue for this here: ESCOMP/CTSM#2133

[jedwards4b]

That said - I think that the issue you are seeing is because create_test is run on the login node and environment and the run phase is from a compute node. I suspect that the conda environment is not being carried forward.

[samsrabin]

Yeah, that's what I figured. So I guess there's not a way to make it carry forward? For example, to make conda activate ctsm_pylib be called at the very beginning of the run phase before Python is ever called?

I guess I don't really understand how the run phase job is submitted. Could you point me to where that happens in CIME?

[jedwards4b]

https://github.com/ESMCI/cime/blob/master/CIME/test_scheduler.py#L1089

[ekluzek]

Great question @samsrabin. I think the underlying problem is that the run phase is normally sent to the batch system in cime. So you'd probably need to change the run method so that it bypasses the batch submit part. This would be doing something similar to the --no-batch option for CIME scripts. Although I'm not sure if --no-batch doesn't still do a subprocess.run call? Maybe it could be triggered by setting the no-batch option for these particular cases. We can't run all tests with --no-batch, but these python tests can certainly be done that way. This just means the python tests will take place as part of the build time so will add a bit to the build time, since I don't think any of these are extraordinarily long that should be fine, and normally they are run under qcmd anyway.

My answer is essentially a longer version of what @jedwards4b, but he got his in first....

[jedwards4b]

Perhaps instead of trying to get this to work on cheyenne you should first move to derecho?

[samsrabin]

@jedwards4b The conda activate/conda run -n issue might be fixed on Derecho, but as illustrated by our Cheyenne experience it seems fragile. We'd really like to avoid subprocess.run() at all.

[jedwards4b]

I believe that subprocess.run is the preferred python way to run an external - I guess I don't understand what you are proposing as an alternative?

[jedwards4b]

There are also a couple of mechanisms to run external scripts available to a normal case (as opposed to a test), have you explored any of those? PRERUN_SCRIPT, POSTRUN_SCRIPT, DATA_ASSIMILATION_SCRIPT.

[samsrabin]

Instead of running it as an external at all, I import it and call it directly from Python. So for FSURDATMODIFYCTSM, for example, it goes from:

subprocess.run(f"conda run -n ctsm_pylib python3 {ctsm_root}/tools/modify_input_files/fsurdat_modifier {cfg_file_path}", ...)

to

from argparse import Namespace
from ctsm.modify_input_files.fsurdat_modifier import fsurdat_modifier
fsurdat_modifier_args = Namespace(cfg_path=self._cfg_file_path, ...)
fsurdat_modifier(fsurdat_modifier_args)

[samsrabin]

@jedwards4b I liked the PRERUN_SCRIPT/POSTRUN_SCRIPT idea. Unfortunately the script I'm calling doesn't work as PRERUN_SCRIPT. If I try the conda run -n ctsm_pylib ... method it seems to ignore the -n ctsm_pylib bit, with the script erroring as numpy isn't found. If I try the conda activate method, I run into "Your shell has not been properly configured to use 'conda activate'," which is a lie but one we've previously been unable to get around.

Annoyingly, if I call the prerun script from a login shell it works fine.