Pumi test fails on perlmutter

[wspear]

@cwsmith

The spack test for this variant of pumi installed on perlmutter by e4s 22.11:

-- linux-sles15-zen3 / [email protected] -------------------------------
[email protected]~fortran~int64~ipo~shared+simmodsuite_version_check~testing~zoltan build_system=cmake build_type=RelWithDebInfo simmodsuite=none

Fails with the following console output:

==> Error: TestFailure: 2 tests failed.


Command exited with status 127:
    './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
./uniform: error while loading shared libraries: libfabric.so.1: cannot open shared object file: No such file or directory



1 error found in test log:
     1    ==> Testing package pumi-2.2.7-llomhac
     2    ==> [2022-11-21-12:25:41.390040] testing pumi uniform mesh refinement
     3    ==> [2022-11-21-12:25:41.390483] './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
     4    ./uniform: error while loading shared libraries: libfabric.so.1: cannot open shared object file: No such file or directory
     5    FAILED: Command exited with status 127:
     6        './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
  >> 7    ./uniform: error while loading shared libraries: libfabric.so.1: cannot open shared object file: No such file or directory
     8    
     9      File "/global/cfs/cdirs/m3896/shared/ParaTools/E4S/22.11/PrgEnv-gnu/spack/bin/spack", line 100, in <module>



Failed to find executable '/opt/cray/pe/mpich/8.1.17/ofi/gnu/9.1/bin/mpiexec'

/global/cfs/cdirs/m3896/shared/ParaTools/E4S/22.11/PrgEnv-gnu/spack/lib/spack/spack/package_base.py:2023, in run_test:
       2020                    # We're below the package context, so get context from
       2021                    # stack instead of from traceback.
       2022                    # The traceback is truncated here, so we can't use it to
  >>   2023                    # traverse the stack.
       2024                    m = "\n".join(spack.build_environment.get_package_context(tb))
       2025
       2026                exc = e  # e is deleted after this block


/global/cfs/cdirs/m3896/shared/ParaTools/E4S/22.11/PrgEnv-gnu/spack/lib/spack/spack/build_environment.py:1086, in _setup_pkg_and_run:
       1083        tb_string = traceback.format_exc()
       1084
       1085        # build up some context from the offending package so we can
  >>   1086        # show that, too.
       1087        package_context = get_package_context(tb)
       1088
       1089        logfile = None

See test log for details:
  /pscratch/sd/w/wspear/perlmutter/spack_user_cache/test/hyjemf5zrzgrnmfmzuddmljbsxwuyat4/pumi-2.2.7-llomhac-test-out.txt

==> Error: 1 test(s) in the suite failed.

--- ==> Spack test hyjemf5zrzgrnmfmzuddmljbsxwuyat4
==> Testing package pumi-2.2.7-llomhac
======================== 1 failed, 0 passed of 1 specs ========================= ---

[cwsmith]

Hi @wspear.

Hmmm. I'll try to reproduce this.

Do the tests run on a front end node or a compute node?

[wspear]

@cwsmith I run these on the compute nodes. I'm not sure how they behave on the login.

[wspear]

@cwsmith I'm seeing similar behavior on Crusher:

==> Error: TestFailure: 2 tests failed.


Command exited with status -6:
    './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
gmi failed: could not open model file



1 error found in test log:
  >> 5    FAILED: Command exited with status -6:
     6        './uniform' '../testdata/pipe.dmg' '../testdata/pipe.smb' 'pipe_unif.smb'
     7    gmi failed: could not open model file
     8    
     9      File "/gpfs/alpine/csc439/world-shared/E4S/ParaTools/22.11/PrgEnv-gnu/spack/bin/spack", line 100, in <module>
     10       sys.exit(spack.main.main())
     11     File "/gpfs/alpine/csc439/world-shared/E4S/ParaTools/22.11/PrgEnv-gnu/spack/lib/spack/spack/main.py", line 10
          03, in main



Failed to find executable '/opt/cray/pe/mpich/8.1.17/ofi/gnu/9.1/bin/mpiexec'

/gpfs/alpine/csc439/world-shared/E4S/ParaTools/22.11/PrgEnv-gnu/spack/lib/spack/spack/package_base.py:2023, in run_test:
       2020                    # We're below the package context, so get context from
       2021                    # stack instead of from traceback.
       2022                    # The traceback is truncated here, so we can't use it to
  >>   2023                    # traverse the stack.
       2024                    m = "\n".join(spack.build_environment.get_package_context(tb))
       2025
       2026                exc = e  # e is deleted after this block


/gpfs/alpine/csc439/world-shared/E4S/ParaTools/22.11/PrgEnv-gnu/spack/lib/spack/spack/build_environment.py:1086, in _setup_pkg_and_run:
       1083        tb_string = traceback.format_exc()
       1084
       1085        # build up some context from the offending package so we can
  >>   1086        # show that, too.
       1087        package_context = get_package_context(tb)
       1088
       1089        logfile = None

See test log for details:
  /ccs/home/wspear/.spack/test/jo3wjmyxtqz4zrufu4wv2pxbbvskugeo/pumi-2.2.7-bh3g2dq-test-out.txt