Installing DeePMD-kit on cluster

[jamesh1234567]

Hi folks,

I’m banging my head against a wall here, just started using depemd kit and I can condo install it no problem on my personal device but I keep getting dependency issues when installing on cluster.

I’m using all the same versions I have on my laptop but I keep hitting dependency issues, it’s like whack-a-mole, I resolve one problem and that creates 3 more.

Is there a better way to do this or any workflow I could follow? Any advice or help would be appreciated.

[njzjz]

We always suggest installing in a new, clean environment, so there is no conflict with the existing packages.

[jamesh1234567]

Thank you for your response.

I did create a new conda environment for installation, module loaded python-3.11 from the cluster and conda installed it as I did on my personal device. I have deepmd-kit 2.2.10 and tensorflow 2.16.1 which is the exact combination that works on my personal device yet when i try to run a command on the cluster like dp train it tells me my deepmd kit and tensorflow are incompatible. Do you have any advice?

Here is the error I get:
2024-07-03 09:27:51.907169: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used. 2024-07-03 09:27:51.912530: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used. 2024-07-03 09:27:51.981899: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-07-03 09:27:53.793894: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT WARNING:tensorflow:From /sfihome/james.hennessy/.local/lib/python3.11/site-packages/tensorflow/python/compat/v2_compat.py:98: disable_resource_variables (from tensorflow.python.ops.resource_variables_toggle) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term WARNING:root:To get the best performance, it is recommended to adjust the number of threads by setting the environment variables OMP_NUM_THREADS, TF_INTRA_OP_PARALLELISM_THREADS, and TF_INTER_OP_PARALLELISM_THREADS. See https://deepmd.rtfd.io/parallelism/ for more information. Traceback (most recent call last): File "/sfihome/james.hennessy/miniconda3/envs/deepmd/lib/python3.11/site-packages/deepmd/env.py", line 396, in get_module module = tf.load_op_library(str(module_file)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/sfihome/james.hennessy/.local/lib/python3.11/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library lib_handle = py_tf.TF_LoadLibrary(library_filename) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tensorflow.python.framework.errors_impl.NotFoundError: /lib64/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /sfihome/james.hennessy/miniconda3/envs/deepmd/lib/python3.11/site-packages/deepmd/lib/libdeepmd_op.so)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/sfihome/james.hennessy/miniconda3/envs/deepmd/bin/dp", line 10, in
sys.exit(main())
^^^^^^
File "/sfihome/james.hennessy/miniconda3/envs/deepmd/lib/python3.11/site-packages/deepmd_utils/main.py", line 655, in main
from deepmd.entrypoints.main import main as deepmd_main
File "/sfihome/james.hennessy/miniconda3/envs/deepmd/lib/python3.11/site-packages/deepmd/init.py", line 11, in
import deepmd.utils.network as network
File "/sfihome/james.hennessy/miniconda3/envs/deepmd/lib/python3.11/site-packages/deepmd/utils/init.py", line 9, in
from .learning_rate import (
File "/sfihome/james.hennessy/miniconda3/envs/deepmd/lib/python3.11/site-packages/deepmd/utils/learning_rate.py", line 8, in
from deepmd.env import (
File "/sfihome/james.hennessy/miniconda3/envs/deepmd/lib/python3.11/site-packages/deepmd/env.py", line 476, in
op_module = get_module("deepmd_op")
^^^^^^^^^^^^^^^^^^^^^^^
File "/sfihome/james.hennessy/miniconda3/envs/deepmd/lib/python3.11/site-packages/deepmd/env.py", line 447, in get_module
raise RuntimeError(error_message) from e
RuntimeError: This deepmd-kit package is inconsitent with TensorFlow Runtime, thus an error is raised when loading deepmd_op. You need to rebuild deepmd-kit against this TensorFlow runtime.
WARNING: devtoolset on RHEL6 and RHEL7 does not support _GLIBCXX_USE_CXX11_ABI=1. See https://bugzilla.redhat.com/show_bug.cgi?id=1546704
`

[njzjz]

module loaded python-3.11 from the cluster

This might be the reason. Conda ships its own libstdc++.so.6 to meet the requirements and libstdc++ is not forward-compatible.