Dipole Model Not Training on the Subset of Atoms

[hanao2]

Hello,

I am trying to train a dipole model on the Wannier centroids for a subset of the atoms in my system (excluding H atoms). Following the example in the 'examples/water/dplr/train' directory and using the "dipole_type" key for excluding unwanted atom types, I have been able to train a model on v2.2.2. However, using the same setup (using either "dipole_type" or "sel_type" for atom type exclusion), I am unable to get the model to run on v3.0.0a0. From the error message, it seems like a 'atomic_dipole.npy' input of size N_frames*(N_all_atoms*3) is required. How should I solve this? Thank you!

[njzjz]

It might be fixed by #3581. I am not sure, but v3.0.0a0 contains lots of bugs. The recent versions are much more stable.

[hanao2]

Thanks for your help. I installed the most recent version and the package is running. I had to include zeros for excluded atom types in the atomic dipole file. However, there is an issue; the training is being extremely slow. Previously, I trained the same model with v2.2.2 which was much faster (both cases on CPU with the same number of cores). I am wondering what the issue is and how I can improve the training speed on this new version? For both versions, I run deepmd as follows: (for v3.0.0b3 I add --pt)
export OMP_NUM_THREADS=16
export TF_INTRA_OP_PARALLELISM_THREADS=16
export TF_INTER_OP_PARALLELISM_THREADS=8
dp train dw.json

details of the new version:
source: v3.0.0b3-24-g5111e9bd
source brach: devel
source commit: 5111e9b
source commit at: 2024-08-28 06:35:18 +0000
use float prec: double
build variant: cpu
Backend: PyTorch
PT ver: v2.4.0+cu121-ge4ee3be4063
Enable custom OP: False
computing device: cpu
CUDA_VISIBLE_DEVICES: unset
Count of visible GPUs: 0
num_intra_threads: 16
num_inter_threads: 16

[njzjz]

It's known to us that PyTorch implementation is slower than TensorFlow implementation in all models.

[hanao2]

Thanks for your quick response! I managed to train a dipole model using the pytorch backend. I am now looking into the outputs of the model through the infer/deep_tensor.py file (the 'results' variable). I see that this dictionary includes the following keys: 'dipole_redu', 'dipole_derv_c', 'dipole_derv_r', 'dipole_derv_c_redu'. Could you please explain what each of them indicates? Thanks!