Regarding the issue of checkpoint resumption in DeepMDkit

[hellolori]

Dear developers,

I have recently been learning to use DeepMDkit (v2.2.11) to train models and have found that resuming training from a checkpoint and training directly for the same number of steps yield very different results.

I have conducted the following two training methods:

• (1)The first method involves intentionally stopping the computation after 5000 steps and then continuing the training to 10000 steps by reading the cache file with the following command.

$ dp train --restart model.ckpt input.json

Below is the corresponding lcurve.out file.

• (2)The second method is to train normally for 10000 steps. Below is the corresponding lcurve.out file.

dp train input.json

By comparing the two, it can be seen that the training data for the first 5000 steps is almost the same, but the data for the last 5000 steps differs greatly. This confuses me. Could you please explain the reason for this?

• Additionally, I have another question. I would like to further train the model obtained from the above 10,000 steps training by adding new training sets. How should I proceed?

[hellolori]

Here are the input.json.

{
    "_comment": " model parameters",
    "model": {
	"type_map":	["S", "Mo"],
	"descriptor" :{
	    "type":		"se_e2_a",
	    "sel":		"auto",
	    "rcut_smth":	0.50,
	    "rcut":		6.00,
	    "neuron":		[25, 50, 100],
	    "resnet_dt":	false,
	    "axis_neuron":	16,
	    "seed":		1,
	    "_comment":		" that's all"
	},
	"fitting_net" : {
	    "neuron":		[240, 240, 240],
	    "resnet_dt":	true,
	    "seed":		1,
	    "_comment":		" that's all"
	},
	"_comment":	" that's all"
    },

    "learning_rate" :{
	"type":		"exp",
	"decay_steps":	50,
	"start_lr":	0.001,
	"stop_lr":	3.51e-8,
	"_comment":	"that's all"
    },

    "loss" :{
	"type":		"ener",
	"start_pref_e":	0.02,
	"limit_pref_e":	1,
	"start_pref_f":	1000,
	"limit_pref_f":	0.02,
	"start_pref_v":	1,
	"limit_pref_v":	1,
	"_comment":	" that's all"
    },

    "training" : {
	"training_data": {
	    "systems":     ["../00.data/training_data"],
	    "batch_size":  "auto",
	    "_comment":	   "that's all"
	},
	"validation_data":{
	    "systems":	   ["../00.data/validation_data"],
	    "batch_size":  "auto",
	    "numb_btch":   1,
	    "_comment":	   "that's all"
	},
	"numb_steps":	10000,
	"seed":		10,
	"disp_file":	"lcurve.out",
	"disp_freq":	200,
	"save_freq":	1000,
	"_comment":	"that's all"
    },

    "_comment":		"that's all"
}

[njzjz]

By comparing the two, it can be seen that the training data for the first 5000 steps is almost the same, but the data for the last 5000 steps differs greatly. This confuses me. Could you please explain the reason for this?

The random state is reset after restarting.

I also noticed that step 5000 is printed twice with different data. I am not sure whether it is expected. cc @wanghan-iapcm

• Additionally, I have another question. I would like to further train the model obtained from the above 10,000 steps training by adding new training sets. How should I proceed?

Use --init-model instead of --restart.

[hellolori]

Thank you for your response and for clarifying my doubts. Moreover, concerning the issue you mentioned about the 5000th step being printed twice with differing data, I have carried out several tests and confirmed that this is indeed happening. Below is the output file from another training session. I am using DeePMD-kit v2.2.11.