feature(whl): add AWR algorithm. #828
Conversation
PaParaZz1
changed the title
|
add the AWR algorithm into the table of README |
| @@ -18,13 +18,16 @@ class LanguageTransformer(nn.Module): | |||
| Interfaces: | |||
| ``__init__``, ``forward`` | |||
| """ | |||
| mode = ['compute_actor', 'compute_critic', 'compute_actor_critic'] | |||
|
|
|||
| def __init__( | |||
| self, | |||
| model_name: str = "bert-base-uncased", | |||
There was a problem hiding this comment.
add some comments about why we use "bert-base-uncased" as default?
|
|
||
| # Prepare train_sample (the question to be answered) and the candidate_samples (the prompts to be selected) | ||
| train_samples, cand_samples = batch["obs"]["train_sample"], batch["obs"]["candidate_samples"] | ||
| for ii in range(len(cand_samples)): |
There was a problem hiding this comment.
change 'ii' into more explainable name?
| adv = torch.clamp( | ||
| return_ - batch['value'], min=self._cfg.learn.norm_range[0], max=self._cfg.learn.norm_range[1] | ||
| ) | ||
| policy_loss = -(log_prob * torch.exp(adv / self._cfg.learn.beta)).mean() |
There was a problem hiding this comment.
add comments about the key part of advantage weighted regression
There was a problem hiding this comment.
Should the operation torch.exp(adv / self._cfg.learn.beta) stop the gradient flow?
There was a problem hiding this comment.
this operation is computed on the batch data rather than the output data, thus it is no need to stop the gradient
| if len(real_act.shape) == 1: | ||
| real_act = real_act.unsqueeze(-1) | ||
| # Calculate loss. | ||
| total_policy_loss, total_entropy_loss, total_value_loss = 0, 0, 0 |
| # (float) Coefficient that controls the exp scale in awr algorithm. | ||
| beta=1.0, | ||
| # (float) Weight of entropy regularization in the loss function. | ||
| entropy_weight=0.01, |
There was a problem hiding this comment.
Should we change it to a more generally applicable constant, such as 0.001?
| adv = torch.clamp( | ||
| return_ - batch['value'], min=self._cfg.learn.norm_range[0], max=self._cfg.learn.norm_range[1] | ||
| ) | ||
| policy_loss = -(log_prob * torch.exp(adv / self._cfg.learn.beta)).mean() |
There was a problem hiding this comment.
this operation is computed on the batch data rather than the output data, thus it is no need to stop the gradient
| from easydict import EasyDict | ||
|
|
||
| tabmwp_prompt_pg_config = dict( | ||
| exp_name='tabmwp_prompt_pg_seed0', |
There was a problem hiding this comment.
polish name, not pg in this file
| @@ -36,10 +39,16 @@ def __init__( | |||
| - embedding_size (:obj:`int`): The embedding size of the added linear layer, such as 128. | |||
| - freeze_encoder (:obj:`bool`): Whether to freeze the encoder language model while training, \ | |||
| defaults to be ``True``. | |||
| - hidden_dim (:obj:`int`): The embedding dimension of the encoding model (e.g. BERT). This value should \ | |||
| correspond to the model you use. For bert-base-uncased, this value is 768. | |||
Description
Implement the algorithm of AWR (languange model as the policy)
Related Issue
TODO
Check List