VisualBertOnnx

[mszsorondo]

What does this PR do?

Start implementing VisualBERT for Onnx exporting

Pending:

Implement outputs
Executing one export for each VisualBERT implementation
VisualBertDummyInputGenerator

Questions

@fxmarty @younesbelkada I would appreciate any help with this

1. Is inputs() is correctly implemented?

>>> base_model = AutoModel.from_pretrained("uclanlp/visualbert-vqa")

raises the following warning:

Some weights of the model checkpoint at uclanlp/visualbert-vqa were not used when initializing VisualBertModel: ['cls.weight', 'cls.bias']
- This IS expected if you are initializing VisualBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing VisualBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

So this is not expected, any hint on how to solve this?

Fixes #555 (issue)

[fxmarty]

Looks good to me so far! Maybe you'll just run into the issue that DummyTextInputGenerator and DummyVisionInputGenerator will probably not accept the inputs (in SUPPORTED_INPUT_NAMES), as we discussed in the issue.

For 2., I guess it's because of a head in VisualBertForQuestionAnswering that is not used in the base model: https://github.com/huggingface/transformers/blob/588faad1062198e45cf3aebed21dc1fc1e1ed0d7/src/transformers/models/visual_bert/modeling_visual_bert.py#L1184 . If it is just for testing purposes, the warnings are expected. Otherwise you can try base_model = AutoModelForQuestionAnswering.from_pretrained("uclanlp/visualbert-vqa"), or try this dummy model for testing: https://huggingface.co/hf-internal-testing/tiny-random-VisualBertModel

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[fxmarty]

Hi @mszsorondo , not sure why these two show in the commit history but it should probably not:

Github commit history / diff is sometimes bugged on PRs, but maybe something is messed up?

[mszsorondo]

@fxmarty
This is indeed strange, those two commits were from a different PR, and there are some commits that are repeated and I don't remember pushing them again. Maybe I accidentally touched some command or it is a Github bug. I also see some of the tests failed.
I'll open a new draft PR from a new branch and commit the changes more separately so that the history is clear.

[fxmarty]

@mszsorondo All good! for the check_code_quality test, you will just need to run make style. For the rest you would have to check in "details" what's wrong. It appears it is that it's missing args in the CLI for the shapes.

[mszsorondo]

@fxmarty moved to draft PR #672