Optim deformable detr

[yonigozlan]

What does this PR do?

This PR is part of an ongoing effort to optimize the inference speed of Transformers' vision models. (along with #33412 for now).
For more info on the specific issues these optimizations target and how they help improve the inference speed of compiled models, you can check out this Notion page.

The following metrics are for model inference only! Pre/post-processing time are not measured here. Currently, Transformers image processors seem to be a big bottleneck for inference speed, so end-to-end inference will sadly be much slower than this.

A few things to note:

• Other models which use deformable attention were modified due to "copied from", but the modifications should be non breaking. I haven't measured it but those models could also be slightly faster due to it :).
• Grounding Dino in particular could really benefit from more changes than those that were simply copied, but this could be part of another PR.
• Contrary to the optimization done on RT-DETR, here both eager and compiled models are improved :).
• I have no idea what is going on with the compiled fp16 model without custom kernel. The trace shows full use of the GPU, and similar changes were made on RT-DETR with considerable gain in the same settings. If anyone has a theory I would be glad to hear it :).
• Deformable DETR (unlike RT-DETR) uses the custom kernel for deformable attention by default. But the user needs to have ninja-build installed for it to work.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@qubvel @molbap @amyeroberts @NielsRogge

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Looks great!

Just two things before being merge ready:

• We'll need to run the slow tests for these models
• Removing the baseline file

[amyeroberts]

To delete?

[qubvel]

Looks good to me! Just some nits and questions :)

Can you please provide a bit more details on what was changed and why that leads to a better performance? As far as I see the general idea of this PR is to avoid .item() or am I missing anything else?

[qubvel]

Just wondering, why should we use float16 instead of float32? shouldn't it be pixel_values.dtype?

[qubvel]

in case it holds here :)

Suggested change

	total_elements = sum(shape[0] * shape[1] for shape in spatial_shapes_list)
	total_elements = sum(height * width for height, width in spatial_shapes_list)