Regarding your proposed evaluation indicators

[SuperHHzy]

Q1:I would like to ask you if your first indicator BLEU score uses BLEU-4?
Q2:What does your third metric, token accuracy, mean?

[lukas-blecher]

1. Yes
2. It's a measure for how many correct tokens were generated at the correct position in the string. It doesn't mean too much in this context. I added it on request in Evaluation #137

Hope this answers your questions.

[lukas-blecher]

When you predict the output for a given image it results in a list of tokens.
This is compared with the tokens of the ground truth. The token accuracy gives the fraction of correct tokens at the correct index.

[SuperHHzy]

What you mean is that for each mathematical expression entered, a list of tokens is generated, which contains predictions for each character of the mathematical expression. Then this list of tokens is compared with the real value, and the percentage of characters identified correctly in the list of tokens is compared with the total number of characters. Can this parameter be interpreted as the recognition accuracy rate? So what is this 0.6 level? Is it high or low?

Looking forward to your reply

[lukas-blecher]

Yes, the intuition is correct.
But recognition accuracy is too strongly worded. Every equation can be written with many different combinations of tokens which all, when rendered, will result in the same image.
So you could say it is related to the percentage of perfect matches (upper bound). I don't like the metric myself very much but it does quantify one aspect of the models performance.
Better would be a metric that looks at both rendered images, but I didn't get around to do that yet.

[SuperHHzy]

I see what you mean. Is there a reference paper for the model and framework used in this project? I would like to have a deeper understanding of this project, thanks!

[lukas-blecher]

I did not (yet) write a paper for this repo. There are a few component that I haven't seen anywhere else but the main components are the ViT and regular transformer decoder (linked in the readme).
If you have any more specific questions, I'm happy to help.