Does AddBertOnnxTextEmbeddingGeneration support multi-lingual models like intfloat / multilingual-e5-small

[gonsss]

Hi,
Question : Should the model multilingual-e5-small work with the AddBertOnnxTextEmbeddingGeneration ?
Context :
I have been evaluating the new support for ONNX based embeddings model.
My goal is to self-host a small(ish) a multi-lingual model like intfloat
/
multilingual-e5-small.

While I dont get an error, I get unexpected results : the most similar results are actually not similar.

Should this model work with SK ?

I know SK only supports BERT based models, it is my understanding e5-small is based on XML-r which it is based on BERT.

LEt me know if I misunderstood or if the model should actually work, it would be great if we could have an example.

[RogerBarreto]

Hi @gonsss, thanks for the comment, I couldn't test this myself as the multilingual-e5-small repo does not have a vocab.txt file which is required.

If you have a vocab.txt file and can provide or update the repo with it that would be helpful to follow up.

For example this one has and works fine TaylorAI-bge-micro-v2

[gonsss]

Thanks @RogerBarreto ,
Yeah,I did more digging and found the multilingual-e5-small model uses a different tokenizer (SentencePiece BPE) which doesnt use vocab.txt files.
So adding support for these models would entail to add support for different tokenizers and models that use different tokenizers.
Regarding how to make that working , I did a little bit of research and I see two paths (untested both) :

1. How to get the sentencepiece vocabulary from .model file and store it to .vocab? google/sentencepiece#668
2. Would any of the tokenizers offered by https://learn.microsoft.com/en-us/dotnet/api/microsoft.ml.tokenizers?view=ml-dotnet-preview work? -> I think so as I didnt find Sentencepiece
3. Use this tokenizers natively in c# https://www.nuget.org/packages/BlingFireNuget/

I will carry on digging, but I gather that unless 1) works , SK will have to expand support for additional tokenizers.