Fix Cuda out of memory issue in model.encode by allowing user to transfer to cpu

[Quetzalcohuatl]

In issue #487 and issue #522, users were running into OOM issues when batch size is large, because the embeddings aren't offloaded onto cpu.

The PR that fixed this only fixes it if convert_to_numpy. That means if you have convert_to_numpy=False, then your problem still exists.

In this PR, I just added an extra flag that allows the embeddings to be offloaded to cpu. This gives the user the flexibility to save the embeddings (for example if they are saving the SentenceTransformer embeddings to disk or keeping them in RAM for knn, which is often the case) instead of keeping all the embeddings on the gpu.

[paolorechia]

Hey, thanks for opening this, I'm facing this OOM issue. I'll checkout this PR and give it a try.

[paolorechia]

Hey, I tried using this PR, however, I ran into the following error:

    con.model.fit(
  File "/home/paolo/dev/openimagegenius/gpu-code/gpu-node/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 749, in fit
    self._eval_during_training(evaluator, output_path, save_best_model, epoch, -1, callback)
  File "/home/paolo/dev/openimagegenius/gpu-code/gpu-node/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 781, in _eval_during_training
    score = evaluator(self, output_path=eval_path, epoch=epoch, steps=steps)
  File "/home/paolo/dev/openimagegenius/gpu-code/gpu-node/lib/python3.10/site-packages/sentence_transformers/evaluation/EmbeddingSimilarityEvaluator.py", line 77, in __call__
    embeddings1 = model.encode(self.sentences1, batch_size=self.batch_size, show_progress_bar=self.show_progress_bar, convert_to_numpy=True)
  File "/home/paolo/dev/openimagegenius/gpu-code/gpu-node/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 201, in encode
    all_embeddings = np.asarray([emb.numpy() for emb in all_embeddings])
  File "/home/paolo/dev/openimagegenius/gpu-code/gpu-node/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 201, in <listcomp>
    all_embeddings = np.asarray([emb.numpy() for emb in all_embeddings])
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Here's how I'm calling the encode function

    def encode(self, X):
        return self.model.encode(X, convert_to_numpy=True, transfer_to_cpu=True)

[Quetzalcohuatl]

@paolorechia It looks like the problem is in the file /home/paolo/dev/openimagegenius/gpu-code/gpu-node/lib/python3.10/site-packages/sentence_transformers/evaluation/EmbeddingSimilarityEvaluator.py specifically on Line 77. You need to edit it to have transfer_to_cpu=True, because by default it's False.

Here is an example of it working correctly.

[paolorechia]

Hey, @Quetzalcohuatl thanks for the advice. Fortunately, I managed to solve my problem in another way.