You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are using autofaiss to build a pipeline for creating indexes of different sizes.
We have embedding datasets ranging from 1M to 1B examples. Currently all the embeddings we use are 768 dims.
We are running on a GCP dataproc cluster. What are the recommended cluster machine sizes and session configs for the master and worker nodes to best utilize autofaiss' parallelization code for all the different stages (training, adding embeddings and merging)?
The text was updated successfully, but these errors were encountered:
We are using autofaiss to build a pipeline for creating indexes of different sizes.
We have embedding datasets ranging from 1M to 1B examples. Currently all the embeddings we use are 768 dims.
We are running on a GCP dataproc cluster. What are the recommended cluster machine sizes and session configs for the master and worker nodes to best utilize autofaiss' parallelization code for all the different stages (training, adding embeddings and merging)?
The text was updated successfully, but these errors were encountered: