Can autofaiss take spark dataframe as input?

[vikasd22]

Hi,
I am trying to parse a spark dataframe as input for the distributed case as follows:

index, index_infos = build_index(
    embeddings=sdf,
    embedding_column= "embeddings",
    distributed="pyspark",
    temporary_indices_folder="/tmp/faiss",
)

Is it possible?

[hitchhicker]

Hey!

Good question, It is not supported unfortunately.

[vikasd22]

@hitchhicker Okay. Do you have recommendations for dealing more than memory embeddings. I have a spark dataframe and I could do sdf.toPandas().to_numpy() but it is probably gonna get killed if big data situations.

[rom1504]

Save it to disk/hdfs as parquet and use the distributed implementation of autofaiss, see readme for details of usage

…

On Thu, May 25, 2023, 12:48 Vikas Dubey ***@***.***> wrote: @hitchhicker <https://github.com/hitchhicker> Okay. Do you have recommendations for dealing more than memory embeddings. I have a spark dataframe and I could do sdf.to_pandas().to_numpy() but it is probably gonna get killed if big data situations. — Reply to this email directly, view it on GitHub <#163 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437VIO5HVHDRVHHJ44NDXH42GVANCNFSM6AAAAAAYM4F4O4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>