why does GenerateEmbeddingsAsync return IList<ReadOnlyMemory<TEmbedding>> ?

[markat1]

why is GenerateEmbeddingsAsync returning IList<ReadOnlyMemory<TEmbedding>> ?

Couldn't it just return Tembedding? why the readonlymemory?

using System;using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.SemanticKernel.Services;

namespace Microsoft.SemanticKernel.Embeddings;

/// <summary>
/// Represents a generator of embeddings.
/// </summary>
/// <typeparam name="TValue">The type from which embeddings will be generated.</typeparam>
/// <typeparam name="TEmbedding">The numeric type of the embedding data.</typeparam>
[Experimental("SKEXP0001")]
public interface IEmbeddingGenerationService<TValue, TEmbedding> : IAIService
    where TEmbedding : unmanaged
{
    /// <summary>
    /// Generates an embedding from the given <paramref name="data"/>.
    /// </summary>
    /// <param name="data">List of strings to generate embeddings for</param>
    /// <param name="kernel">The <see cref="Kernel"/> containing services, plugins, and other state for use throughout the operation.</param>
    /// <param name="cancellationToken">The <see cref="CancellationToken"/> to monitor for cancellation requests. The default is <see cref="CancellationToken.None"/>.</param>
    /// <returns>List of embeddings</returns>
    Task<IList<ReadOnlyMemory<TEmbedding>>> GenerateEmbeddingsAsync(
        IList<TValue> data,
        Kernel? kernel = null,
        CancellationToken cancellationToken = default);
}

[evchaki]

@westey-m can you take a look at this?

[westey-m]

@markat1, ReadOnlyMemory<T> is used in .net to work more efficiently with large blocks of data, like in the case of vectors.

E.g. it's possible to create views over the data without needing to create a new instance / copy of the data, avoiding potentially expensive allocations. You can read more about the benefits and usage here:
https://learn.microsoft.com/en-us/dotnet/standard/memory-and-spans/

Other vector related .net APIs consume ReadOnlySpan<T> as input, which is interrelated with ReadOnlyMemory<T> and can be accessed as a property on ReadOnlyMemory<T>.
See e.g. the TensorPrimitives.CosineSimilarity method:
https://learn.microsoft.com/en-us/dotnet/api/system.numerics.tensors.tensorprimitives.cosinesimilarity

Converting to and from an array is also not difficult in cases where an array is needed, but it's important to note that ToArray does a copy which incurs an allocation cost.

ReadOnlyMemory<float> vector1 = new float[] { 30f, 31f, 32f, 33f };
float[] array1 = vector1.ToArray();

See https://learn.microsoft.com/en-us/dotnet/api/system.readonlymemory-1.toarray

[markat1]

thank you for your elaborated answer @westey-m! - one last question, if that's okay!

In the context of implementing GenerateEmbeddingsAsync - any recommended/optimization thoughts on how I should handle embeddings returned back from for example ollama? Here's an example of me implementing GenerateEmbeddingsAsync from IEmbeddingGenerationService:

1. method takes in prompt data from the user
2. then makes a http request to ollama api in order to get a embedding representation of the user data
3. then deserializing the response from ollama into json nodes
4. converting embeddings into a readonly memory of floats (the tricky part)
5. returns back the result of 4 in a list of readonlymemory.

// 1. method takes in data from the user

public async Task<IList<ReadOnlyMemory<float>>> GenerateEmbeddingsAsync(IList<string> data, Kernel? kernel = null, CancellationToken cancellationToken = default)
 {
 
     // 2. then makes a http request to  ollama api in order to get a embedding representation of the user data

     var request = new HttpRequestMessage(HttpMethod.Post, "/api/embeddings")
     {
         Content = PrepareChatRequestContent(data)
     };

     var response = await _httpClient.SendAsync(request, cancellationToken);

     var embeddings = await response.Content.ReadAsStringAsync(cancellationToken);
  
     // 3. deserializing  response into json nodes

     var content = JsonSerializer.Deserialize<JsonNode>(embeddings);
    
    
     // 4.  Here I'm trying to handle converting embeddings into a readonly memory  of floats (the tricky part)

     ReadOnlyMemory<float> embedding = new ReadOnlyMemory<float>(content!["embedding"]?.AsArray().GetValues<float>().ToArray());

     // 5. returns back the result of 4 in a list of readonlymemory.

     return new List<ReadOnlyMemory<float>>()
     {
         embedding,
     };
 }

 private JsonContent PrepareChatRequestContent(IList<string> data)
 {
     return JsonContent.Create(new
     {
         Model = "mxbai-embed-large",
         Prompt = data.FirstOrDefault(),
     }, options: _jsonSerializerOptions);
 }

[westey-m]

@markat1, the nice thing is that Sytem.Text.Json.JsonSerializer has built in support for ReadonlyMemory<> from version 8, so you can deserialize directly into that type.

We also have a backfill in SK for older versions of .net:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/InternalUtilities/src/Text/ReadOnlyMemoryConverter.cs

Note that we have just released a preview Ollama connector that includes embedding support:
https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Ollama/1.19.0-alpha#versions-body-tab

See here for a usage example:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Memory/Ollama_EmbeddingGeneration.cs

Finally, if you still need to do your own implementation for a different service, here is an example snippet that you can run locally, which shows the .net 8 ReadOnlyMemory<> built in support.

using System.Text.Json;
using System.Text.Json.Serialization;

class Program
{
    static async Task Main()
    {
        var json = """
            {
                "embeddings": [
                    [0.1, 0.2, 0.3, 0.4],
                    [0.5, 0.6, 0.7, 0.8]
                ]
            }
            """;
        
        var embeddings = JsonSerializer.Deserialize<EmbeddingResults>(json);
    }

    private class EmbeddingResults
    {
        [JsonPropertyName("embeddings")]
        public ReadOnlyMemory<float>[] Embeddings { get; set; }
    }
}