Addition of an Optional Configuration for Field Selection in Document Store Queries

[luigirende]

Feature Description

I propose introducing a new optional configuration for the Query functionality in document-type stores. This configuration would allow specifying a limited number of fields to be extracted directly from the store, reducing the amount of data returned in the response.

Motivation

The current implementation of queries for document-type stores always returns all the fields of the documents in the data store, regardless of the document size or the number of fields actually needed by the application. This approach has several significant inefficiencies:

• Excessive Network Traffic: When entire documents are returned, the volume of data transferred from the data store to the client can be very high, especially when documents contain many fields or large amounts of data. This results in inefficient bandwidth usage and can slow down overall system performance.
• Computational Overhead: Sending complete documents means that the client must process a larger volume of data, even when only a few fields are needed. This overhead can increase the time and computational resources required for unmarshalling and processing the response items at the application level.
• Poor Resource Optimization: Always returning all fields of a document does not allow for optimal resource use, especially in scenarios where large documents are managed, and queries are performed frequently. This approach can reduce overall system efficiency, particularly in high-demand environments or with resource constraints.

These inefficiencies highlight the need for a configuration that allows selecting only the necessary fields for query operations, optimizing both network usage and computational resources.

Implementation Details

In the metadata configuration, the user will be able to explicitly define the desired response structure by specifying the field names and the corresponding JSON paths to retrieve the values from the store.

The proposal has already been partially implemented and covers the states with query functionality enabled, including Redis, MongoDB, CosmosDB, PostgreSQL, and CockroachDB.

Benefits

• Efficiency: Significant reduction in data traffic between the data store and the client, optimizing query operations.
• Flexibility: Developers can more precisely configure the data they want to retrieve, avoiding unnecessary overhead.
• Scalability: This approach is particularly useful in big data scenarios or where large documents are handled.

Conclusion

This feature would improve the efficiency and flexibility of queries in document stores, reducing both network usage and computational overhead at the application level.