First pass at batched versions of insert / upsert

[erichare]

No description provided.

[cbornet]

If some futures error, it would be nice to return the result of successful futures and the exceptions of failed ones.
Especially if the flag partial_failures_allowed is true.
The returned value could be List[Union[API_RESPONSE, Exception]] and we know that an Exception corresponds to a batch size.

[hemidactylus]

The Cassandra Python driver enriches that with a boolean to easily tell them apart, so the return type is List[Tuple[bool, Union[API_RESPONSE,Exception]]] (see here). I wonder if that would be a good model to adopt here.

[cbornet]

Why not. I find that it's not too hard to test if the result is an instance of Exception. But I don't have a strong opinion on this.

[hemidactylus]

Same here - it can be left as simple as possible, no boolean is all good for me.

[cbornet]

Same here, the returned value could be List[Union[str, Exception]] with an optional partial_failures_allowed flag

[cbornet]

This long list could be replaced by a repeated element.
Something like

[{"_id": str(uuid.uuid4()), "name": "Abba"} for _ in range(30)]

or even

[{"name": "Abba"}] * 30

as we can let cassandra set the id.

[hemidactylus]

I have rewritten the whole test code.

[cbornet]

Set concurrency=2 ?

[cbornet]

If concurrency > 1, we could maybe set options["ordered"] = False as the insertions will be unordered anyway.

[hemidactylus]

I'd like to raise a point which goes back to the suggestion to collate the response from the various API calls. I am not sure what the best design would be.

1. The API specs make it clear that the only fields are status.insertedIds (a list) and errors (a list), so collating seems to be as futureproof as it gets
2. In support of collating, the caller hardly needs to know the details of the splitting in chunks. They pass a list of 45 documents and would be surprised to get a list of 3 response JSON from the API, each about 20, 20, 5 insertions.
3. But then what about a single call failing with an exception? (not speaking of the case of "partial_failures_allowed", where the caller would not get an exception and know how to deal with the non-inserted entries).

To elaborate more on the last point: the caller sends 45 documents and the second API call raises an exception. What is more useful to the caller:

• getting back [{...}, Exception, {...}], where they have to collate the insertedIds partial results and find out what went through (at the cost of being exposed to chunking, originally not a planned breach of abstraction)
• getting back a merged response (but then how would the Exceptions be passed back ?)
• seeing an exception being raised (but then how to find out which id got inserted, a very useful info) ?

Probably any choice other than keeping the response/exception separate by chunk is sacrificing some info, and I would avoid that - at the cost of exposing the "surprising" chunked structure in the response item.