Can we support for dynamic struct

[xxhZs]

Now, We need to define a structure If we want to use Insert. But I want to connect clickhouse to other databases, which means that the definition of the structure cannot be clearly defined at compile time

[ta3pks]

in order to be able to use data dont you anyway need to know the shape of it? This seems like an x y problem. If you perhaps share your real use case and why you think you need dynamic shapes we could be of better help

[fracek]

My use case is a data pipeline that imports data from json files to clickhouse. Data could have any shape and it's known only by the user. Looks like the main issue is with Row::COLUMN_NAMES being static.

[GeorgeLeePatterson]

BUMP This is an extremely useful problem to solve. Even just finishing out the ability to use new types would probably solve this (allowing the new types to wrap a vec for instance). Clickhouse allows inserts using "", so at LEAST for impl Row for Vec you could provide a default const of "". It's forcing me to look at other implementations.

[ta3pks]

this seems Really easy to fix actually row trait just contains a property of column names making the type impl AsRef<str> would just solve everything without breaking the current implementations as well let me open a pr for it and see how it goes

[ta3pks]

as I was implementing I again did not understand the purpose of this. you told about dynamic columns but again i dont understand how are you going to insert something that you dont know the type of your table must have a type after all ?

[fracek]

I don't know the names/types at compile time. In my case, I know the name of the columns (but not the type, tho I can work around that) at runtime.

For example, imagine a CLI tool to import arbitrary csv files into clickhouse. When I implement that tool, I don't know on which files my users will run it on so I need to work with dynamic columns.

[jjtt]

That tool exists. It's called clickhouse-client

[fracek]

Obviously I'm not implementing that, it's just a self-contained example.

[ta3pks]

I see now @fracek it seems join_column_names function is doing type to column conversion and theoretically one could implement it for something like HashMap<String,T> but that would break some guarantees i guess since on every iteration you could pass some different shaped map in

[fracek]

In my case even a simple DynamicRow type where the column names are passed in new would work. I tried to implement it but it requires changing more code than I initially thought.

[GeorgeLeePatterson]

I think even updating the current serde serializer to take advantage of some of serde’s features could help. Serde does allow serializing a HashMap for instance. And the current serializer just doesn’t have implementations for NewType structs among other things. If I get some time I can try and see what I can add in this regard. If anyone has any ideas on how serde’s built in attribute tags might help then maybe even just updating the library’s macro would get it there. At the end of the day I think something that iterates over key/value tuples would at least allow constructing batch inserts. This is essentially what I’m doing, while using the execute method, but I imagine I’m losing some control along the way. Maybe not. If anyone’s interested I can share how I’m doing it currently.

[ta3pks]

I see now @fracek it seems join_column_names function is doing type to column conversion and theoretically one could implement it for something like HashMap<String,T> but that would break some guarantees i guess since on every iteration you could pass some different shaped map in

@GeorgeLeePatterson what do you think about this idea ?