fix: coalesce schema issues #12308
fix: coalesce schema issues #12308
Conversation
github-actions
bot
added
logical-expr
mesejo
force-pushed
the
fix/coalesce-null
branch
7 times, most recently
from
6954677
to
01fab57
Compare
mesejo
force-pushed
the
fix/coalesce-null
branch
3 times, most recently
from
4c7989e
to
30a5c5d
Compare
mesejo
force-pushed
the
fix/coalesce-null
branch
2 times, most recently
from
5c57ac6
to
1cc4344
Compare
There was a problem hiding this comment.
Thanks @mesejo and @jayzhan211
cc @findepi who I believe is also working in this area / thinking about functions
| Self { | ||
| signature: Signature::one_of( |
There was a problem hiding this comment.
It seems to me that moving the signature from a data driven description (aka describe "what" is needed and letting some other code compute if the given arguments match that signature), this PR is moving many of the functions towards more functional (each function has to implement its own custom coercion, likely resulting in significant duplication).
What do you think (perhaps as a follow on PR) of adding DataType::Null support to the Signature calculations somehow rather than inlining / duplicating the coercion logic?
Maybe something like
Signature::allow_null(..)
that would support automatically coercing arguments from null?
Or maybe we should always support coercing Null to any type
There was a problem hiding this comment.
Alternative signature like Signature::String, similar to Signature::numeric that includes converting null to string too?
There was a problem hiding this comment.
I am not sure -- I was just reacting that this "handle null" pattern seems common and it seems like this approach will require custom coerce logic for all functions 🤔
There was a problem hiding this comment.
Null to T coercion needs to be handled elsewhere anyway (eg when computing type of a UNION, etc.).
We can free functions from having to bother about coercions at all and let the engine calculate coercions when building the logical plan.
This is actually super fundamental for DataFusion vision as a composable query engine. Coercion rules are very implementation-specific. If we had functions spiced up with coercions inside them, that would make those functions non-reusable.
There was a problem hiding this comment.
100%
It seems to me like Signature is supposed to communicate what types the function implementation has a native implementation for and the coercion of whatever the user provided doesn't match one of the supported types
There was a problem hiding this comment.
@findepi Are you suggesting something like general coercion that is non-function specific? But what if we want different coercion rule for different function, we might need to do coercion function wise
There was a problem hiding this comment.
Why would we want different coercion rules for different functions?
There was a problem hiding this comment.
My idea is that it is more flexible to the user, although, without the real use case, it might be a premature optimization 🤔.
There was a problem hiding this comment.
Sorry to chip in late; this PR addresses other issues, such as #12307. I wonder if I could split it and leave the changes regarding the coercion of functions in this one (to keep the discussion in one place) and the others in a new PR. Would that be ok?
| func.name(), | ||
| func.signature().clone(), | ||
| &arg_data_types, | ||
| let new_data_types = data_types_with_scalar_udf(&arg_data_types, func) |
There was a problem hiding this comment.
I think this is the root cause of the issue and to solve this other changes are necessary. Therefore, I think we should go with this change and maybe further optimize the coercion in another PR.
There was a problem hiding this comment.
Ok, so should I leave it as it is? Or change it back to how it was:
data_types_with_scalar_udf(&arg_data_types, func)There was a problem hiding this comment.
I defer to @jayzhan211 -- if he is good to merge this PR, let's get the conflicts resolved and merge it in.
If there is additional work we know is needed / could be cleaned up, let's try and file them as tickets
mesejo
force-pushed
the
fix/coalesce-null
branch
from
1cc4344
to
854ed60
Compare
|
Thanks @mesejo and @alamb @findepi @andygrove for the review |
Which issue does this PR close?
Closes #12307.
Are these changes tested?
Yes
Are there any user-facing changes?
No