Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: array-empty #7313

Merged
merged 14 commits into from
Aug 22, 2023
Merged

feat: array-empty #7313

merged 14 commits into from
Aug 22, 2023

Conversation

Weijun-H
Copy link
Member

Which issue does this PR close?

Closes #7290

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added logical-expr Logical plan and expressions physical-expr Physical Expressions sqllogictest SQL Logic Tests (.slt) labels Aug 17, 2023
@@ -2363,6 +2363,16 @@ from flatten_table;
[1, 2, 3] [1, 2, 3, 4, 5, 6] [1, 2, 3] [1.0, 2.1, 2.2, 3.2, 3.3, 3.4]
[1, 2, 3, 4, 5, 6] [8] [1, 2, 3] [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]

query I
Copy link
Contributor

@izveigor izveigor Aug 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should add some comments (like # empty scalar function #1).

@@ -548,6 +551,7 @@ impl BuiltinScalarFunction {
"The {self} function can only accept list as the first argument"
),
},
BuiltinScalarFunction::ArrayEmpty => Ok(UInt8),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function can return only 2 values (0 and 1), so it is reasonable to use Boolean instead of UInt8

let array = as_list_array(&args[0])?;
let mut builder = UInt8Array::builder(1);

for arr in array.iter() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like using builder method for array functions. I think you can use map method and collect the results as BooleanArray:

It seems to me like:

Suggested change
for arr in array.iter() {
array.iter().map(|arr| arr.is_empty()).collect::<BooleanArray>()

P.S. arr is Option, so you should consider None case too.

@izveigor
Copy link
Contributor

Thank you, @Weijun-H!
I left some comments about your feature.
cc @jayzhan211 @alamb

Copy link
Contributor

@jayzhan211 jayzhan211 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

}
})
.collect::<BooleanArray>();
// let mut builder = UInt8Array::builder(1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remember to cleanup this

@Weijun-H Weijun-H requested a review from izveigor August 18, 2023 10:56
Copy link
Contributor

@izveigor izveigor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you, @Weijun-H!

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me other than one stray println!

Thank you @Weijun-H @izveigor @jayzhan211 and @crepererum . This is a great team effort

}
})
.collect::<BooleanArray>();
println!("\n");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This println! seems like it is an oversight.

Suggested change
println!("\n");

datafusion/physical-expr/src/array_expressions.rs Outdated Show resolved Hide resolved
@@ -986,6 +986,26 @@ macro_rules! general_repeat_list {
}};
}

/// Array_empty SQL function
pub fn array_empty(args: &[ArrayRef]) -> Result<ArrayRef> {
if args[0].as_any().downcast_ref::<NullArray>().is_some() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this doesn't catch all nulls. This only works within the test setup because array_empty(NULL) creates a NULL array. I think this will fail w/:

SELECT array_empty(a)
FROM VALUES(NULL, make_array(), make_array(1)) sub (a);

I suggest you remove this check and change the else part within array.iter().map(...) to None (from Some(true)).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole handling of nulls in the array function logic is "maturing" I would say (e.g. #7142 and others listed on #6980)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but that doesn't mean we have to add more immature code, esp. when the fix is rather simple (I would argue the fixed code is even simpler than the current version).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest you remove this check and change the else part within array.iter().map(...) to None (from Some(true)).

It cannot handle all NULL array cases in this way, but I think we can wait #7142 to make this pr better.

@alamb alamb merged commit 6281637 into apache:main Aug 22, 2023
22 checks passed
@alamb
Copy link
Contributor

alamb commented Aug 22, 2023

Thanks everyone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
logical-expr Logical plan and expressions physical-expr Physical Expressions sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement empty function
5 participants