Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(query): function about convert_timezone #16177 #16181

Open
wants to merge 71 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
499a4ac
Feature: function about convert_timezone #16177
florann Aug 4, 2024
947110c
Merge branch 'main' into feature_#16177
florann Aug 4, 2024
04cf38e
Correction trait type
florann Aug 4, 2024
9b93737
Merge remote-tracking branch 'myfork/feature_#16177' into feature_#16177
florann Aug 4, 2024
8a26246
Formatting the function
florann Aug 4, 2024
2ba4ef6
Solving compiling issue
florann Aug 4, 2024
72331aa
Correcting file format
florann Aug 4, 2024
ef04fd0
adding tests
florann Aug 4, 2024
86942fc
Correction unti test
florann Aug 4, 2024
3e3cada
Correction unit tests 2
florann Aug 4, 2024
4a7a668
Correction unit tests 3
florann Aug 4, 2024
ab8d39e
Correction unit tests 4
florann Aug 4, 2024
286b259
Correction unit tests 5
florann Aug 4, 2024
cf248e3
Correction unit tests 6
florann Aug 4, 2024
be4161a
Correction unit tests 6
florann Aug 4, 2024
85e70b6
Correction unit tests 8
florann Aug 4, 2024
b33db2e
Correction unit tests 9
florann Aug 4, 2024
b2e33b0
Implementation of convert_timezone
florann Aug 5, 2024
490aa6e
Correction registry and error handling
florann Aug 5, 2024
f93d572
Correction eval function
florann Aug 5, 2024
261fd1e
Correction eval function 2
florann Aug 5, 2024
1a748fe
Correction eval function 3
florann Aug 5, 2024
7c71cbe
Correction eval function 4
florann Aug 5, 2024
05f019b
Correction eval function 5
florann Aug 5, 2024
c08e5ee
Correction eval function 6
florann Aug 5, 2024
53c64ce
Correction eval function 7
florann Aug 5, 2024
a0b3327
Correction eval function 8
florann Aug 5, 2024
1d9c360
Merge branch 'main' into feature_#16177
florann Aug 5, 2024
bcdaf5e
Correction eval function 9
florann Aug 5, 2024
1a514be
Merge branch 'main' of https://github.com/datafuselabs/databend into …
florann Aug 5, 2024
f244b23
Merge branch 'feature_#16177' of https://github.com/florann/databend …
florann Aug 5, 2024
819c225
Correction eval function 10
florann Aug 5, 2024
1898ea5
Correction eval function 11
florann Aug 5, 2024
90d3017
Correction eval function 12
florann Aug 5, 2024
609c74e
Merge branch 'main' into feature_#16177
florann Aug 6, 2024
311f4e0
Correction eval function 13 + add of the 3 arguments function version
florann Aug 6, 2024
ae55f91
Code formatting
florann Aug 6, 2024
15b8c3e
Correction third parameter convert_timezone()
florann Aug 6, 2024
e20df6d
Correction type
florann Aug 6, 2024
2d80243
Correction 1
florann Aug 6, 2024
dba9dbf
Correction 2
florann Aug 6, 2024
a1f1b9d
Adding unit test
florann Aug 6, 2024
871cb8b
Adding unit test 2
florann Aug 6, 2024
f931c90
Adding unit test 3
florann Aug 6, 2024
9acde6d
Correciton unit test 4
florann Aug 6, 2024
07765bc
Correciton unit test 5
florann Aug 6, 2024
40eaa13
Correciton unit test 6
florann Aug 6, 2024
d0dc745
Correciton unit test 7
florann Aug 6, 2024
d5555c7
Correction unit test 8
florann Aug 6, 2024
5b7e576
Correction checked expr
florann Aug 7, 2024
191c57d
Correction convert_timezone
florann Aug 7, 2024
6ebe9ca
Correction convert_timezone 2
florann Aug 7, 2024
637b9ea
Correction convert_timezone 3
florann Aug 7, 2024
5ac0b1e
Correction convert_timezone 4
florann Aug 7, 2024
98d6fe3
Correction convert_timezone 5
florann Aug 7, 2024
09e6ee4
Correction convert_timezone 6
florann Aug 7, 2024
a2a6b2a
Correction convert_timezone 7
florann Aug 7, 2024
70d1176
Correction convert_timezone 8
florann Aug 7, 2024
ccb349c
Correction convert_timezone 9
florann Aug 8, 2024
34fb1c2
Correction convert_timezone 10
florann Aug 8, 2024
65ae38b
Correction convert_timezone 11
florann Aug 8, 2024
3865762
Correction convert_timezone 12
florann Aug 8, 2024
9d7b425
Correction convert_timezone 12
florann Aug 8, 2024
00abbef
Correction convert_timezone 13
florann Aug 8, 2024
aec53b6
Unit test
florann Aug 9, 2024
0c504aa
Unit test 2
florann Aug 9, 2024
542daca
Unit test 3
florann Aug 9, 2024
a596664
Unit test 4
florann Aug 9, 2024
23b7a02
Unit test 5
florann Aug 9, 2024
dcb449c
Unit test 6
florann Aug 9, 2024
6283302
Unit test 7
florann Aug 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions src/query/functions/src/scalars/datetime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ use databend_common_expression::vectorize_1_arg;
use databend_common_expression::vectorize_2_arg;
use databend_common_expression::vectorize_with_builder_1_arg;
use databend_common_expression::vectorize_with_builder_2_arg;
use databend_common_expression::vectorize_with_builder_3_arg;
use databend_common_expression::EvalContext;
use databend_common_expression::FunctionDomain;
use databend_common_expression::FunctionProperty;
Expand Down Expand Up @@ -108,6 +109,9 @@ pub fn register(registry: &mut FunctionRegistry) {

// [date | timestamp] +/- number
register_timestamp_add_sub(registry);

// convert_timezone( target_timezone, date, src_timezone)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The arguments are in the wrong order. They should be src_timezone, target_timezone, source_timestamp

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

Need support

CONVERT_TIMEZONE( <source_tz> , <target_tz> , <source_timestamp_ntz> )

and

CONVERT_TIMEZONE( <target_tz> , <source_timestamp> )

register_convert_timezone(registry);
}

/// Check if timestamp is within range, and return the timestamp in micros.
Expand All @@ -131,6 +135,109 @@ fn int64_domain_to_timestamp_domain<T: AsPrimitive<i64>>(
})
}

fn register_convert_timezone(registry: &mut FunctionRegistry) {
// 2 arguments function [target_timezone, src_timestamp]
registry.register_passthrough_nullable_2_arg::<StringType, TimestampType, TimestampType, _, _>(
"convert_timezone",
|_, _, _| FunctionDomain::MayThrow,
eval_convert_timezone,
);

// 3 arguments function [src_timezone, target_timezone, src_timestamp]
registry.register_passthrough_nullable_3_arg::<StringType, StringType, TimestampType, TimestampType, _, _>(
"convert_timezone",
|_, _, _, _| FunctionDomain::MayThrow,
vectorize_with_builder_3_arg::<StringType, StringType, TimestampType, TimestampType>(
|src_tz, target_tz, src_timestamp, output, ctx| {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check validity here to handle NULL value

            if let Some(validity) = &ctx.validity {
                if !validity.get_bit(output.len()) {
                    output.push_default();
                    return;
                }
            }

// Parsing parameters
let t_tz: Tz = match target_tz.parse() {
Ok(tz) => tz,
Err(e) => {
return ctx.set_error(
output.len(),
format!("cannot parse target `timezone`. {}", e),
);
}
};
let s_tz: Tz = match src_tz.parse() {
Ok(tz) => tz,
Err(e) => {
return ctx.set_error(
output.len(),
format!("cannot parse src `timezone`. {}", e),
);
}
};

let p_src_timestamp: i64 = match Some(src_timestamp){
Ok(timestamp) => {
timestamp.Utc.timestamp_opt(src_timestamp, 0).unwrap();
},
Err(e) => {
return ctx.set_error(
output.len(),
format!("cannot parse target `timezone`. {}", e),
);
}
};


// Create dummy timezone
let utc_now: DateTime<Utc> = Utc::now();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think no need to create dummy timezone. You can use this https://docs.rs/chrono/latest/chrono/offset/trait.TimeZone.html#method.timestamp_opt to convert ts to source timezone.

Then convert the ts with src tz to target tz.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


let src_time = utc_now.with_timezone(&s_tz);
let target_time = utc_now.with_timezone(&t_tz);

// Calculate the difference in seconds
let delta = target_time.signed_duration_since(src_time);
let result_timestamp = p_src_timestamp + delta.num_seconds();

output.push(result_timestamp)
},
),
);

fn eval_convert_timezone(
target_tz: ValueRef<StringType>,
src_timestamp: ValueRef<TimestampType>,
ctx: &mut EvalContext,
) -> Value<TimestampType> {
vectorize_with_builder_2_arg::<StringType, TimestampType, TimestampType>(
|target_tz, src_timestamp, output, ctx| {
// Parse the target timezone
let t_tz: Tz = match target_tz.parse() {
Ok(tz) => tz,
Err(e) => {
return ctx.set_error(
output.len(),
format!("cannot parse target `timezone`. {}", e),
);
}
};

// Assume the source timestamp is in UTC
match Some(src_timestamp) {
Ok(src_timestamp) => {
let timestamp: i64 = src_timestamp;
let datetime: DateTime<Utc> = Utc.timestamp_opt(src_timestamp, 0).unwrap();

// Convert the UTC time to the specified target timezone
let target_datetime: DateTime<Tz> = datetime.with_timezone(&t_tz);
let result_timestamp = target_datetime.timestamp();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timestamp_micros()

// Return the adjusted timestamp as a Unix timestamp in seconds
output.push(result_timestamp)
}
Err(e) => {
return ctx.set_error(
output.len(),
format!("cannot parse target `timezone`. {}", e),
);
}
};
},
)(target_tz, src_timestamp, ctx)
}
}
fn register_string_to_timestamp(registry: &mut FunctionRegistry) {
registry.register_aliases("to_date", &["str_to_date", "date"]);
registry.register_aliases("to_year", &["str_to_year", "year"]);
Expand Down
36 changes: 36 additions & 0 deletions src/query/functions/tests/it/scalars/datetime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,42 @@ fn test_datetime() {
test_timestamp_arith(file);
test_to_number(file);
test_rounder_functions(file);
test_convert_timezone(file);
}

fn test_convert_timezone(file: &mut impl Write) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool.

SELECT CONVERT_TIMEZONE(
'America/Los_Angeles',
'America/New_York',
to_timestamp('2024-01-01 14:00:00')
) AS conv;

SELECT CONVERT_TIMEZONE(
'Europe/Warsaw',
'UTC',
'2024-01-01 00:00:00'::TIMESTAMP
) AS conv;

SELECT CONVERT_TIMEZONE(
'America/Los_Angeles',
'2024-04-05 12:00:00 +02:00'::TIMESTAMP
) AS time_in_la;

SELECT
CURRENT_TIMESTAMP() AS now_in_la,
CONVERT_TIMEZONE('America/New_York', CURRENT_TIMESTAMP()) AS now_in_nyc,
CONVERT_TIMEZONE('Europe/Paris', CURRENT_TIMESTAMP()) AS now_in_paris,
CONVERT_TIMEZONE('Asia/Tokyo', CURRENT_TIMESTAMP()) AS now_in_tokyo;

Need to add these tests in logical test.

run_ast(
file,
"convert_timezone('America/New_York', to_timestamp(100))",
&[],
);
run_ast(
file,
"convert_timezone('America/New_York', to_timestamp(-100))",
&[],
);
run_ast(
file,
"convert_timezone('America/New_York', to_timestamp(0))",
&[],
);
run_ast(
file,
"convert_timezone('America/New_York', to_timestamp(315360000000))",
&[],
);

run_ast(
file,
"convert_timezone('America/New_York', 'Europe/Simferopol', to_timestamp(100))",
&[],
);

run_ast(
file,
"convert_timezone('America/New_York', 'Europe/Simferopol', to_timestamp(0))",
&[],
);
}

fn test_to_timestamp(file: &mut impl Write) {
Expand Down
8 changes: 7 additions & 1 deletion src/query/functions/tests/it/scalars/testdata/datetime.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3344,4 +3344,10 @@ output type : Timestamp
output domain : {1630812366000000..=1630812366000000}
output : '2021-09-05 03:26:06.000000'


ast : convert_timezone('2024-08-06 14:30:00 +02:00','America/New_York')
raw expr : convert_timezone('2024-08-06 14:30:00 +02:00','America/New_York')
checked expr : convert_timezone<String,String,Timestamp> ('2024-08-06T14:30:00+02:00','America/New_York')
optimized expr : '2024-08-06T08:30:00.000000'
output type : Timestamp
output domain : {'2024-08-06T08:30:00.000000'..='2024-08-06T08:30:00.000000'}
output : '2024-08-06 08:30:00.000000'
Loading