Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP(iox-11398): patched df upgrade 2024-07-08 #33

Closed
wants to merge 8 commits into from

Commits on Jul 12, 2024

  1. fix: Incorrect LEFT JOIN evaluation result on OR conditions (apache#1…

    …1203)
    
    * fix: Incorrect LEFT JOIN evaluation result on OR conditions
    
    * Add a few more test cases
    
    * Don't push join filter predicates into join_conditions
    
    * Add test case and fix typo
    
    * Add test case
    
    ---------
    
    Co-authored-by: Andrew Lamb <[email protected]>
    2 people authored and appletreeisyellow committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    ccf2661 View commit details
    Browse the repository at this point in the history
  2. feat: add UDF to_local_time() (apache#11347)

    * feat: add UDF `to_local_time()`
    
    * chore: support column value in array
    
    * chore: lint
    
    * chore: fix conversion for us, ms, and s
    
    * chore: add more tests for daylight savings time
    
    * chore: add function description
    
    * refactor: update tests and add examples in description
    
    * chore: add description and example
    
    * chore: doc
    
    chore: doc
    
    chore: doc
    
    chore: doc
    
    chore: doc
    
    * chore: stop copying
    
    * chore: fix typo
    
    * chore: mention that the offset varies based on daylight savings time
    
    * refactor: parse timezone once and update examples in description
    
    * refactor: replace map..concat with flat_map
    
    * chore: add hard code timestamp value in test
    
    chore: doc
    
    chore: doc
    
    * chore: handle errors and remove panics
    
    * chore: move some test to slt
    
    * chore: clone time_value
    
    * chore: typo
    
    ---------
    
    Co-authored-by: Andrew Lamb <[email protected]>
    appletreeisyellow and alamb committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    3fbe3d4 View commit details
    Browse the repository at this point in the history
  3. Track parquet writer encoding memory usage on MemoryPool (apache#11345)

    * feat(11344): track memory used for non-parallel writes
    
    * feat(11344): track memory usage during parallel writes
    
    * test(11344): create bounded stream for testing
    
    * test(11344): test ParquetSink memory reservation
    
    * feat(11344): track bytes in file writer
    
    * refactor(11344): tweak the ordering to add col bytes to rg_reservation, before selecting shrinking for data bytes flushed
    
    * refactor: move each col_reservation and rg_reservation to match the parallelized call stack for col vs rg
    
    * test(11344): add memory_limit enforcement test for parquet sink
    
    * chore: cleanup to remove unnecessary reservation management steps
    
    * fix: fix CI test failure due to file extension rename
    wiedld authored and appletreeisyellow committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    4f98494 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. fix(11397): surface proper errors in ParquetSink (apache#11399)

    * fix(11397): do not surface errors for closed channels, and instead let the task join errors be surfaced
    
    * fix(11397): terminate early on channel send failure
    
    Add Optimizer Sanity Checker, improve sortedness equivalence properties (apache#11196)
    
    * Initial optimizer sanity checker.
    
    Only includes sort reqs, docs will be added.
    
    * Add distro and pipeline friendly checks
    
    * Also check the plans we create are correct.
    
    * Add distribution test cases using global limit exec.
    
    * Add test for multiple children using SortMergeJoinExec.
    
    * Move PipelineChecker to SanityCheckPlan
    
    * Fix some tests and add docs
    
    * Add some test docs and fix clippy diagnostics.
    
    * Fix some failing tests
    
    * Replace PipelineChecker with SanityChecker in .slt files.
    
    * Initial commit
    
    * Slt tests pass
    
    * Resolve linter errors
    
    * Minor changes
    
    * Minor changes
    
    * Minor changes
    
    * Minor changes
    
    * Sort PreservingMerge clear per partition
    
    * Minor changes
    
    * Update output_requirements.rs
    
    * Address reviews
    
    * Update datafusion/core/src/physical_optimizer/optimizer.rs
    
    Co-authored-by: Mehmet Ozan Kabak <[email protected]>
    
    * Update datafusion/core/src/physical_optimizer/sanity_checker.rs
    
    Co-authored-by: Mehmet Ozan Kabak <[email protected]>
    
    * Address reviews
    
    * Minor changes
    
    * Apply suggestions from code review
    
    Co-authored-by: Andrew Lamb <[email protected]>
    
    * Update comment
    
    * Add map implementation
    
    ---------
    
    Co-authored-by: Erman Yafay <[email protected]>
    Co-authored-by: berkaysynnada <[email protected]>
    Co-authored-by: Mehmet Ozan Kabak <[email protected]>
    Co-authored-by: Andrew Lamb <[email protected]>
    5 people authored and appletreeisyellow committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    fac9e69 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6756fab View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5feaa41 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    dd28455 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    acce8ce View commit details
    Browse the repository at this point in the history