Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use NullBuffer in ArrayData #3775

Closed
tustvold opened this issue Feb 28, 2023 · 2 comments · Fixed by #3778
Closed

Use NullBuffer in ArrayData #3775

tustvold opened this issue Feb 28, 2023 · 2 comments · Fixed by #3778
Assignees
@tustvold
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@tustvold
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

#3749 added a new NullBuffer abstraction, as part of #1176 it is necessary to switch ArrayData over to using it.

A complicating factor is that ArrayData::offset currently applies to both the values and the null buffer, this leads to two options:

  1. Continue to provide ArrayData::null_buffer returning the underlying Buffer relying on clients to apply the offset
  2. Change the return type of ArrayData::null_buffer to NullBuffer breaking downstreams

Describe the solution you'd like

Option 2. puts us in a better place for #1176 as it leaves the offset as just applying to the array values, allowing for the null buffer to have a different offset. It also leads to a better API for users.

Describe alternatives you've considered

Additional context
@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label Feb 28, 2023
@tustvold tustvold self-assigned this Feb 28, 2023
tustvold added a commit to tustvold/arrow-rs that referenced this issue Feb 28, 2023
@tustvold
Use NullBuffer in ArrayData (apache#3775)
cab7257
tustvold added a commit to tustvold/arrow-rs that referenced this issue Feb 28, 2023
@tustvold
Use NullBuffer in ArrayData (apache#3775)
63829e3
@tustvold tustvold mentioned this issue Feb 28, 2023
Merged
tustvold added a commit that referenced this issue Mar 2, 2023
@tustvold
Use NullBuffer in ArrayData (#3775) (#3778)
eff058f 
* Use NullBuffer in ArrayData (#3775)

* Clippy

* Format

* Doc

* Tweaks

* Review feedback
@tustvold tustvold added the parquet Changes to the parquet crate label Mar 10, 2023
@tustvold
Copy link
Contributor Author

label_issue.py automatically added labels {'parquet'} from #3778

@tustvold tustvold added the arrow Changes to the arrow crate label Mar 10, 2023
@tustvold
Copy link
Contributor Author

label_issue.py automatically added labels {'arrow'} from #3778

@tustvold tustvold mentioned this issue Jun 2, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tustvold tustvold

Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use NullBuffer in ArrayData (#3775) tustvold/arrow-rs
1 participant
@tustvold