Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Image -> Arrow support #8330

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Image -> Arrow support #8330

wants to merge 7 commits into from

Conversation

wiredfool
Copy link
Member

Partial implementation of #8329.

This is for discussion -- there are a number of issues.

Changes proposed in this pull request:

  • Implements basic __arrow_c_array__ and __arrow_c_schema__ support for reading Pillow images in Arrow capable libraries.

Issues:

  • Currently fails on large images.
  • There's extraneous stuff in Arrow.c that should be cleaned up.
  • Some changes in Storage.c aren't required.
  • Unclear if we're exporting the most useful version of the data.

* Fixed format, only for 4 channel images
* structs can't be encoded this way, they need to have one child array
per struct member.
* i.e. struct of arrays, rather than an array of structs.
@wiredfool wiredfool mentioned this pull request Aug 25, 2024

def __arrow_c_array__(
self, requested_schema: object | None = None
) -> Tuple[object, object]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
) -> Tuple[object, object]:
) -> tuple[object, object]:

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyarrow still supports Python 3.8 - https://pypi.org/project/pyarrow/
Since we don't, it can be updated - https://peps.python.org/pep-0585/#implementation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants