Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exploring tiledb's dataloaders #38

Open
jkanche opened this issue Aug 4, 2024 · 1 comment
Open

exploring tiledb's dataloaders #38

jkanche opened this issue Aug 4, 2024 · 1 comment
Assignees

Comments

@jkanche
Copy link
Member

jkanche commented Aug 4, 2024

I ran into Tiledb's ML repo (https://github.com/TileDB-Inc/TileDB-ML/blob/master/tiledb/ml/readers/pytorch.py), which seems to implement a pytorch based dataloader. It looks like we can speed up our dataloaders without setting threads to 1.

Mostly exploration and then figuring out if we can adopt the same logic to our implementations.

@tony-kuo
Copy link
Collaborator

Setting context to "spawn" will remove the need to setting threads to 1.

  1. I'm going to refactor the dataloader to reduce the number of database accesses to one per mini-batch
  2. I'm going to move to a sample based epoch rather than multi-set. I think it makes more sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants