Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support reading input directly from files #3

Open
jasonnance opened this issue Sep 6, 2019 · 0 comments
Open

Support reading input directly from files #3

jasonnance opened this issue Sep 6, 2019 · 0 comments
Labels
enhancement New feature or request

Comments

@jasonnance
Copy link
Collaborator

Feature

Tasks that take input can accept an in-memory list or a path to a file with some specified format with optional validation.

Motivation

Prevents users' datasets from having to fit in memory on gobbli's side (although we can't easily control how some models are reading it in, so this may not make a difference for those models). May save time in some cases, since we have to write data to disk anyway to make it available to model Docker containers, so if it was there to begin with, the initial read/write is somewhat unnecessary.

Additional Details

Ideally we'd modify the model wrappers where we control data input (ex. Transformer, MT-DNN) to optionally lazily load data from files to completely eliminate the possibility of exhausting RAM.

@jasonnance jasonnance added the enhancement New feature or request label Sep 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant