Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add individual download progress #42

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add individual download progress #42

wants to merge 5 commits into from

Conversation

ptgolden
Copy link
Member

This PR makes a few major changes:

  1. There are now progress bars for individual downloads. They look like this: image

  2. Instead of erroring out when a download runs into an error, the program will log that an error occurred and continue on. The error can be reported with a new verbose flag/parameter. The log above the current progress bars looks like this:

SKIPPING: ../monarch-ingest/data/string/10090.protein_links.txt.gz already exists
OK: Downloaded https://stringdb-downloads.org/download/protein.links.detailed.v12.0/9615.protein.links.detailed.v12.0.txt.gz                                                                                                               
SKIPPING: ../monarch-ingest/data/string/9913.protein_links.txt.gz already exists
SKIPPING: ../monarch-ingest/data/string/9031.protein_links.txt.gz already exists
ERROR: Failed to download gs://monarch-test/kghub_downloader_test_file.yaml
SKIPPING: ../monarch-ingest/data/xenbase/LiteratureMatchedGenesByPaper.txt already exists
SKIPPING: ../monarch-ingest/data/xenbase/xb_xpo_spo_v_v1.tab already exists
OK: Downloaded http://ftp.xenbase.org/pub/GenePageReports/XenbaseGeneHumanOrthologMapping.txt          
SKIPPING: ../monarch-ingest/data/xenbase/XenbaseGeneMouseOrthologMapping.txt already exists
  1. The command generates a report at the end of the download showing some statistics:
Download completed in 14.12 seconds.  

    successful:   18
    skipped:      73
    unsuccessful: 2

Some downloads were unsuccessful. Run with --verbose to see errors

@ptgolden
Add progress bars for individual downloads
566a11e 
Additionally, this commit prevents the program from exiting if a single
resource has an error. Instead, that error will be noted (and details
will be given if the --verbose flag is passed), and downloading of
subsequent resources will continue.

One bug is that the `gdown` library manages its own progress bars, so
they do not appear in the same way as the ones generated here. There are
three possible solutions I see, from easiest to hardest

  1. Ignore it

  2. Pass the quiet option to gdown (eliminating any progress bar)

  3. Send a patch to gdown to accept a file descriptor to which to
     download, rather than a filename. (Unlikely to be accepted)

Taking (1) for now!
@ptgolden
Don't call downloader functions from other downloader functions
8c6fec8 
This prevents the resource from being logged twice. I thought about
doing something more complex to allow the functions to call each other,
but it was to messy to justify not repeating a couple lines of code.
@ptgolden
Use boto.resource, rather than boto.client
4f00931
@ptgolden
Raise for non-200 status errors in http downloader
a43956e
@ptgolden
Fix bug in s3 download code
da03a63
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ptgolden