Add progress log to HTTP downloads

[leonardehrenfried]

Summary

This adds a progress log for HTTP downloads which give you a bit of visibillity when OTP does large downloads.

Issue

❌

Unit tests

❌

[codecov]

Codecov Report

Attention: Patch coverage is 14.28571% with 6 lines in your changes missing coverage. Please review.

Project coverage is 69.78%. Comparing base (e9f847d) to head (84199bf).
Report is 4 commits behind head on dev-2.x.

Files with missing lines	Patch %	Lines
...ipplanner/datastore/https/HttpsFileDataSource.java	0.00%	4 Missing ⚠️
...anner/datastore/https/HttpsDataSourceMetadata.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             dev-2.x    #6065      +/-   ##
=============================================
- Coverage      69.78%   69.78%   -0.01%     
- Complexity     17358    17359       +1     
=============================================
  Files           1962     1962              
  Lines          74359    74362       +3     
  Branches        7624     7624              
=============================================
  Hits           51895    51895              
- Misses         19821    19825       +4     
+ Partials        2643     2642       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[habrahamsson-skanetrafiken]

good catch

[t2gran]

Currently progress tracking is done one abstraction level up. E.g. this will result in double tracking of OSM loading data. The benefit of adding this one level up, is that we have better/more detailed control of the context like use-case, expected speed and source. If added to the data-source it should be added as a generic feature that is opt-in and with configurable log message. Adding it to one DataSource and not the others is inconsistent. If I load it from HTTP I get tracking, but not if I load it from Google Cloud. Since we have few data-sources I think keeping this responsibility out of the DataSource component is a good thing.

I suggests searching for the use of the asInputStream() and asOutputStream() and add progress tracking those locations instead - where the content is expected to take more than 5 seconds for a big source. In places where it is small, I would not add it - since it has a small overhead(synchronization - ignorable if we are fetching data outside mem).

Another reason for not adding it the the data-source is that the progress tracker always log, so for things that are so small that we do not want to track them, the tracker will create noice.

Another way to identify places to add tracking is to look through the logs and look for gaps in the logging that is more than 5 sec. If found, it is likely that there is a big task in the gap that shuld be tracked.

[t2gran]

When I looked at this I identified on place witch is missing logging - reading the graph:

@SuppressWarnings("Convert2MethodRef")
  public static SerializedGraphObject load(DataSource source) {
    return load(source.asInputStream(), source.path());
    InputStream in = ProgressTracker.track("Reading graph '" + source.name() + "'",
      64*1024,
      source.size(),
      source.asInputStream(),
      // Keep lambda, do not convert to method-ref. Make the logger
      // report the correct class & line number.
      m -> LOG.info(m)
      );
    return load(in, source.path());
  }

(SerializedGraphObject:124)