Update gardener daily start time and yesterday calculation

Fix bug in computing starting yesterday (#212)
Tracker and JobState persistence (#201)
Use dsfake.Client for testing (#214)
Offset dailyDelay to 04:10 to allow GCS transfers to complete (#216)

Source disco data from archive-measurement-lab/utilization/switch

This modifies gardener to source disco data from utilization/switch instead of legacy switch directory.

Just two PRs:
Read switch data from utilization/switch folder (#208) …
define Job (#210) …

Adjustments to load balancing parameters, 3am reprocessing of yesterday's data.

Many small changes. Most notable are:

1. the introduction of the "universal" instance, in sandbox and staging only,
2. adjustments to the processing load balancing.
3. New feature to process yesterday's data at 3:00 am UTC
4. Updates to allow using go 1.13 in travis.

use most recent travis library (#189) …
fix prom startup and add block profiling (#190) …
Refactor main (#191) …
Add custom network info to README.md (#192) …
Cleanup main() (#194) …
Add gardener-universal with static ip (#193) …
update readme for dp network (#195) …
Add general persistence utility. (#197) …
faster tcpinfo, slower sidestream and ndt (#199) …
add firewall commands (#196) …
Add DatastoreSaver.FetchAll (#200) …
Add reprocessing of yesterday at 3:00 am UTC. (#202) …
Update travis to use go 1.13 (#206) …

feed ndt5 and tcp-info from archive bucket instead of pusher bucket

Update yaml file to read from archive-* bucket instead of pusher-* bu…

…cket (#186)

* Update etl-gardener-ndt5.yml

* Update etl-gardener-tcpinfo.yml

* Update etl-gardener-ndt5.yml

* Update etl-gardener-tcpinfo.yml

Add task queues for Scamper JSON parser

add yaml file for scamper json output (#184)

* Create etl-gardener-scamper.yml

* Update etl-gardener-scamper.yml

* Update persistent-volumes.yml

* Update etl-gardener-scamper.yml

* Update etl-gardener-scamper.yml

* Update persistent-volumes.yml

* Update etl-gardener-scamper.yml

* Update etl-gardener-scamper.yml

Updates for NDT5 parsing

Update ndt5 configuration (#175) …
Check for ndt5 before ndt to prevent false positive (#176) …

Update Gardener deployment for unified NDT result (retiring "legacy")

This release adds a new deployment for the unified NDT result tables and retires the now obsolete 'legacy' deployment.

Changes include:

• Sandbox ratelimitexceeded (#172)
• Support NDT Result unified schema reprocessing for ndt5 (#173)

Fix panic in error path

This fixes a nil pointer dereference in error path when Job.Wait() returns an error. This was encountered in new traceroute instance, but only in prod. Unclear why.

Overrun prevention, and new traceroute schema handling.

This release:

1. Adds paris traceroute instance.
2. Adds overrun prevention, so we don't get multiple tasks trying to manage same date.

Merged #168 update traceroute dedup sql 4 days ago
Merged #166 fix behavior of nosuchentity 6 days ago
Merged #162 Sandbox avoid overrun 7 days ago
Merged #165 bring pt back (from clean master) 8 days ago
Merged #163 Sandbox ptback 8 days ago

There are no major changes, but advancing to v1.0.0, as it should have been long ago.

Add NDTLegacy Processing

This release coincides with https://github.com/m-lab/etl/releases/tag/batch-prod-v1.4.0 to begin reprocessing the ndt-server legacy test format. This is a temporary configuration to get access to that data as soon as possible. Ultimately, the ndtLegacy format will be unified with the NDT7 results schema.