Skip to content

Releases: m-lab/etl-gardener

Fix broken sidestream source bucket

15 Oct 23:11
Compare
Choose a tag to compare
Pre-release

Sidestream reprocessing should now pull from the unified archive in archive-mlab-oti. This updates the config accordingly.
This also fixes a bug with CopyConfig that was causing final copies to fail.

Note that, at time of release, the archive-mlab-oti change is not yet integrated into master.

Run Gardener in production with more datasets

10 Oct 20:01
9717a4b
Compare
Choose a tag to compare

This release of gardener includes recent changes, such as:

  • Separate NDT, Sidestream, and Switch (DISCO) dataset configurations
  • Update Datastore schema to partition 'tasks' by Experiment name
  • Adds Copy&Replace with sanity checking in the Finalize stage
  • Add contexts throughout
  • Improve error handling with bqiface
  • Handler additional task error states
  • Other major code improvements

Production release - task based with restart

20 Sep 15:45
9557f5e
Compare
Choose a tag to compare
  1. There is now one goroutine per task (date*experiment). The config is set up only for sidestream, from 2017/06/01 forward.
  2. Task state is stored in datastore, and tasks are retrieved and restarted when deployment restarts.

Currently, partitions immediately overwrite mlab-oti:private.sidestream, so once we set up the public view, public data will be updated without prior sanity checking.

Production Release

16 Jul 13:08
46258a9
Compare
Choose a tag to compare

Improvements in state tracking.
Most changes since last release are in preparation for switching to task state machines, but this is not enabled yet. There have been a handful of other changes that impact the active code in fairly minor ways, mostly improving logging, and recording of state changes in datastore.

Prerelease to process sidestream data for discard analysis

12 Jun 02:23
Compare
Choose a tag to compare

Adds nodeSelector.

Prerelease to process sidestream data for discard analysis

11 Jun 19:56
88d3bdd
Compare
Choose a tag to compare

This is a temporary release intended only to process the full year of sidestream data for discard analysis.

Hack to reprocess sidestream for discard analysis

08 Jun 13:09
Compare
Choose a tag to compare

This is a temporary version, hacked to reprocess the sidestream data needed for the final discard analysis for a blog post.

Prerelease to test production cluster config.

19 Apr 14:37
568d30a
Compare
Choose a tag to compare

Adds bug workaround for task queue statistics bug, and overrides BQ project from mlab-oti to measurement-lab

Prerelease to test production cluster config.

18 Apr 19:51
f3e2e92
Compare
Choose a tag to compare

Adds bug workaround for task queue statistics bug.

Prerelease to test production cluster config.

03 Apr 02:42
Compare
Choose a tag to compare

Small code hack to select measurement-lab BQ project, instead of mlab-oti