[EPIC] DCAT-US Harvesting Pipeline MVP

[btylerburton]

Feature/what we're after

Create an MVP for a DCAT-US Harvesting Pipeline

Anticipated/hypothesized benefits

• Will allow team to define approach to pipeline development
• Will allow us to "design on the fly" while still holding to best practices
• Will allow us to test our assumptions

Measurements/metrics

• Benchmark processing speed for each pipeline component
• [... more metrics here ...]

Implmentation/tickets

For the MVP, what would be cast as epics will be organized into singular stories. New tickets created for the MVP will reference the parent Production ticket that will remain in "New Dev". Anything marked with [TODO] has open questions.

Infrastructure

• [TODO] //do changes to the SSB need to be done for the MVP?
• [TODO] //do we need to modify brokerpaks to get a working MVP?

Controller

• [MVP] Initialize Flask app #4398
• [TODO] Controller API routes //define the routes w/ team
• [TODO] Controller blueprint and dashboard URL //do we need a dashboard for MVP?
• Integrate DB client to track harvest jobs
  • Harvest source table:
    • contains configs for each harvest source
  • Harvest job table:
    • contains job state information for each job that's run through the pipeline

Extract

• Integrate & Extend S3 client
• Integrate & Extend DB client

Compare

• Integrate & Extend S3 client
• Integrate & Extend DB client
• Integrate Catalog UI API call

Transform

• Not necessary since it's a no-op DCAT -> DCAT

Load

Validate

Modules

Add additional module tickets here as they are created

• Refactor S3 client
• Refactor Postgres client
• Create DB tables using Alembic
• [MVP] Define DB using SQLAlchemy #4400

[btylerburton]

This effort has been superceded by Airflow Spike #4422