Profile memory consumption of harvest runner under load

[btylerburton]

User Story

In order to identify memory leaks in the harvester, datagovteam wants to conduct a formal analysis using industry standard memory profiling tools.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN I have integrated a memory profiler into the datagov-harvester codebase
  AND it has been tested locally and show to perform
  THEN I want to push it to cloud.gov and run a proper load test of a large harvest source, such as DOI.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Select a memory profiler library
• • such as https://github.com/bloomberg/memray
• Confirm it outputs meaningful data locally
• Push to cloud.gov
• Test on a small source
• Confirm output is meaningful
• Test on DOI source

[btylerburton]

Discussed whether breaking the harvest up into discrete processes: extract, transform, validate, sync/load will make this a non-issue. We can leave this ticket as a might-do just in case that doesn't work.