Reparse Memory Management #3172

elipe17 · 2024-09-05T14:34:38Z

Summary of Changes

Updated record deletion to use a paginator to avoid loading large quantities of records into memory
Added a little more logging for convenience
Pull request closes Reparse Command Fails when Queryset is Large #3170

How to Test

List the steps to test the PR
These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose up
cd tdrs-backend && docker-compose up

Open http://localhost:3000/ and sign in.
Submit ADS.E2J.NDM1.TS53_fake.txt four or five times to get one million or more records into the DB. (This goes faster if you add more celery workers to gunicorn_start.sh!)
Exec into web and run python manage.py clean_and_reparse -y 2023. If you use the -a argument you actually won't see the issue occur because -a doesn't pass the queryset to Elastic DSL which is what causes it to come into memory.
Monitor the backend memory consumption while deleting records. It should increase, but only around 50-100MB.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

Clean and reparse command does not get killed (in general) when large querysets are involved.

Deliverable 2: Tested Code

Are all areas of code introduced in this PR meaningfully tested?
- If this PR introduces backend code changes, are they meaningfully tested?
- If this PR introduces frontend code changes, are they meaningfully tested?
Are code coverage minimums met?
- Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
- Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

Are backend code style checks passing on CircleCI?
Are frontend code style checks passing on CircleCI?
Are code maintainability principles being followed?

Deliverable 4: Accessible

Does this PR complete the epic?
Are links included to any other gov-approved PRs associated with epic?
Does PR include documentation for Raft's a11y review?
Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

Does this PR provide background for why coding decisions were made?
If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces dependencies, are their licenses documented?
Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

Does the OWASP Scan pass on CircleCI?
Do manual code review and manual testing detect any new security issues?
If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

the purpose of the research
methods used to conduct the research
who participated in the research
what was tested and how
impact of research on TDP
(if applicable) final design mockups produced for TDP development

…whole queryset into memory

- Added extra console logging - changed log level on frequent exception

tdrs-backend/tdpservice/parsers/validators/category3.py

codecov · 2024-09-05T14:44:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.66%. Comparing base (95fc24b) to head (a981311).
Report is 49 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3172   +/-   ##
========================================
  Coverage    92.66%   92.66%           
========================================
  Files           47       47           
  Lines         1009     1009           
  Branches       169      169           
========================================
  Hits           935      935           
  Misses          42       42           
  Partials        32       32

Flag	Coverage Δ
dev-frontend	`92.66% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95fc24b...a981311. Read the comment docs.

andrew-jameson

LGTM

raftmsohani

I suggest we start writing test for clean_and_reparse

- Add tests

- Update all files done logic - Add new test for sequential execution

…ch/TANF-app into 3170-reparse-memory-mgmt

- change order

ADPennington · 2024-09-24T13:50:37Z

testing re-parsing command again for FY23Q1 files. started at 9:18a ET...

ADPennington · 2024-09-24T16:58:01Z

temporarily blocked

testing re-parsing command again for FY23Q1 files. started at 9:18a ET...

temporarily blocked on testing. command froze after attempting to delete approx 1.5mil TANF_T3 records and now the rds is down.

ADPennington · 2024-09-26T14:03:38Z

temporarily blocked

testing re-parsing command again for FY23Q1 files. started at 9:18a ET...

temporarily blocked on testing. command froze after attempting to delete approx 1.5mil TANF_T3 records and now the rds is down.

per @elipe17, and with the help of cloudgov support, the dev environment rds service instance was upgraded to medium-gp-psql with storage_type=gp3 to support qasp review. Will re-try the test today.

…ch/TANF-app into 3170-reparse-memory-mgmt

ADPennington · 2024-09-26T20:04:03Z

temporarily blocked

testing re-parsing command again for FY23Q1 files. started at 9:18a ET...

temporarily blocked on testing. command froze after attempting to delete approx 1.5mil TANF_T3 records and now the rds is down.

per @elipe17, and with the help of cloudgov support, the dev environment rds service instance was upgraded to medium-gp-psql with storage_type=gp3 to support qasp review. Will re-try the test today.

terraform/dev/main.tf

ADPennington · 2024-09-27T17:46:05Z

QASP review update:

this ticket is not blocked, but we are standing by for a response from cloudgov support team re: if there are cost implications for us if we implement the following rds service changes:

Dev: from micro-psql to medium-gp-psql
Staging: from micro-psql to medium-gp-psql
Prod: from medium-psql to medium-gp-psql or large-gp-psql

https://cloud.gov/docs/services/relational-database/

another consideration (if there are cost implications) is #3106 or #3108, which will give us more control over how many records are re-parsed at a time.

ADPennington · 2024-09-30T13:20:19Z

QASP review update:

this ticket is not blocked, but we are standing by for a response from cloudgov support team re: if there are cost implications for us if we implement the following rds service changes:

Dev: from micro-psql to medium-gp-psql

Staging: from micro-psql to medium-gp-psql

Prod: from medium-psql to medium-gp-psql or large-gp-psql

https://cloud.gov/docs/services/relational-database/

another consideration (if there are cost implications) is #3106 or #3108, which will give us more control over how many records are re-parsed at a time.

we received confirmation that, at least for now, there are no extra costs for the upgrade. @elipe17 should we proceed with making the terraform changes to staging and prod in this ticket?

elipe17 · 2024-09-30T13:30:39Z

QASP review update:
this ticket is not blocked, but we are standing by for a response from cloudgov support team re: if there are cost implications for us if we implement the following rds service changes:

Dev: from micro-psql to medium-gp-psql

Staging: from micro-psql to medium-gp-psql

Prod: from medium-psql to medium-gp-psql or large-gp-psql

https://cloud.gov/docs/services/relational-database/
another consideration (if there are cost implications) is #3106 or #3108, which will give us more control over how many records are re-parsed at a time.

we received confirmation that, at least for now, there are no extra costs for the upgrade. @elipe17 should we proceed with making the terraform changes to staging and prod in this ticket?

@ADPennington I think we should. If/when cost comes into the equation we can change the TF again to mitigate that. Since we don't have to take any manual steps to change the service plan this is easy to manipulate and change as our needs change. I will go ahead and make the changes. Note the tests are going to fail on this after a I make changes until this PR merges.

- Give Prod at least 500GB

ADPennington

looks great @elipe17 🚀 testing notes below cc: @ttran-hub @klinkoberstar ⬇️

after upgrading rds to medium-gp-psql, tried the following fiscal periods:

python manage.py clean_and_reparse -y 2023 -q Q1 (N=90 files) 👍
python manage.py clean_and_reparse -y 2024 (N=300 files) 👍
python manage.py clean_and_reparse -y 2023 -q Q2 (N=20 files) 👍

evidence of logentries

evidence of deleting objects

Backup complete! Commencing clean and reparse.
2024-09-30 17:40:02,518 INFO clean_and_reparse.py::_delete_summaries:L88 :  Deleting 20 datafile summary objects.
Deleting 20 datafile summary objects.
2024-09-30 17:40:02,525 INFO clean_and_reparse.py::_delete_summaries:L90 :  Successfully deleted datafile summary objects.
Successfully deleted datafile summary objects.
2024-09-30 17:40:03,306 INFO clean_and_reparse.py::_delete_errors:L158 :  Deleting 2154232 parser errors.
Deleting 2154232 parser errors.
2024-09-30 17:40:08,748 INFO clean_and_reparse.py::_delete_errors:L160 :  Successfully deleted parser errors.
Successfully deleted parser errors.
2024-09-30 17:40:08,868 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T1'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T1'>.
2024-09-30 17:40:09,636 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T2'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T2'>.
2024-09-30 17:40:10,765 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T3'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T3'>.
2024-09-30 17:40:11,426 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T4'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T4'>.
2024-09-30 17:40:11,435 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T5'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T5'>.
2024-09-30 17:40:11,446 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T6'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T6'>.
2024-09-30 17:40:11,452 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T7'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tanf.TANF_T7'>.
2024-09-30 17:40:11,512 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 266541 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M1'>.
Deleting 266541 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M1'>.
2024-09-30 17:41:40,659 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 401730 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M2'>.
Deleting 401730 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M2'>.
2024-09-30 17:44:08,643 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 652485 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M3'>.
Deleting 652485 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M3'>.
2024-09-30 17:47:32,931 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 30 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M4'>.
Deleting 30 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M4'>.
2024-09-30 17:47:32,970 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 104 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M5'>.
Deleting 104 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M5'>.
2024-09-30 17:47:33,012 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M6'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M6'>.
2024-09-30 17:47:33,019 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M7'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.ssp.SSP_M7'>.
2024-09-30 17:47:33,028 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T1'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T1'>.
2024-09-30 17:47:33,039 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T2'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T2'>.
2024-09-30 17:47:33,050 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T3'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T3'>.
2024-09-30 17:47:33,059 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T4'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T4'>.
2024-09-30 17:47:33,065 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T5'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T5'>.
2024-09-30 17:47:33,073 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 6 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T6'>.
Deleting 6 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T6'>.
2024-09-30 17:47:33,152 INFO clean_and_reparse.py::_delete_records:L129 :  Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T7'>.
Deleting 0 records of type: <class 'tdpservice.search_indexes.models.tribal.Tribal_TANF_T7'>.
2024-09-30 17:47:33,156 INFO clean_and_reparse.py::_calculate_timeout:L255 :  Setting timeout for the reparse event to be 2:06:02.973328 seconds from meta model creation date.
Setting timeout for the reparse event to be 2:06:02.973328 seconds from meta model creation date.
2024-09-30 17:47:33,159 INFO clean_and_reparse.py::handle:L367 :  Deleted a total of 1320896 records accross 20 files.
Deleted a total of 1320896 records accross 20 files.
2024-09-30 17:47:33,159 INFO clean_and_reparse.py::handle:L370 :  Deleting and re-parsing 20 files
Deleting and re-parsing 20 files
2024-09-30 17:47:33,379 INFO clean_and_reparse.py::handle:L380 :  Done. All tasks have been queued to parse the selected datafiles.
Done. All tasks have been queued to parse the selected datafiles.

evidence of reparse model

elipe17 added 3 commits September 4, 2024 13:19

- Updated method to batch delete records in elastic to avoid pulling …

cc536b5

…whole queryset into memory

- Instantiate document once

a27086c

- Updated clean and reparse to not load entire queryset into memory

6f2105f

- Added extra console logging - changed log level on frequent exception

elipe17 added bug backend dev raft review This issue is ready for raft review labels Sep 5, 2024

elipe17 requested review from andrew-jameson, raftmsohani and jtimpe September 5, 2024 14:34

elipe17 self-assigned this Sep 5, 2024

elipe17 commented Sep 5, 2024

View reviewed changes

tdrs-backend/tdpservice/parsers/validators/category3.py Show resolved Hide resolved

Merge branch 'develop' into 3170-reparse-memory-mgmt

223bc40

andrew-jameson approved these changes Sep 6, 2024

View reviewed changes

raftmsohani reviewed Sep 9, 2024

View reviewed changes

elipe17 and others added 6 commits September 10, 2024 15:25

- Update names of functions

a8093e5

- Add tests

- Update sequential function to return boolean to allow testing

f34b014

- Update all files done logic - Add new test for sequential execution

Merge branch 'develop' into 3170-reparse-memory-mgmt

ccd1816

- new indices

d96f455

Merge branch '3170-reparse-memory-mgmt' of https://github.com/raft-te…

b1f71dd

…ch/TANF-app into 3170-reparse-memory-mgmt

- linting

5621f7f

jtimpe approved these changes Sep 11, 2024

View reviewed changes

elipe17 added 7 commits September 11, 2024 10:49

- Added tests for most exception paths

9a03ff3

-linting

778c827

- Adding refresh

4d2ee69

- testing not duplicating container

d8a3b44

- revert test change

fa77bac

- change order

- remove param

01dc2c3

- Add missing tests

876260c

Merge branch 'develop' into 3170-reparse-memory-mgmt

8187285

ADPennington added the Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI label Sep 24, 2024

ADPennington added Blocked Label for Pull Requests that are currently blocked by a dependency and removed Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI labels Sep 24, 2024

Merge branch 'develop' into 3170-reparse-memory-mgmt

e752589

ADPennington added Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI and removed Blocked Label for Pull Requests that are currently blocked by a dependency labels Sep 26, 2024

elipe17 added 2 commits September 26, 2024 11:01

- update terraform for testing

23924fb

Merge branch '3170-reparse-memory-mgmt' of https://github.com/raft-te…

93bd544

…ch/TANF-app into 3170-reparse-memory-mgmt

ADPennington added Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI and removed Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI labels Sep 26, 2024

ADPennington reviewed Sep 26, 2024

View reviewed changes

terraform/dev/main.tf Show resolved Hide resolved

elipe17 and others added 2 commits September 30, 2024 09:35

- Update terraform based on cloud.gov response

0543f29

- Give Prod at least 500GB

Merge branch 'develop' into 3170-reparse-memory-mgmt

f0a6c55

ADPennington removed the Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI label Sep 30, 2024

Merge branch 'develop' into 3170-reparse-memory-mgmt

a981311

ADPennington added the Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI label Sep 30, 2024

ADPennington approved these changes Sep 30, 2024

View reviewed changes

ADPennington added Ready to Merge and removed QASP Review Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI labels Sep 30, 2024

elipe17 merged commit c5f87eb into develop Sep 30, 2024
23 checks passed

elipe17 deleted the 3170-reparse-memory-mgmt branch September 30, 2024 19:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reparse Memory Management #3172

Reparse Memory Management #3172

elipe17 commented Sep 5, 2024 •

edited by ADPennington

Loading

codecov bot commented Sep 5, 2024 •

edited

Loading

andrew-jameson left a comment

raftmsohani left a comment

ADPennington commented Sep 24, 2024

ADPennington commented Sep 24, 2024 •

edited

Loading

ADPennington commented Sep 26, 2024

ADPennington commented Sep 26, 2024

ADPennington commented Sep 27, 2024

ADPennington commented Sep 30, 2024 •

edited

Loading

elipe17 commented Sep 30, 2024

ADPennington left a comment

Reparse Memory Management #3172

Reparse Memory Management #3172

Conversation

elipe17 commented Sep 5, 2024 • edited by ADPennington Loading

Summary of Changes

How to Test

Deliverables

codecov bot commented Sep 5, 2024 • edited Loading

Codecov Report

andrew-jameson left a comment

Choose a reason for hiding this comment

raftmsohani left a comment

Choose a reason for hiding this comment

ADPennington commented Sep 24, 2024

ADPennington commented Sep 24, 2024 • edited Loading

ADPennington commented Sep 26, 2024

ADPennington commented Sep 26, 2024

ADPennington commented Sep 27, 2024

ADPennington commented Sep 30, 2024 • edited Loading

elipe17 commented Sep 30, 2024

ADPennington left a comment

Choose a reason for hiding this comment

elipe17 commented Sep 5, 2024 •

edited by ADPennington

Loading

codecov bot commented Sep 5, 2024 •

edited

Loading

ADPennington commented Sep 24, 2024 •

edited

Loading

ADPennington commented Sep 30, 2024 •

edited

Loading