-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFS refactoring into separate component programs + new work coordination mechanisms #739
Merged
gregschohn
merged 31 commits into
opensearch-project:main
from
gregschohn:BuildReindexFromSnapshotViaNewDocumentProject
Jun 21, 2024
Merged
Changes from all commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
dd4b1d1
POC/unit test code to use painless scripts to do lease acquisition.
gregschohn f29c786
Drastically simplify taking a snapshot, assuming that there will be n…
gregschohn 6a457ff
Merge branch 'main' into RfsCoordinationWork
gregschohn dd360ac
Bugfix in update logic of the CMS unit test POC.
gregschohn 3d5a5a2
Checkpoint on the coordination POC test harness. No guarantees on th…
gregschohn dc24af9
WIP: In the process of refactoring the unit test POC into a more prod…
gregschohn 7746f5f
Bugfixes and test improvement to show doling out work in the happy ca…
gregschohn f805701
Make some improvements to mitigate contention for acquiring work.
gregschohn b7923f6
Start converting the document migration phase to use the new primitives.
gregschohn 611140f
Starting to change how process management (death) works in cases wher…
gregschohn 4949388
Convert the remaining Runner classes to plain old java functions, w/o…
gregschohn 3683c44
Remove "CMS" implementations as they're being replaced by the new Wor…
gregschohn e33c187
Merge branch 'main' into RfsCoordinationWork.
gregschohn e075d83
Merge branch 'main' into RfsCoordinationWork
gregschohn 8703c20
WIP, doesn't compile. A lot of in-progress changes to support a full…
gregschohn 0275dd5
A lot of bugfixes to get the FullTest moving further along
gregschohn 8733c01
Update the DocumentsReader API to not take indexname/shard and instea…
gregschohn a9b6709
Extract some of the build logic to build a docker image for RFS and m…
gregschohn d2514dc
Update the DocumentsReader API to not take indexname/shard and instea…
gregschohn 10866ff
WIP - start thinking about overall termination of RFS.
gregschohn 69d7d58
Fix some lingering reporting issues to try to get better certainty in…
gregschohn 19cf133
Make the FullTest more of a realistic functional test...
gregschohn 0db6f63
Merge branch 'BuildReindexFromSnapshotViaNewDocumentProject' into Rfs…
gregschohn ee6fbc8
Merge branch 'main' into RfsCoordinationWork
gregschohn 7905f4c
Fix documentation linting errors
gregschohn 9f075c7
Minor fixes to build the rfs_source image and to get the FullTest to …
gregschohn 6f50219
Path updates to move "RFS" references to "DocumentsFromSnapshotMigrat…
gregschohn 9aead15
Minor bugfixes and refactoring improvements as per PR feedback.
gregschohn 787fd26
Merge branch 'main' into BuildReindexFromSnapshotViaNewDocumentProject
gregschohn 8ba51fc
Merge branch 'BuildReindexFromSnapshotViaNewDocumentProject' into Coo…
gregschohn 8089d8b
Test bugfix for when not EVERY worker throws an exception.
gregschohn File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This slows down the slow tests even more. Why can't we use test containers and populate the documents during the test - this decoupling of test setup and test validation will slow us down over time sustainably IMO.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now, I think that the time is roughly the same. It takes about 2 minutes to load the test data that I'm using. The hope with the prebuilt image is that if I'm running repeatedly, I don't need to keep regenerating my source data. I'll work on making the prebuilt stuff cacheable (it isn't now, which admittedly sucks), but the additional tax is just a couple extra docker operations (and an extra startup cycle of ES).