Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make RemoteStoreReplicationSource#getSegmentFiles asynchronous #10725

Merged
merged 6 commits into from
Oct 20, 2023

Conversation

mch2
Copy link
Member

@mch2 mch2 commented Oct 19, 2023

Description

This change makes Make RemoteStoreReplicationSource#getSegmentFiles asynchronous.
It also ensures that SegmentReplicationTargets are properly closed during cancellation with remote store enabled if the cancel is issued during a file download.

Related Issues

Resolves #8089
Resolves #10619

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions

This comment was marked as outdated.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 19, 2023

Compatibility status:

Checks if related components are compatible with change cd06a0d

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git]

@github-actions github-actions bot added distributed framework enhancement Enhancement or improvement to existing feature or request Indexing:Replication Issues and PRs related to core replication framework eg segrep labels Oct 19, 2023
@mch2 mch2 marked this pull request as ready for review October 19, 2023 04:59
@mch2 mch2 requested a review from VachaShah as a code owner October 19, 2023 04:59
@github-actions

This comment was marked as outdated.

@mch2 mch2 force-pushed the parallel-download-async-v2 branch from 5f4e916 to 02172cc Compare October 19, 2023 16:11
@github-actions

This comment was marked as outdated.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

andrross and others added 6 commits October 19, 2023 12:47
Also make the remote store download process cancellable in case the
replication event is canceled.

Signed-off-by: Andrew Ross <[email protected]>
… metadata and segment fetch steps.

Signed-off-by: Marc Handalian <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
@mch2 mch2 force-pushed the parallel-download-async-v2 branch from f5519ec to cd06a0d Compare October 19, 2023 19:47
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@mch2
Copy link
Member Author

mch2 commented Oct 19, 2023

Gradle Check (Jenkins) Run Completed with:

#6090

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@codecov
Copy link

codecov bot commented Oct 19, 2023

Codecov Report

Merging #10725 (cd06a0d) into main (69f6f4e) will increase coverage by 0.04%.
Report is 2 commits behind head on main.
The diff coverage is 77.19%.

@@             Coverage Diff              @@
##               main   #10725      +/-   ##
============================================
+ Coverage     71.11%   71.15%   +0.04%     
- Complexity    58530    58535       +5     
============================================
  Files          4854     4854              
  Lines        276049   276066      +17     
  Branches      40168    40170       +2     
============================================
+ Hits         196311   196448     +137     
+ Misses        63385    63214     -171     
- Partials      16353    16404      +51     
Files Coverage Δ
.../indices/replication/common/ReplicationTarget.java 80.00% <100.00%> (+1.05%) ⬆️
...ain/java/org/opensearch/search/DocValueFormat.java 59.17% <0.00%> (-5.33%) ⬇️
...nsearch/index/store/RemoteStoreFileDownloader.java 92.15% <84.00%> (-0.87%) ⬇️
...ices/replication/RemoteStoreReplicationSource.java 85.24% <75.00%> (-5.83%) ⬇️

... and 437 files with indirect coverage changes

Copy link
Member

@andrross andrross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me! But I authored one of the commits in this PR so I'd love to have approval from another maintainer.

@github-actions github-actions bot added the bug Something isn't working label Oct 19, 2023
@mch2 mch2 merged commit 41a12e2 into opensearch-project:main Oct 20, 2023
24 checks passed
@mch2 mch2 added the backport 2.x Backport to 2.x branch label Oct 20, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 20, 2023
* Make RemoteStoreReplicationSource#getSegmentFiles asynchronous

Also make the remote store download process cancellable in case the
replication event is canceled.

Signed-off-by: Andrew Ross <[email protected]>

* Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps.

Signed-off-by: Marc Handalian <[email protected]>

* Wrap metadata fetch in cancellableThreads.executeIO

Signed-off-by: Marc Handalian <[email protected]>

* self review

Signed-off-by: Marc Handalian <[email protected]>

* spotless

Signed-off-by: Marc Handalian <[email protected]>

* Add missing node settings when bootstrapping nodes in tests.

Signed-off-by: Marc Handalian <[email protected]>

---------

Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
(cherry picked from commit 41a12e2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
mch2 pushed a commit that referenced this pull request Oct 20, 2023
… (#10771)

* Make RemoteStoreReplicationSource#getSegmentFiles asynchronous

Also make the remote store download process cancellable in case the
replication event is canceled.



* Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps.



* Wrap metadata fetch in cancellableThreads.executeIO



* self review



* spotless



* Add missing node settings when bootstrapping nodes in tests.



---------




(cherry picked from commit 41a12e2)

Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Andrew Ross <[email protected]>
austintlee pushed a commit to austintlee/OpenSearch that referenced this pull request Oct 23, 2023
…earch-project#10725)

* Make RemoteStoreReplicationSource#getSegmentFiles asynchronous

Also make the remote store download process cancellable in case the
replication event is canceled.

Signed-off-by: Andrew Ross <[email protected]>

* Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps.

Signed-off-by: Marc Handalian <[email protected]>

* Wrap metadata fetch in cancellableThreads.executeIO

Signed-off-by: Marc Handalian <[email protected]>

* self review

Signed-off-by: Marc Handalian <[email protected]>

* spotless

Signed-off-by: Marc Handalian <[email protected]>

* Add missing node settings when bootstrapping nodes in tests.

Signed-off-by: Marc Handalian <[email protected]>

---------

Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…earch-project#10725)

* Make RemoteStoreReplicationSource#getSegmentFiles asynchronous

Also make the remote store download process cancellable in case the
replication event is canceled.

Signed-off-by: Andrew Ross <[email protected]>

* Add ITs ensuring segRep targets are cleaned up on cancellation during metadata and segment fetch steps.

Signed-off-by: Marc Handalian <[email protected]>

* Wrap metadata fetch in cancellableThreads.executeIO

Signed-off-by: Marc Handalian <[email protected]>

* self review

Signed-off-by: Marc Handalian <[email protected]>

* spotless

Signed-off-by: Marc Handalian <[email protected]>

* Add missing node settings when bootstrapping nodes in tests.

Signed-off-by: Marc Handalian <[email protected]>

---------

Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
Co-authored-by: Andrew Ross <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch bug Something isn't working distributed framework enhancement Enhancement or improvement to existing feature or request Indexing:Replication Issues and PRs related to core replication framework eg segrep skip-changelog
Projects
None yet
3 participants