Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Shallow Snapshot flows to support remote path type & hash algo #12988

Merged
merged 7 commits into from
Apr 5, 2024

Conversation

ashking94
Copy link
Member

@ashking94 ashking94 commented Mar 31, 2024

Description

This PR is a sub task of feature request #12567 and being tracked in meta issue - #12589.

In this PR, we have wired the RemoteStorePathStrategy to Shallow Snapshots. We have written new tests and modified the existing tests as required. We handle following senarios -

  1. Snapshots and restores
  2. Snapshot clone
  3. Lock file deletion during snapshot deletion
  4. Remote store cleanup after every snapshot deletion

Related Issues

Resolves #12987

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Performance Storage:Resiliency Issues and PRs related to the storage resiliency v2.14.0 labels Mar 31, 2024
Copy link
Contributor

github-actions bot commented Mar 31, 2024

Compatibility status:

Checks if related components are compatible with change 9f15bbf

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

Copy link
Contributor

✅ Gradle check result for 81e3a03: SUCCESS

Copy link

codecov bot commented Mar 31, 2024

Codecov Report

Attention: Patch coverage is 66.15385% with 44 lines in your changes are missing coverage. Please review.

Project coverage is 71.54%. Comparing base (b15cb0c) to head (c2bb40e).
Report is 134 commits behind head on main.

Files Patch % Lines
...blobstore/RemoteStoreShardShallowCopySnapshot.java 57.14% 12 Missing and 12 partials ⚠️
.../org/opensearch/index/remote/RemoteStoreEnums.java 79.41% 4 Missing and 3 partials ⚠️
...ensearch/index/remote/RemoteStorePathStrategy.java 50.00% 2 Missing and 2 partials ⚠️
.../main/java/org/opensearch/index/IndexSettings.java 57.14% 0 Missing and 3 partials ⚠️
...ch/repositories/blobstore/BlobStoreRepository.java 50.00% 3 Missing ⚠️
...h/cluster/metadata/MetadataCreateIndexService.java 92.30% 0 Missing and 1 partial ⚠️
.../index/remote/RemoteStorePathStrategyResolver.java 80.00% 0 Missing and 1 partial ⚠️
...java/org/opensearch/index/shard/StoreRecovery.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #12988      +/-   ##
============================================
+ Coverage     71.42%   71.54%   +0.12%     
- Complexity    59978    60495     +517     
============================================
  Files          4985     5026      +41     
  Lines        282275   284522    +2247     
  Branches      40946    41213     +267     
============================================
+ Hits         201603   203562    +1959     
- Misses        63999    64122     +123     
- Partials      16673    16838     +165     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

github-actions bot commented Apr 3, 2024

❕ Gradle check result for 2ef85af: UNSTABLE

  • TEST FAILURES:
      3 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@ashking94
Copy link
Member Author

❕ Gradle check result for 2ef85af: UNSTABLE

  • TEST FAILURES:
      3 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Flaky tests - #10006, #11547 & #5426

Copy link
Contributor

github-actions bot commented Apr 3, 2024

✅ Gradle check result for 6b53314: SUCCESS

Copy link
Contributor

github-actions bot commented Apr 4, 2024

❕ Gradle check result for 86a77a3: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testIndexReopenClose

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Ashish Singh <[email protected]>
@ashking94
Copy link
Member Author

org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testIndexReopenClose

Flaky test - #13084

Copy link
Contributor

github-actions bot commented Apr 4, 2024

✅ Gradle check result for dcccd05: SUCCESS

Copy link
Contributor

@harishbhakuni harishbhakuni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes lgtm

Signed-off-by: Ashish Singh <[email protected]>
@ashking94 ashking94 requested a review from gbbafna April 5, 2024 07:35
Signed-off-by: Ashish Singh <[email protected]>
Copy link
Contributor

github-actions bot commented Apr 5, 2024

❕ Gradle check result for 9f15bbf: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimaryMultipleReplicaShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link
Contributor

github-actions bot commented Apr 5, 2024

❌ Gradle check result for c2bb40e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@gbbafna
Copy link
Collaborator

gbbafna commented Apr 5, 2024

org.opensearch.discovery.ClusterDisruptionIT.testAckedIndexing failed . Retrying gradle build .

Copy link
Contributor

github-actions bot commented Apr 5, 2024

❕ Gradle check result for c2bb40e: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@ashking94
Copy link
Member Author

testConcurrentDecommissionAction

Flaky test - #12197

@gbbafna gbbafna merged commit e713175 into opensearch-project:main Apr 5, 2024
28 of 31 checks passed
ashking94 added a commit to ashking94/OpenSearch that referenced this pull request Apr 23, 2024
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
@ashking94 ashking94 added the backport 2.x Backport to 2.x branch label Apr 26, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Apr 26, 2024
…#12988)

Signed-off-by: Ashish Singh <[email protected]>
(cherry picked from commit e713175)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
sachinpkale pushed a commit that referenced this pull request Apr 27, 2024
…#12988) (#13410)

(cherry picked from commit e713175)

Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch enhancement Enhancement or improvement to existing feature or request skip-changelog Storage:Performance Storage:Resiliency Issues and PRs related to the storage resiliency v2.14.0
Projects
Status: ✅ Done
Status: Planned work items
Development

Successfully merging this pull request may close these issues.

[Remote Store] Update shallow snapshot flows to support new path types and hash algorithm
3 participants