Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.9] Backport #1302, #1355, #1432, #1435, #1445, #1403, #1430, #1441 to 2.9 #1469

Merged
merged 12 commits into from
Mar 14, 2024
Merged

[Backport 2.9] Backport #1302, #1355, #1432, #1435, #1445, #1403, #1430, #1441 to 2.9 #1469

merged 12 commits into from
Mar 14, 2024

Conversation

eirsep
Copy link
Member

@eirsep eirsep commented Mar 13, 2024 •
edited
Loading

@eirsep eirsep changed the title Backport 2.9 optimize doc-level monitor execution workflow for datastreams (#1302) [Backport 2.9] Backport #1302 and #1355 to 2.9 Mar 13, 2024
@eirsep eirsep changed the title [Backport 2.9] Backport #1302 and #1355 to 2.9 [Backport 2.9] Backport #1302, #1355, #1432 to 2.9 Mar 13, 2024
sbcd90 and others added 10 commits March 14, 2024 01:00
@sbcd90 @eirsep
optimize doc-level monitor execution workflow for datastreams (opense…
6a41aed 
…arch-project#1302)

* optimize doc-level monitor execution for datastreams

Signed-off-by: Subhobrata Dey <[email protected]>

* add more tests to address comments

Signed-off-by: Subhobrata Dey <[email protected]>

* add integTest for multiple datastreams inside a single index pattern

* add integTest for multiple datastreams inside a single index pattern

Signed-off-by: Subhobrata Dey <[email protected]>

---------

Signed-off-by: Subhobrata Dey <[email protected]>
@goyamegh @eirsep
Bulk index findings and sequentially invoke auto-correlations (opense…
b69bb56 
…arch-project#1355)

* Bulk index findings and sequentially invoke auto-correlations

Signed-off-by: Megha Goyal <[email protected]>

* Bulk index findings in batches of 10000 and make it configurable

Signed-off-by: Megha Goyal <[email protected]>

* Addressing review comments

Signed-off-by: Megha Goyal <[email protected]>

* Add integ tests to test bulk index findings

Signed-off-by: Megha Goyal <[email protected]>

* Fix ktlint formatting

Signed-off-by: Megha Goyal <[email protected]>

---------

Signed-off-by: Megha Goyal <[email protected]>
@opensearch-trigger-bot @eirsep
fix for MapperException[the [enabled] parameter can't be updated for …
c1e2316 
…the object mapping [metadata.source_to_query_index_mapping] (opensearch-project#1432) (opensearch-project#1434)

Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@eirsep
Add jvm aware setting and max num docs settings for batching docs for…
9ece656 
… percolate queries (opensearch-project#1435)

* add jvm aware and max docs settings for batching docs for percolate queries

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix stats logging

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add queryfieldnames field in findings mapping

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
@jowg-amazon @eirsep
clean up doc level queries on dry run (opensearch-project#1430)
e23dd95 
Signed-off-by: Joanne Wang <[email protected]>
@eirsep
optimize to fetch only fields relevant to doc level queries in doc le…
2e16049 
…vel monitor instead of entire _source for each doc (opensearch-project#1441)

* optimize to fetch only fields relevant to doc level queries in doc level monitor

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test for settings check

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix ktlint

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
@eirsep
optimize sequence number calculation and reduce search requests in do…
48eb050 
…c level monitor execution (opensearch-project#1445)

* optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix tests

Signed-off-by: Surya Sashank Nistala <[email protected]>

* optimize check indices and execute to query only write index of aliases and datastreams during monitor creation

Signed-off-by: Surya Sashank Nistala <[email protected]>

* fix test

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add javadoc

Signed-off-by: Surya Sashank Nistala <[email protected]>

* add tests to verify seq_no calculation

Signed-off-by: Surya Sashank Nistala <[email protected]>

---------

Signed-off-by: Surya Sashank Nistala <[email protected]>
@sbcd90 @eirsep
add distributed locking to jobs in alerting (opensearch-project#1403)
78955a6 
Signed-off-by: Subhobrata Dey <[email protected]>
@eirsep
2.9.1 version bump
7d1096e 
Signed-off-by: Surya Sashank Nistala <[email protected]>
@eirsep
fix compilation issues
e535ade 
Signed-off-by: Surya Sashank Nistala <[email protected]>
@eirsep
dummy commit
b5b9e34 
Signed-off-by: Surya Sashank Nistala <[email protected]>
@eirsep eirsep changed the title [Backport 2.9] Backport #1302, #1355, #1432 to 2.9 [Backport 2.9] Backport #1302, #1355, #1432, #1435, #1445, #1403, #1430, #1441 to 2.9 Mar 14, 2024
@eirsep
fix findings index schema version tsts
e74b166 
Signed-off-by: Surya Sashank Nistala <[email protected]>
@codecov Codecov
Copy link

codecov bot commented Mar 14, 2024

Codecov Report

Attention: Patch coverage is 78.65330% with 149 lines in your changes are missing coverage. Please review.

Project coverage is 68.64%. Comparing base (0269dd1) to head (e74b166).
Report is 2 commits behind head on 2.9.

Files Patch % Lines
.../opensearch/alerting/DocumentLevelMonitorRunner.kt 82.89% 31 Missing and 21 partials ⚠️
...n/org/opensearch/alerting/core/lock/LockService.kt 58.73% 45 Missing and 7 partials ⚠️
.../org/opensearch/alerting/MonitorMetadataService.kt 52.63% 9 Missing and 9 partials ⚠️
...opensearch/alerting/util/DocLevelMonitorQueries.kt 77.14% 5 Missing and 3 partials ⚠️
...in/org/opensearch/alerting/MonitorRunnerService.kt 88.00% 4 Missing and 2 partials ⚠️
.../kotlin/org/opensearch/alerting/util/IndexUtils.kt 70.58% 1 Missing and 4 partials ⚠️
...lerting/transport/TransportDeleteWorkflowAction.kt 60.00% 2 Missing ⚠️
.../alerting/transport/TransportIndexMonitorAction.kt 66.66% 0 Missing and 2 partials ⚠️
...lin/org/opensearch/alerting/core/lock/LockModel.kt 96.55% 0 Missing and 2 partials ⚠️
...opensearch/alerting/model/IndexExecutionContext.kt 88.88% 1 Missing ⚠️
... and 1 more
Additional details and impacted files 
@@             Coverage Diff              @@
##                2.9    #1469      +/-   ##
============================================
+ Coverage     68.35%   68.64%   +0.29%     
- Complexity      120      143      +23     
============================================
  Files           160      162       +2     
  Lines         10740    11268     +528     
  Branches       1628     1697      +69     
============================================
+ Hits           7341     7735     +394     
- Misses         2680     2774      +94     
- Partials        719      759      +40

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jowg-amazon
Copy link
Collaborator

Are we backporting #1139 to 2.9?

@eirsep eirsep merged commit a82c4f1 into opensearch-project:2.9 Mar 14, 2024
10 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sbcd90 sbcd90 sbcd90 approved these changes

@praveensameneni praveensameneni praveensameneni approved these changes

@jowg-amazon jowg-amazon jowg-amazon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@eirsep @jowg-amazon @sbcd90 @praveensameneni @goyamegh