Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/main' into NodeStatsOptimisation…
Browse files Browse the repository at this point in the history
…-ClusterStats-NodeIndicesStats3
  • Loading branch information
Pranshu-S committed Jul 10, 2024
2 parents a267aef + bda8393 commit 517baa5
Show file tree
Hide file tree
Showing 115 changed files with 2,263 additions and 547 deletions.
13 changes: 9 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,24 +13,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add batching supported processor base type AbstractBatchingProcessor ([#14554](https://github.com/opensearch-project/OpenSearch/pull/14554))
- Fix race condition while parsing derived fields from search definition ([14445](https://github.com/opensearch-project/OpenSearch/pull/14445))
- Add allowlist setting for ingest-common and search-pipeline-common processors ([#14439](https://github.com/opensearch-project/OpenSearch/issues/14439))
- Create SystemIndexRegistry with helper method matchesSystemIndex ([#14415](https://github.com/opensearch-project/OpenSearch/pull/14415))

### Dependencies
- Bump `org.gradle.test-retry` from 1.5.8 to 1.5.9 ([#13442](https://github.com/opensearch-project/OpenSearch/pull/13442))
- Update to Apache Lucene 9.11.0 ([#14042](https://github.com/opensearch-project/OpenSearch/pull/14042))
- Bump `netty` from 4.1.110.Final to 4.1.111.Final ([#14356](https://github.com/opensearch-project/OpenSearch/pull/14356))
- Bump `org.wiremock:wiremock-standalone` from 3.3.1 to 3.6.0 ([#14361](https://github.com/opensearch-project/OpenSearch/pull/14361))
- Bump `reactor` from 3.5.17 to 3.5.18 ([#14395](https://github.com/opensearch-project/OpenSearch/pull/14395))
- Bump `reactor-netty` from 1.1.19 to 1.1.20 ([#14395](https://github.com/opensearch-project/OpenSearch/pull/14395))
- Bump `reactor` from 3.5.17 to 3.5.19 ([#14395](https://github.com/opensearch-project/OpenSearch/pull/14395), [#14697](https://github.com/opensearch-project/OpenSearch/pull/14697))
- Bump `reactor-netty` from 1.1.19 to 1.1.21 ([#14395](https://github.com/opensearch-project/OpenSearch/pull/14395), [#14697](https://github.com/opensearch-project/OpenSearch/pull/14697))
- Bump `commons-net:commons-net` from 3.10.0 to 3.11.1 ([#14396](https://github.com/opensearch-project/OpenSearch/pull/14396))
- Bump `com.nimbusds:nimbus-jose-jwt` from 9.37.3 to 9.40 ([#14398](https://github.com/opensearch-project/OpenSearch/pull/14398))
- Bump `org.apache.commons:commons-configuration2` from 2.10.1 to 2.11.0 ([#14399](https://github.com/opensearch-project/OpenSearch/pull/14399))
- Bump `com.gradle.develocity` from 3.17.4 to 3.17.5 ([#14397](https://github.com/opensearch-project/OpenSearch/pull/14397))
- Bump `opentelemetry` from 1.36.0 to 1.39.0 ([#14457](https://github.com/opensearch-project/OpenSearch/pull/14457))
- Bump `azure-identity` from 1.11.4 to 1.13.0, Bump `msal4j` from 1.14.3 to 1.15.1, Bump `msal4j-persistence-extension` from 1.2.0 to 1.3.0 ([#14506](https://github.com/opensearch-project/OpenSearch/pull/14506))
- Bump `opentelemetry` from 1.36.0 to 1.40.0 ([#14457](https://github.com/opensearch-project/OpenSearch/pull/14457), [#14674](https://github.com/opensearch-project/OpenSearch/pull/14674))
- Bump `opentelemetry-semconv` from 1.25.0-alpha to 1.26.0-alpha ([#14674](https://github.com/opensearch-project/OpenSearch/pull/14674))
- Bump `azure-identity` from 1.11.4 to 1.13.0, Bump `msal4j` from 1.14.3 to 1.15.1, Bump `msal4j-persistence-extension` from 1.2.0 to 1.3.0 ([#14506](https://github.com/opensearch-project/OpenSearch/pull/14673))
- Bump `com.azure:azure-storage-common` from 12.21.2 to 12.25.1 ([#14517](https://github.com/opensearch-project/OpenSearch/pull/14517))
- Bump `com.microsoft.azure:msal4j` from 1.15.1 to 1.16.0 ([#14610](https://github.com/opensearch-project/OpenSearch/pull/14610))
- Bump `com.github.spullara.mustache.java:compiler` from 0.9.13 to 0.9.14 ([#14672](https://github.com/opensearch-project/OpenSearch/pull/14672))
- Bump `net.minidev:accessors-smart` from 2.5.0 to 2.5.1 ([#14673](https://github.com/opensearch-project/OpenSearch/pull/14673))
- Bump `jackson` from 2.17.1 to 2.17.2 ([#14687](https://github.com/opensearch-project/OpenSearch/pull/14687))

### Changed
- [Tiered Caching] Move query recomputation logic outside write lock ([#14187](https://github.com/opensearch-project/OpenSearch/pull/14187))
Expand All @@ -49,6 +52,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Fix bug in SBP cancellation logic ([#13259](https://github.com/opensearch-project/OpenSearch/pull/13474))
- Fix handling of Short and Byte data types in ScriptProcessor ingest pipeline ([#14379](https://github.com/opensearch-project/OpenSearch/issues/14379))
- Switch to iterative version of WKT format parser ([#14086](https://github.com/opensearch-project/OpenSearch/pull/14086))
- Fix match_phrase_prefix_query not working on text field with multiple values and index_prefixes ([#10959](https://github.com/opensearch-project/OpenSearch/pull/10959))
- Fix the computed max shards of cluster to avoid int overflow ([#14155](https://github.com/opensearch-project/OpenSearch/pull/14155))
- Fixed rest-high-level client searchTemplate & mtermVectors endpoints to have a leading slash ([#14465](https://github.com/opensearch-project/OpenSearch/pull/14465))
- Write shard level metadata blob when snapshotting searchable snapshot indexes ([#13190](https://github.com/opensearch-project/OpenSearch/pull/13190))
Expand All @@ -59,6 +63,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Fix file cache initialization ([#14004](https://github.com/opensearch-project/OpenSearch/pull/14004))
- Handle NPE in GetResult if "found" field is missing ([#14552](https://github.com/opensearch-project/OpenSearch/pull/14552))
- Refactoring FilterPath.parse by using an iterative approach ([#14200](https://github.com/opensearch-project/OpenSearch/pull/14200))
- Refactoring Grok.validatePatternBank by using an iterative approach ([#14206](https://github.com/opensearch-project/OpenSearch/pull/14206))

### Security

Expand Down
12 changes: 6 additions & 6 deletions buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ bundled_jdk = 21.0.3+9
# optional dependencies
spatial4j = 0.7
jts = 1.15.0
jackson = 2.17.1
jackson_databind = 2.17.1
jackson = 2.17.2
jackson_databind = 2.17.2
snakeyaml = 2.1
icu4j = 70.1
supercsv = 2.4.0
Expand All @@ -33,8 +33,8 @@ netty = 4.1.111.Final
joda = 2.12.7

# project reactor
reactor_netty = 1.1.20
reactor = 3.5.18
reactor_netty = 1.1.21
reactor = 3.5.19

# client dependencies
httpclient5 = 5.2.1
Expand Down Expand Up @@ -74,5 +74,5 @@ jzlib = 1.1.3
resteasy = 6.2.4.Final

# opentelemetry dependencies
opentelemetry = 1.39.0
opentelemetrysemconv = 1.25.0-alpha
opentelemetry = 1.40.0
opentelemetrysemconv = 1.26.0-alpha
1 change: 0 additions & 1 deletion client/sniffer/licenses/jackson-core-2.17.1.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions client/sniffer/licenses/jackson-core-2.17.2.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
969a35cb35c86512acbadcdbbbfb044c877db814

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
147b7b9412ffff24339f8aba080b292448e08698

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e6deb029e5901e027c129341fac39e515066b68c
1 change: 0 additions & 1 deletion libs/core/licenses/jackson-core-2.17.1.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions libs/core/licenses/jackson-core-2.17.2.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
969a35cb35c86512acbadcdbbbfb044c877db814
122 changes: 88 additions & 34 deletions libs/grok/src/main/java/org/opensearch/grok/Grok.java
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,18 @@
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Deque;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Stack;
import java.util.Set;
import java.util.function.Consumer;

import org.jcodings.specific.UTF8Encoding;
Expand Down Expand Up @@ -86,6 +90,7 @@ public final class Grok {
UTF8Encoding.INSTANCE,
Syntax.DEFAULT
);
private static final int MAX_PATTERN_DEPTH_SIZE = 500;

private static final int MAX_TO_REGEX_ITERATIONS = 100_000; // sanity limit

Expand Down Expand Up @@ -128,7 +133,7 @@ private Grok(
expressionBytes.length,
Option.DEFAULT,
UTF8Encoding.INSTANCE,
message -> logCallBack.accept(message)
logCallBack::accept
);

List<GrokCaptureConfig> captureConfig = new ArrayList<>();
Expand All @@ -144,7 +149,7 @@ private Grok(
*/
private void validatePatternBank() {
for (String patternName : patternBank.keySet()) {
validatePatternBank(patternName, new Stack<>());
validatePatternBank(patternName);
}
}

Expand All @@ -156,33 +161,84 @@ private void validatePatternBank() {
* a reference to another named pattern. This method will navigate to all these named patterns and
* check for a circular reference.
*/
private void validatePatternBank(String patternName, Stack<String> path) {
String pattern = patternBank.get(patternName);
boolean isSelfReference = pattern.contains("%{" + patternName + "}") || pattern.contains("%{" + patternName + ":");
if (isSelfReference) {
throwExceptionForCircularReference(patternName, pattern);
} else if (path.contains(patternName)) {
// current pattern name is already in the path, fetch its predecessor
String prevPatternName = path.pop();
String prevPattern = patternBank.get(prevPatternName);
throwExceptionForCircularReference(prevPatternName, prevPattern, patternName, path);
}
path.push(patternName);
for (int i = pattern.indexOf("%{"); i != -1; i = pattern.indexOf("%{", i + 1)) {
int begin = i + 2;
int syntaxEndIndex = pattern.indexOf('}', begin);
if (syntaxEndIndex == -1) {
throw new IllegalArgumentException("Malformed pattern [" + patternName + "][" + pattern + "]");
private void validatePatternBank(String initialPatternName) {
Deque<Frame> stack = new ArrayDeque<>();
Set<String> visitedPatterns = new HashSet<>();
Map<String, List<String>> pathMap = new HashMap<>();

List<String> initialPath = new ArrayList<>();
initialPath.add(initialPatternName);
pathMap.put(initialPatternName, initialPath);
stack.push(new Frame(initialPatternName, initialPath, 0));

while (!stack.isEmpty()) {
Frame frame = stack.peek();
String patternName = frame.patternName;
List<String> path = frame.path;
int startIndex = frame.startIndex;
String pattern = patternBank.get(patternName);

if (visitedPatterns.contains(patternName)) {
stack.pop();
continue;
}

visitedPatterns.add(patternName);
boolean foundDependency = false;

for (int i = startIndex; i < pattern.length(); i++) {
if (pattern.startsWith("%{", i)) {
int begin = i + 2;
int syntaxEndIndex = pattern.indexOf('}', begin);
if (syntaxEndIndex == -1) {
throw new IllegalArgumentException("Malformed pattern [" + patternName + "][" + pattern + "]");
}

int semanticNameIndex = pattern.indexOf(':', begin);
int end = semanticNameIndex == -1 ? syntaxEndIndex : Math.min(syntaxEndIndex, semanticNameIndex);

String dependsOnPattern = pattern.substring(begin, end);

if (dependsOnPattern.equals(patternName)) {
throwExceptionForCircularReference(patternName, pattern);
}

if (pathMap.containsKey(dependsOnPattern)) {
throwExceptionForCircularReference(patternName, pattern, dependsOnPattern, path.subList(0, path.size() - 1));
}

List<String> newPath = new ArrayList<>(path);
newPath.add(dependsOnPattern);
pathMap.put(dependsOnPattern, newPath);

stack.push(new Frame(dependsOnPattern, newPath, 0));
frame.startIndex = i + 1;
foundDependency = true;
break;
}
}
int semanticNameIndex = pattern.indexOf(':', begin);
int end = syntaxEndIndex;
if (semanticNameIndex != -1) {
end = Math.min(syntaxEndIndex, semanticNameIndex);

if (!foundDependency) {
pathMap.remove(patternName);
stack.pop();
}

if (stack.size() > MAX_PATTERN_DEPTH_SIZE) {
throw new IllegalArgumentException("Pattern references exceeded maximum depth of " + MAX_PATTERN_DEPTH_SIZE);
}
String dependsOnPattern = pattern.substring(begin, end);
validatePatternBank(dependsOnPattern, path);
}
path.pop();
}

private static class Frame {
String patternName;
List<String> path;
int startIndex;

Frame(String patternName, List<String> path, int startIndex) {
this.patternName = patternName;
this.path = path;
this.startIndex = startIndex;
}
}

private static void throwExceptionForCircularReference(String patternName, String pattern) {
Expand All @@ -192,13 +248,13 @@ private static void throwExceptionForCircularReference(String patternName, Strin
private static void throwExceptionForCircularReference(
String patternName,
String pattern,
String originPatterName,
Stack<String> path
String originPatternName,
List<String> path
) {
StringBuilder message = new StringBuilder("circular reference in pattern [");
message.append(patternName).append("][").append(pattern).append("]");
if (originPatterName != null) {
message.append(" back to pattern [").append(originPatterName).append("]");
if (originPatternName != null) {
message.append(" back to pattern [").append(originPatternName).append("]");
}
if (path != null && path.size() > 1) {
message.append(" via patterns [").append(String.join("=>", path)).append("]");
Expand All @@ -217,9 +273,7 @@ private String groupMatch(String name, Region region, String pattern) {
int begin = region.getBeg(number);
int end = region.getEnd(number);
return new String(pattern.getBytes(StandardCharsets.UTF_8), begin, end - begin, StandardCharsets.UTF_8);
} catch (StringIndexOutOfBoundsException e) {
return null;
} catch (ValueException e) {
} catch (StringIndexOutOfBoundsException | ValueException e) {
return null;
}
}
Expand Down
10 changes: 10 additions & 0 deletions libs/grok/src/test/java/org/opensearch/grok/GrokTests.java
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,16 @@ public void testCircularReference() {
"circular reference in pattern [NAME5][!!!%{NAME1}!!!] back to pattern [NAME1] " + "via patterns [NAME1=>NAME2=>NAME3=>NAME4]",
e.getMessage()
);

e = expectThrows(IllegalArgumentException.class, () -> {
Map<String, String> bank = new TreeMap<>();
for (int i = 1; i <= 501; i++) {
bank.put("NAME" + i, "!!!%{NAME" + (i + 1) + "}!!!");
}
String pattern = "%{NAME1}";
new Grok(bank, pattern, false, logger::warn);
});
assertEquals("Pattern references exceeded maximum depth of 500", e.getMessage());
}

public void testMalformedPattern() {
Expand Down
1 change: 0 additions & 1 deletion libs/x-content/licenses/jackson-core-2.17.1.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions libs/x-content/licenses/jackson-core-2.17.2.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
969a35cb35c86512acbadcdbbbfb044c877db814

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
57fa7c1b5104bbc4599278d13933a937ee058e68

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
20e956b9b6f67138edd39fab7a506ded19638bcb

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
78d2c73dbec62044d7cf3b544b2e0d24a1a093b0

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
147b7b9412ffff24339f8aba080b292448e08698

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e6deb029e5901e027c129341fac39e515066b68c

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
147b7b9412ffff24339f8aba080b292448e08698

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e6deb029e5901e027c129341fac39e515066b68c

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
147b7b9412ffff24339f8aba080b292448e08698

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
e6deb029e5901e027c129341fac39e515066b68c
Original file line number Diff line number Diff line change
Expand Up @@ -138,17 +138,15 @@ public int getTopNSize() {
* @param size the wanted top N size
*/
public void validateTopNSize(final int size) {
if (size > QueryInsightsSettings.MAX_N_SIZE) {
if (size < 1 || size > QueryInsightsSettings.MAX_N_SIZE) {
throw new IllegalArgumentException(
"Top N size setting for ["
+ metricType
+ "]"
+ " should be smaller than max top N size ["
+ " should be between 1 and "
+ QueryInsightsSettings.MAX_N_SIZE
+ "was ("
+ ", was ("
+ size
+ " > "
+ QueryInsightsSettings.MAX_N_SIZE
+ ")"
);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,10 @@ public void testValidateTopNSize() {
assertThrows(IllegalArgumentException.class, () -> { topQueriesService.validateTopNSize(QueryInsightsSettings.MAX_N_SIZE + 1); });
}

public void testValidateNegativeTopNSize() {
assertThrows(IllegalArgumentException.class, () -> { topQueriesService.validateTopNSize(-1); });
}

public void testGetTopQueriesWhenNotEnabled() {
topQueriesService.setEnabled(false);
assertThrows(IllegalArgumentException.class, () -> { topQueriesService.getTopQueriesRecords(false); });
Expand Down

This file was deleted.

Loading

0 comments on commit 517baa5

Please sign in to comment.