Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated to use jexl3 #15

Open
wants to merge 71 commits into
base: feature/queryMicroservicesAccumulo2.1
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
d2d3075
Updated to use jexl3
jwomeara Jun 13, 2023
175663f
Merge remote-tracking branch 'origin/feature/queryMicroservicesAccumu…
jwomeara Jun 24, 2023
9015e6b
formatted import statements with impsort plugin
jwomeara Jun 27, 2023
1b514fb
Merge branch 'feature/queryMicroservicesAccumulo2.1' into feature/jexl3
jwomeara Jun 28, 2023
eabe3cd
Merge branch 'feature/queryMicroservicesAccumulo2.1' into feature/jexl3
jwomeara Sep 22, 2023
a6fdc4e
Merge remote-tracking branch 'origin/feature/queryMicroservicesAccumu…
jwomeara Nov 2, 2023
87fc476
Correct combining logic for setupTime, createCallTime, loginTime; def…
billoley Nov 17, 2023
9f12a71
Correct combining logic for setupTime, createCallTime, loginTime; def…
billoley Nov 17, 2023
f388ed0
JexlFormattedStringBuildingVisitor.formatMetrics can throw StackOverf…
billoley Nov 20, 2023
cf19fbf
enabled confirm acks and added retry logic.
jwomeara Nov 20, 2023
30eb104
Merge remote-tracking branch 'origin/main' into feature/queryMicroser…
jwomeara Nov 20, 2023
1c5e154
Merge branch 'main' into feature/queryMicroservices
jwomeara Nov 20, 2023
6e48eb7
Write lock until Hazelcast instance set up, prevent deadlock on subse…
billoley Nov 22, 2023
0e09b61
Write lock until Hazelcast instance set up, prevent deadlock on subse…
billoley Nov 22, 2023
517c16f
Closes #13 - Updating DW XMLNS for webservice responses (#14)
keith-ratcliffe Nov 22, 2023
985f0cd
fixed confirm ack configuration, and updated query client response type.
jwomeara Nov 22, 2023
21790a4
Merge remote-tracking branch 'origin/main'
jwomeara Nov 22, 2023
aa8b6a1
Fixed bootstrap configuration, and fixed rest-based query client.
jwomeara Nov 22, 2023
1ec83a3
Added metric client confirm ack configuration
jwomeara Nov 22, 2023
c78feb0
Correct comment in bootstrap.yml
billoley Nov 22, 2023
8db4c53
Merge branch 'main' into feature/queryMicroservices
jwomeara Nov 22, 2023
c52497f
formatting
jwomeara Nov 22, 2023
81eac0d
Merge remote-tracking branch 'origin/main'
jwomeara Nov 22, 2023
5f94dfb
Merge remote-tracking branch 'origin/main' into feature/queryMicroser…
jwomeara Nov 22, 2023
a7c85a3
Release datawave-query-metric api/service 2.1.4
billoley Nov 27, 2023
c6e064d
Set version to 2.1.5-SNAPSHOT
billoley Nov 27, 2023
823b78b
Release datawave-query-metric api/service 3.0.1
billoley Nov 27, 2023
7898de8
Set version to 3.0.2-SNAPSHOT
billoley Nov 27, 2023
a122293
Specify datatype in ShardTableQueryMetricHandler to enable caching in…
billoley Dec 4, 2023
8243e4d
Enable the persisting of a changed query plan in query metric
billoley Dec 4, 2023
d11bf9b
Require that updated Lifecycle is greater than current Lifecycle
billoley Dec 4, 2023
74eb77c
Correlate metric updates by queryId to avoid filling hazelcast operat…
billoley Dec 4, 2023
0d599b1
Change lastWrittenQueryMetrics cache to a Caffeine cache instead of H…
billoley Dec 5, 2023
cd480d7
Limit metric fields retrieved from Accumulo and compared for mutation…
billoley Dec 5, 2023
0e858af
Add retry logic on sending message to rabbit queue, add timer on rest…
billoley Dec 7, 2023
74c8aea
Updated producer confirm acks to perform better with bulk updates.
jwomeara Dec 7, 2023
0da1878
Merge branch 'release/version2.x'
billoley Dec 7, 2023
00d00d2
Merge commit '0da1878'
billoley Dec 7, 2023
e0ac962
Increase cache size and ttl for lastWrittenQueryMetrics cache in testing
billoley Dec 8, 2023
c7a8739
Add timer to record the time it takes in AccumuloMapStore to fetch la…
billoley Dec 8, 2023
92a88a7
Merge branch 'release/version2.x'
billoley Dec 8, 2023
9326be1
bumped query metric api to 3.0.2
jwomeara Dec 8, 2023
2aea1d3
[maven-release-plugin] prepare release 3.0.2
jwomeara Dec 8, 2023
799e7b9
[maven-release-plugin] prepare for next development iteration
jwomeara Dec 8, 2023
d46deba
[maven-release-plugin] prepare release api-3.0.2
jwomeara Dec 8, 2023
ca577ad
[maven-release-plugin] prepare for next development iteration
jwomeara Dec 8, 2023
8619aa6
[maven-release-plugin] prepare release service-3.0.2
jwomeara Dec 8, 2023
dd97acf
[maven-release-plugin] prepare for next development iteration
jwomeara Dec 8, 2023
469386d
Merge remote-tracking branch 'origin/feature/queryMicroservices' into…
jwomeara Dec 9, 2023
95b5eaf
Merge remote-tracking branch 'origin/main' into feature/queryMicroser…
jwomeara Dec 10, 2023
c3b7d65
Merge remote-tracking branch 'origin/feature/queryMicroservices' into…
jwomeara Dec 10, 2023
aab0117
Fix intermittent test failures in CorrelatorTest, QueryMetricOperatio…
billoley Dec 12, 2023
1814886
Merge branch 'release/version2.x'
billoley Dec 12, 2023
cac73bf
Release datawave-query-metric api/service 3.0.3
billoley Dec 12, 2023
d4b1808
Set version to 3.0.4-SNAPSHOT
billoley Dec 12, 2023
24d3917
Release datawave-query-metric api/service 2.1.5
billoley Dec 12, 2023
26c79e5
Set version to 2.1.6-SNAPSHOT
billoley Dec 12, 2023
2324ada
elapsedTime=0 if lastUpdated set to createDate in AccumuloMapStore; e…
billoley Dec 14, 2023
ffc13a3
Move geo response classes to query-metric-api, ensure json/xml/html r…
billoley Dec 14, 2023
e780ccc
Remove errant test configuration setting
billoley Dec 14, 2023
653e170
elapsedTime=0 if lastUpdated set to createDate in AccumuloMapStore; e…
billoley Dec 14, 2023
b2a8fa6
Move geo response classes to query-metric-api, ensure json/xml/html r…
billoley Dec 14, 2023
3df3aef
Merge branch 'release/version2.x'
billoley Dec 14, 2023
48e8572
Release datawave-query-metric api/service 3.0.4
billoley Dec 14, 2023
01c1749
Set version to 3.0.5-SNAPSHOT
billoley Dec 14, 2023
72fa601
Merge remote-tracking branch 'origin/main' into feature/queryMicroser…
jwomeara Dec 15, 2023
62991a1
Fixed concurrent modification exception
jwomeara Dec 15, 2023
d642da5
Merge branch 'main' into feature/queryMicroservices
jwomeara Dec 15, 2023
ad20bfe
Merge branch 'feature/queryMicroservices' into HEAD
jwomeara Dec 16, 2023
7c8da1c
updated with latest from integration
jwomeara Jan 17, 2024
2068e0b
Merge remote-tracking branch 'origin/feature/queryMicroservices' into…
jwomeara Jan 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -617,7 +617,7 @@ public int getFieldNumber(String name) {
@XmlElement
protected String queryId = null;
@XmlElement
protected long setupTime = 0;
protected long setupTime = -1;
@XmlElement
protected String query = null;
@XmlElement
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
import java.text.SimpleDateFormat;
import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;

import javax.xml.bind.annotation.XmlElement;
Expand All @@ -13,6 +15,8 @@
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.text.StringEscapeUtils;

import com.fasterxml.jackson.annotation.JsonIgnore;

import datawave.microservice.querymetric.BaseQueryMetric.PageMetric;
import datawave.webservice.HtmlProvider;
import datawave.webservice.result.BaseResponse;
Expand All @@ -27,10 +31,27 @@ public abstract class BaseQueryMetricListResponse<T extends BaseQueryMetric> ext
protected List<T> result = null;
@XmlElement
protected int numResults = 0;
@XmlElement
protected boolean isGeoQuery = false;
@XmlTransient
private boolean administratorMode = false;
@XmlTransient
private boolean isGeoQuery = false;
private String JQUERY_INCLUDES;
protected String BASE_URL = "/DataWave/Query/Metrics";

public BaseQueryMetricListResponse() {
setHtmlIncludePaths(new HashMap<>());
}

public void setHtmlIncludePaths(Map<String,String> pathMap) {
// @formatter:off
JQUERY_INCLUDES =
"<script type='text/javascript' src='" + pathMap.getOrDefault("jquery", "") + "/jquery.min.js'></script>\n";
// @formatter:on
}

public void setBaseUrl(String baseUrl) {
this.BASE_URL = baseUrl;
}

private static String numToString(long number) {
return (number == -1 || number == 0) ? "" : Long.toString(number);
Expand Down Expand Up @@ -69,21 +90,27 @@ public void setGeoQuery(boolean geoQuery) {
isGeoQuery = geoQuery;
}

@JsonIgnore
@XmlTransient
@Override
public String getTitle() {
return TITLE;
}

@JsonIgnore
@XmlTransient
@Override
public String getPageHeader() {
return getTitle();
}

@JsonIgnore
@XmlTransient
@Override
public String getHeadContent() {
if (isGeoQuery) {
// @formatter:off
return "<script type='text/javascript' src='/jquery.min.js'></script>" +
return JQUERY_INCLUDES +
"<script type='text/javascript'>" +
"$(document).ready(function() {" +
" var currentUrl = window.location.href.replace(/\\/+$/, '');" +
Expand All @@ -97,7 +124,9 @@ public String getHeadContent() {
return EMPTY;
}
}


@JsonIgnore
@XmlTransient
@Override
public String getMainContent() {
StringBuilder builder = new StringBuilder();
Expand Down Expand Up @@ -155,7 +184,7 @@ public String getMainContent() {
builder.append("<td style=\"word-wrap: break-word; min-width:300px !important;\">").append(queryAuths).append("</td>");

builder.append("<td>").append(metric.getHost()).append("</td>");
builder.append("<td>").append(metric.getSetupTime()).append("</td>");
builder.append("<td>").append(numToString(metric.getSetupTime())).append("</td>");
builder.append("<td>").append(numToString(metric.getCreateCallTime())).append("</td>\n");
builder.append("<td>").append(metric.getNumPages()).append("</td>");
builder.append("<td>").append(metric.getNumResults()).append("</td>");
Expand Down Expand Up @@ -188,5 +217,4 @@ public String getMainContent() {

return builder.toString();
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

@XmlRootElement(name = "CacheStats")
@XmlAccessorType(XmlAccessType.NONE)
@XmlType(propOrder = {"serviceStats", "incomingQueryMetrics", "lastWrittenQueryMetrics"})
@XmlType(propOrder = {"serviceStats", "incomingQueryMetrics"})
public class CacheStats implements Serializable {

private static final long serialVersionUID = 1L;
Expand All @@ -34,10 +34,6 @@ public class CacheStats implements Serializable {
@XmlJavaTypeAdapter(StringMapAdapter.class)
private Map<String,String> incomingQueryMetrics = new HashMap<>();

@XmlElement(name = "lastWrittenQueryMetrics")
@XmlJavaTypeAdapter(StringMapAdapter.class)
private Map<String,String> lastWrittenQueryMetrics = new HashMap<>();

public CacheStats() {

}
Expand Down Expand Up @@ -66,14 +62,6 @@ public Map<String,String> getIncomingQueryMetrics() {
return incomingQueryMetrics;
}

public void setLastWrittenQueryMetrics(Map<String,String> stats) {
this.lastWrittenQueryMetrics = stats;
}

public Map<String,String> getLastWrittenQueryMetrics() {
return lastWrittenQueryMetrics;
}

public void setServiceStats(Map<String,String> serviceStats) {
this.serviceStats = serviceStats;
}
Expand All @@ -90,11 +78,11 @@ public boolean equals(Object o) {
return false;
CacheStats that = (CacheStats) o;
return host.equals(that.host) && memberUuid.equals(that.memberUuid) && incomingQueryMetrics.equals(that.incomingQueryMetrics)
&& lastWrittenQueryMetrics.equals(that.lastWrittenQueryMetrics) && serviceStats.equals(that.serviceStats);
&& serviceStats.equals(that.serviceStats);
}

@Override
public int hashCode() {
return Objects.hash(host, memberUuid, incomingQueryMetrics, lastWrittenQueryMetrics, serviceStats);
return Objects.hash(host, memberUuid, incomingQueryMetrics, serviceStats);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
package datawave.microservice.querymetric;

import java.io.Serializable;

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

import org.apache.commons.text.StringEscapeUtils;

/**
* This represents a datawave geo function and the associated query geometry in GeoJSON format.
*/
@XmlRootElement
@XmlAccessorType(XmlAccessType.NONE)
public class QueryGeometry implements Serializable {

@XmlElement
private String function;
@XmlElement
private String geometry;

public QueryGeometry() {

}

public QueryGeometry(String function, String geometry) {
this.function = function;
this.geometry = geometry;
}

public String getFunction() {
return function;
}

public void setFunction(String function) {
this.function = function;
}

public String getGeometry() {
return geometry;
}

public void setGeometry(String geometry) {
this.geometry = geometry;
}

public String toGeoJsonFeature() {
return "{'type': 'Feature', 'properties': {'function': \"" + StringEscapeUtils.escapeHtml4(function) + "\"},'geometry': " + geometry + "}";
}

@Override
public boolean equals(Object o) {
if (this == o)
return true;
if (!(o instanceof QueryGeometry))
return false;

QueryGeometry that = (QueryGeometry) o;

if (function != null ? !function.equals(that.function) : that.function != null)
return false;
return geometry != null ? geometry.equals(that.geometry) : that.geometry == null;
}

@Override
public int hashCode() {
int result = function != null ? function.hashCode() : 0;
result = 31 * result + (geometry != null ? geometry.hashCode() : 0);
return result;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
package datawave.microservice.querymetric;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

import javax.xml.bind.annotation.XmlAccessOrder;
import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorOrder;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlElementWrapper;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlTransient;

import com.fasterxml.jackson.annotation.JsonIgnore;

import datawave.webservice.HtmlProvider;
import datawave.webservice.result.BaseResponse;

/**
* This response includes information about what query geometries were present in a given query. The geometries are displayed on a map using leaflet.
*/
@XmlRootElement(name = "QueryGeometry")
@XmlAccessorType(XmlAccessType.NONE)
@XmlAccessorOrder(XmlAccessOrder.ALPHABETICAL)
public class QueryGeometryResponse extends BaseResponse implements HtmlProvider {

private static final long serialVersionUID = 1L;

private static final String TITLE = "Query Geometry";

private String LEAFLET_INCLUDES;
private String JQUERY_INCLUDES;
private String MAP_INCLUDES;

public QueryGeometryResponse() {
this(null, null);
}

public QueryGeometryResponse(String queryId, String basemaps) {
this.queryId = queryId;
this.basemaps = basemaps;
setHtmlIncludePaths(new HashMap<>());
}

public void setHtmlIncludePaths(Map<String,String> pathMap) {
// @formatter:off
LEAFLET_INCLUDES =
"<link rel='stylesheet' type='text/css' href='" + pathMap.getOrDefault("leaflet", "") + "/leaflet.css' />\n" +
"<script type='text/javascript' src='" + pathMap.getOrDefault("leaflet", "") + "/leaflet.js'></script>\n";
JQUERY_INCLUDES =
"<script type='text/javascript' src='" + pathMap.getOrDefault("jquery", "") + "/jquery.min.js'></script>\n";
MAP_INCLUDES =
"<link rel='stylesheet' type='text/css' href='" + pathMap.getOrDefault("css", "") + "/queryMap.css' />\n" +
"<script type='text/javascript' src='" + pathMap.getOrDefault("js", "") + "/queryMap.js'></script>";
// @formatter:on
}

@XmlElement(name = "queryId", nillable = true)
protected String queryId = null;

@JsonIgnore
@XmlTransient
protected String basemaps = null;

@XmlElementWrapper(name = "features")
@XmlElement(name = "feature")
protected List<QueryGeometry> result = null;

@JsonIgnore
@XmlTransient
@Override
public String getTitle() {
if (queryId != null)
return TITLE + " - " + queryId;
return TITLE;
}

@JsonIgnore
@XmlTransient
@Override
public String getHeadContent() {
String basemapData = "<script type='text/javascript'>var basemaps = " + basemaps + ";</script>\n";
String featureData = "<script type='text/javascript'>var features = " + toGeoJsonFeatures() + ";</script>\n";
return String.join("\n", featureData, JQUERY_INCLUDES, LEAFLET_INCLUDES, basemapData, MAP_INCLUDES);
}

@JsonIgnore
@XmlTransient
@Override
public String getPageHeader() {
return getTitle();
}

@JsonIgnore
@XmlTransient
@Override
public String getMainContent() {
return "<div align='center'><div id='map' style='height: calc(100% - 85px); width: 100%; position: fixed; top: 86px; left: 0px;'></div></div>";
}

private String toGeoJsonFeatures() {
if (!this.result.isEmpty())
return "[ " + this.result.stream().map(QueryGeometry::toGeoJsonFeature).collect(Collectors.joining(", ")) + " ]";
else
return "undefined";
}

public String getQueryId() {
return queryId;
}

public void setQueryId(String queryId) {
this.queryId = queryId;
}

public List<QueryGeometry> getResult() {
return result;
}

public void setResult(List<QueryGeometry> result) {
this.result = result;
}

public String getBasemaps() {
return basemaps;
}

public void setBasemaps(String basemaps) {
this.basemaps = basemaps;
}
}
Loading
Loading