HPCC4J Tracing Support
Tracing is a critical feature for client libraries as it enhances performance monitoring, debugging, dependency mapping, operational insights, user experience, compliance, and continuous improvement. By providing detailed visibility into the execution flow and performance metrics, tracing helps developers build more reliable, efficient, and maintainable systems.
The HPCC4J, DFSClient and Spark-HPCC client libraries leverage the OpenTelemetry SDK and follows their standards for tracing information propogation across distributed components. HPCC4J's tracing features have shifted to heavily rely on OpenTelemetry's zero-code JavaAgent starting in v.9.6.48 and v.9.8.22. Client applications incorporating the HPCC Java client libraries are encouraged to utilize the Otel JavaAgent in order to enable in-depth tracing. Even though it is not always possible, any client application whishing to trace interactions with HPCC should target the same trace aggregation system (Elastic, Jaeger, etc) as the target HPCC Platform.
+---------------------+ +---------------------+ +---------------------+ +---------------------+
| | | | | | | |
| Client Application |<-----> | hpcc4j Library |<-----> | HPCC Platform |<-----> | External Gateways |
| | | | | ESP Roxie Thor | | |
+---------------------+ +---------------------+ +---------------------+ +---------------------+
| | | | |
Collector Collector Collector Collector (Collector topology is optional)
| | | |
-----------------------------------------------------------------------
|
+---------------------+
| |
|Observability Service| -----> Trace Visualizations
| |
+---------------------+
(Elastic+APM, Jaeger, Zipkin, etc.)
HPCC4J library 9.6.48+ 9.8.22+
OpenTelemetry Java Agent 2.6.0+
Access to observability service (Elastic with APM, Jaeger, Zipkin, etc.) for span information aggregation and trace visualization
OpenTelemetry collector (Intermediary trace information handler sometimes needed as a gateway to observability service)
Tracing information is most useful when aggregated and colated by Observability tools which provide visualizations and query interfaces. Ideally all components involved in the distributed system exports their respetive trace information to a central service in order to achieve a comprehensive view of all transactions. Many observability service options such as Elastic+APM, Jaeger, Zipkin, are available, and their deployment and configuration varies. Some of these tools require an intermediary gateway process which collects exported trace data and forwards it to the service via a supported protocol. OponTelemetry offers such a gateway service called a "collector", the collector can be either deployed local to the application to be instrumented, or as a centralized service.
Once the service or OpenTelemetry collector(s) is setup and accessible, the client application can be configured to export trace data to it. All observability service (or Otel Collector) connectivity inforamtion should be identified and used to configure the trace exporter in the next steps.
The application to be instrumented is responsible for enabling and configuring tracing for the application and hpcc4j client library. Configure the OpenTelemetry SDK using environment variables or system properties. Trace exporter information should target the observability service discussed above. Here are some common configurations options:
System property Description
* otel.java.global-autoconfigure.enabled Enable opentelemetry auto configuration based on environment variables or system properties
* otel.sdk.disabled If true, disable the OpenTelemetry SDK.
* otel.service.name Specify logical service name. Takes precedence over service.name defined with otel.resource.attributes.
* otel.resource.attributes Specify resource attributes in the following format: key1=val1,key2=val2,key3=val3.
* otel.traces.sampler The sampler to use. Known values include always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, parentbased_traceidratio, jaeger_remote. [1] parentbased_always_on
* otel.traces.sampler.arg An argument to the configured tracer if supported, for example a ratio.
* otel.span.attribute.value.length.limit The maximum length of span attribute values. Takes precedence over otel.attribute.value.length.limit. No limit
* otel.span.attribute.count.limit The maximum number of attributes per span. Takes precedence over otel.attribute.count.limit. 128
* otel.span.event.count.limit The maximum number of events per span. 128
* otel.span.link.count.limit The maximum number of links per span. 128
* otel.traces.exporter Comma-separated list of span exporters. Known values include otlp, zipkin, console, logging-otlp, none.
* otel.exporter.otlp.protocol The transport protocol to use on OTLP trace, metric, and log requests. Options include grpc and http/protobuf. Default: grpc
* otel.exporter.otlp.{signal}.protocol The transport protocol to use on OTLP {signal} requests. Options include grpc and http/protobuf. grpc [1]
* otel.exporter.otlp.endpoint The endpoint to send all OTLP traces, metrics, and logs to. Often the address of an OpenTelemetry Collector. Must be a URL with a scheme of either http or https based on the use of TLS. If protocol is http/protobuf the version and signal will be appended to the path (e.g. v1/traces, v1/metrics, or v1/logs). http://localhost:4317 when protocol is grpc, and http://localhost:4318/v1/{signal} when protocol is http/protobuf.
* otel.exporter.otlp.{signal}.endpoint The endpoint to send OTLP {signal} to. Often the address of an OpenTelemetry Collector. Must be a URL with a scheme of either http or https based on the use of TLS. http://localhost:4317 when protocol is grpc, and http://localhost:4318/v1/{signal} when protocol is http/protobuf.
* otel.exporter.otlp.certificate The path to the file containing trusted certificates to use when verifying an OTLP trace, metric, or log server’s TLS credentials. The file should contain one or more X.509 certificates in PEM format. The host platform’s trusted root certificates are used.
* otel.exporter.otlp.{signal}.certificate The path to the file containing trusted certificates to use when verifying an OTLP {signal} server’s TLS credentials. The file should contain one or more X.509 certificates in PEM format. The host platform’s trusted root certificates are used
* otel.exporter.otlp.client.key The path to the file containing private client key to use when verifying an OTLP trace, metric, or log client’s TLS credentials. The file should contain one private key PKCS8 PEM format. No client key file is used.
* otel.exporter.otlp.{signal}.client.key The path to the file containing private client key to use when verifying an OTLP {signal} client’s TLS credentials. The file should contain one private key PKCS8 PEM format. No client key file is used.
* otel.exporter.otlp.client.certificate The path to the file containing trusted certificates to use when verifying an OTLP trace, metric, or log client’s TLS credentials. The file should contain one or more X.509 certificates in PEM format. No chain file is used.
* otel.exporter.otlp.{signal}.client.certificate The path to the file containing trusted certificates to use when verifying an OTLP {signal} server’s TLS credentials. The file should contain one or more X.509 certificates in PEM format. No chain file is used.
* otel.exporter.otlp.headers Key-value pairs separated by commas to pass as request headers on OTLP trace, metric, and log requests.
* otel.exporter.otlp.{signal}.headers Key-value pairs separated by commas to pass as request headers on OTLP {signal} requests.
* otel.exporter.otlp.compression The compression type to use on OTLP trace, metric, and log requests. Options include gzip. No compression will be used.
* otel.exporter.otlp.{signal}.compression The compression type to use on OTLP {signal} requests. Options include gzip. No compression will be used.
* otel.exporter.otlp.timeout The maximum waiting time, in milliseconds, allowed to send each OTLP trace, metric, and log batch. 10000
* otel.exporter.otlp.{signal}.timeout The maximum waiting time, in milliseconds, allowed to send each OTLP {signal} batch. 10000
java -jar my-hpcc-application.jar
-Dotel.java.global-autoconfigure.enabled=true
-Dotel.traces.exporter=otlp
-Dotel.exporter.otlp.protocol=http/protobuf
-Dotel.exporter.otlp.endpoint=http://localhost:4318
-Dotel.service.name=MyHPCCAppName
To achieve full instrumentation of the HPCC4J library, the client application deployment should include the OpenTelemetry zero-code JavaAgent. To attach the JavaAgent, the jar should be downloaded from the OpenTelemetry website and targeted with the "-javaagent" JVM option:
java -javaagent:/path/to/opentelemetry-javaagent.jar -jar my-hpcc-application.jar
-Dotel.java.global-autoconfigure.enabled=true
-Dotel.traces.exporter=otlp
-Dotel.exporter.otlp.protocol=http/protobuf
-Dotel.exporter.otlp.endpoint=http://localhost:4318
-Dotel.service.name=MyHPCCAppName
Sample visualization of a transaction triggered from an HPCC4J API call. The diagram includes spans representing units of work within the HPCC4J library, network requests to HPCC components, ESP and HThor spans.
