Added example on CRI parsing longlogs (#242)

* Added example on CRI parsing longlogs * Update logs/fluentd/k8s-helm/http/README.md Co-authored-by: Matej Gera <[email protected]> * Update logs/fluentd/k8s-helm/http/README.md Co-authored-by: Matej Gera <[email protected]> * Update logs/fluentd/k8s-helm/http/README.md Co-authored-by: Matej Gera <[email protected]> * Update logs/fluentd/k8s-helm/http/README.md Co-authored-by: Matej Gera <[email protected]> * fmt-docs update --------- Co-authored-by: Matej Gera <[email protected]>
coralogix · Jul 10, 2023 · 81ee28d · 81ee28d
1 parent 80cd711
commit 81ee28d
Showing 1 changed file with 29 additions and 0 deletions.
diff --git a/logs/fluentd/k8s-helm/http/README.md b/logs/fluentd/k8s-helm/http/README.md
@@ -88,3 +88,32 @@ By default this chart installs additional dependent chart:
 ## Coralogix Fluentd Buffer Alert
 
 In order to create an alert on Fluentd buffer in Coralogix, please see [coralogix-alert doc](https://github.com/coralogix/telemetry-shippers/blob/master/logs/fluentd/docs/coralogix-alerts.md)
+
+## Log Logs: containerd / CRI partial logs
+
+If your application is generating logs longer than 16k, you should note that the Docker driver will split the logs into multiple messages.
+To resolve this, you can use the `concat` filter.
+
+First ensure that in the override file used to deploy the Helm chart has `logtag` as one of the regex group keys, like the following:
+
+```yaml
+<pattern>
+  format /^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$/
+  time_format %Y-%m-%dT%H:%M:%S.%L%z
+  keep_time_key true
+</pattern>
+```
+
+If that is not the case please replace the existing one with this one.
+
+Then, next to the source, you'll need to add the following filter that will concat the logs:
+
+```yaml
+<filter raw.containers.**>
+  @type concat
+  key message
+  use_partial_cri_logtag true
+  partial_cri_logtag_key logtag
+  partial_cri_stream_key stream
+</filter>
+```