From e28cd90a6382140990301e9f08ab3fedb4c81e33 Mon Sep 17 00:00:00 2001 From: juan-coralogix Date: Wed, 21 Jun 2023 15:20:27 -0300 Subject: [PATCH 1/6] Added example on CRI parsing longlogs --- logs/fluentd/k8s-helm/http/README.md | 29 ++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/logs/fluentd/k8s-helm/http/README.md b/logs/fluentd/k8s-helm/http/README.md index 2fcdf7b4..d3cb6dd3 100644 --- a/logs/fluentd/k8s-helm/http/README.md +++ b/logs/fluentd/k8s-helm/http/README.md @@ -88,3 +88,32 @@ By default this chart installs additional dependent chart: ## Coralogix Fluentd Buffer Alert In order to create an alert on Fluentd buffer in Coralogix, please see [coralogix-alert doc](https://github.com/coralogix/telemetry-shippers/blob/master/logs/fluentd/docs/coralogix-alerts.md) + + +## Log Logs: containerd / CRI partial logs + +If your application is generating logs longer than 16k you should notice that docker dirver is splitting the log in multiple messages. +To fix this we can use concat to fix this. + +First lets make sure that in the override file, that you use to deploy the helm, has logtag as one of the regex group keys, just like this. + +```yaml + + format /^(? +``` +If that is not the case please replace the existing one with this one. + +Then next to the source we will add the following filter that will concat the logs: + +```yaml + + @type concat + key message + use_partial_cri_logtag true + partial_cri_logtag_key logtag + partial_cri_stream_key stream + +``` From 788296a99ee5091ceb5b172dc39b420505af405e Mon Sep 17 00:00:00 2001 From: juan-coralogix <89215136+juan-coralogix@users.noreply.github.com> Date: Mon, 10 Jul 2023 08:57:20 -0300 Subject: [PATCH 2/6] Update logs/fluentd/k8s-helm/http/README.md Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> --- logs/fluentd/k8s-helm/http/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/logs/fluentd/k8s-helm/http/README.md b/logs/fluentd/k8s-helm/http/README.md index d3cb6dd3..6bbef9b3 100644 --- a/logs/fluentd/k8s-helm/http/README.md +++ b/logs/fluentd/k8s-helm/http/README.md @@ -92,7 +92,7 @@ In order to create an alert on Fluentd buffer in Coralogix, please see [coralogi ## Log Logs: containerd / CRI partial logs -If your application is generating logs longer than 16k you should notice that docker dirver is splitting the log in multiple messages. +If your application is generating logs longer than 16k, you should note that the Docker driver will split the logs into multiple messages. To fix this we can use concat to fix this. First lets make sure that in the override file, that you use to deploy the helm, has logtag as one of the regex group keys, just like this. From 5771d8f94f6c12fa1334e1f4f35ed7f720909e9f Mon Sep 17 00:00:00 2001 From: juan-coralogix <89215136+juan-coralogix@users.noreply.github.com> Date: Mon, 10 Jul 2023 08:57:34 -0300 Subject: [PATCH 3/6] Update logs/fluentd/k8s-helm/http/README.md Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> --- logs/fluentd/k8s-helm/http/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/logs/fluentd/k8s-helm/http/README.md b/logs/fluentd/k8s-helm/http/README.md index 6bbef9b3..3a7240da 100644 --- a/logs/fluentd/k8s-helm/http/README.md +++ b/logs/fluentd/k8s-helm/http/README.md @@ -93,7 +93,7 @@ In order to create an alert on Fluentd buffer in Coralogix, please see [coralogi ## Log Logs: containerd / CRI partial logs If your application is generating logs longer than 16k, you should note that the Docker driver will split the logs into multiple messages. -To fix this we can use concat to fix this. +To resolve this, you can use the `concat` filter. First lets make sure that in the override file, that you use to deploy the helm, has logtag as one of the regex group keys, just like this. From 5e144cef379d7bd962ba33f524d7157a48317da3 Mon Sep 17 00:00:00 2001 From: juan-coralogix <89215136+juan-coralogix@users.noreply.github.com> Date: Mon, 10 Jul 2023 08:58:08 -0300 Subject: [PATCH 4/6] Update logs/fluentd/k8s-helm/http/README.md Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> --- logs/fluentd/k8s-helm/http/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/logs/fluentd/k8s-helm/http/README.md b/logs/fluentd/k8s-helm/http/README.md index 3a7240da..4682887c 100644 --- a/logs/fluentd/k8s-helm/http/README.md +++ b/logs/fluentd/k8s-helm/http/README.md @@ -95,7 +95,7 @@ In order to create an alert on Fluentd buffer in Coralogix, please see [coralogi If your application is generating logs longer than 16k, you should note that the Docker driver will split the logs into multiple messages. To resolve this, you can use the `concat` filter. -First lets make sure that in the override file, that you use to deploy the helm, has logtag as one of the regex group keys, just like this. +First ensure that in the override file used to deploy the Helm chart has `logtag` as one of the regex group keys, like the following: ```yaml From f92ef41a992a5d416e91e9c962b8f49bfe8701e0 Mon Sep 17 00:00:00 2001 From: juan-coralogix <89215136+juan-coralogix@users.noreply.github.com> Date: Mon, 10 Jul 2023 08:58:18 -0300 Subject: [PATCH 5/6] Update logs/fluentd/k8s-helm/http/README.md Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com> --- logs/fluentd/k8s-helm/http/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/logs/fluentd/k8s-helm/http/README.md b/logs/fluentd/k8s-helm/http/README.md index 4682887c..7bffbf88 100644 --- a/logs/fluentd/k8s-helm/http/README.md +++ b/logs/fluentd/k8s-helm/http/README.md @@ -106,7 +106,7 @@ First ensure that in the override file used to deploy the Helm chart has `logtag ``` If that is not the case please replace the existing one with this one. -Then next to the source we will add the following filter that will concat the logs: +Then, next to the source, you'll need to add the following filter that will concat the logs: ```yaml From b68706653459cb181dcb3108243aa69377347a75 Mon Sep 17 00:00:00 2001 From: juan-coralogix Date: Mon, 10 Jul 2023 09:04:57 -0300 Subject: [PATCH 6/6] fmt-docs update --- logs/fluentd/k8s-helm/http/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/logs/fluentd/k8s-helm/http/README.md b/logs/fluentd/k8s-helm/http/README.md index 7bffbf88..9dcee5f0 100644 --- a/logs/fluentd/k8s-helm/http/README.md +++ b/logs/fluentd/k8s-helm/http/README.md @@ -89,7 +89,6 @@ By default this chart installs additional dependent chart: In order to create an alert on Fluentd buffer in Coralogix, please see [coralogix-alert doc](https://github.com/coralogix/telemetry-shippers/blob/master/logs/fluentd/docs/coralogix-alerts.md) - ## Log Logs: containerd / CRI partial logs If your application is generating logs longer than 16k, you should note that the Docker driver will split the logs into multiple messages. @@ -104,6 +103,7 @@ First ensure that in the override file used to deploy the Helm chart has `logtag keep_time_key true ``` + If that is not the case please replace the existing one with this one. Then, next to the source, you'll need to add the following filter that will concat the logs: