diff --git a/logs/fluent-bit/ecs-fargate/README.md b/logs/fluent-bit/ecs-fargate/README.md index 8468191e..5cb34619 100644 --- a/logs/fluent-bit/ecs-fargate/README.md +++ b/logs/fluent-bit/ecs-fargate/README.md @@ -1,5 +1,7 @@ # fluentbit ECS Fargate container +# Note: This integration is for logs only. You can now collect logs from ECS fargate tasks through OTEL using our otel-ecs-fargate integration. Only use this integration if you only intend to ingest logs. + fluentbit is a lightweight data shipper that we are using as a logs shipper for AWS ECS Fargate workloads. Here we explain how to deploy the fluentbit log_router into an existing AWS ECS Fargate task definition. We use an AWS customized fluentbit image called aws-for-fluent-bit, init version, as it has several features that allow for more convenient management of the configuration. We also have an example cloudformation template for review [here](https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/aws-integrations/ecs-fargate) @@ -76,7 +78,7 @@ In order to allow container access to the S3 object, you'll need to provide the } ``` -Note: Don't confuse Task Execution Role for Task Role, this permission needs to be added to the Task Role. (Contrary to the ADOT (OTEL) Metrics and Traces integration) +Note: Don't confuse Task Execution Role for Task Role, this permission needs to be added to the Task Role. After you've added the above container to your existing Task Definition, you need to adjust the logConfiguration for the containers you wish to forward to Coralogix. diff --git a/otel-ecs-fargate/CHANGELOG.md b/otel-ecs-fargate/CHANGELOG.md new file mode 100644 index 00000000..30f407b8 --- /dev/null +++ b/otel-ecs-fargate/CHANGELOG.md @@ -0,0 +1,14 @@ +# Changelog + +## otel-ecs-fargate + + + + + + + +### 0.0.1 / 2024-09-11 + +### 🛑 Breaking changes 🛑 +* [UPDATE] Update ecs-fargate integration to OTEL only (remove fluentbit logrouter) diff --git a/otel-ecs-fargate/README.md b/otel-ecs-fargate/README.md index 2734ef59..f7c7f78d 100644 --- a/otel-ecs-fargate/README.md +++ b/otel-ecs-fargate/README.md @@ -2,15 +2,17 @@ ### Note: Previous versions of this integration used an ADOT (AWS Distribution for OpenTelemetry) collector image. If you are upgrading an existing deployment, make sure you upgrade both the configuration and the task definition. +### Note: Previous versions of this integration required logs to be processed using fluentbit logrouter. This is no longer necessary and logs can be collected by OTEL along with the metrics and traces. + The OpenTelemetry collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data. In this document, we'll explain how to add the OTEL collector as a sidecar agent to your ECS Task Definitions. We use the standard Opentelemetry Collector Contrib distribution but leverage the envprovider to generate the configuration from an AWS SSM Parameter Store. There is an example cloudformation template for review [here](https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/aws-integrations/ecs-fargate) The envprovider is used for loading of the OpenTelemetry configuration via Systems Manager Parameter Stores. This makes adjusting your configuration more convenient and more dynamic than baking a static configuration into your container image. -Our config.yaml file includes a standard configuration that'll ensure proper ingestion by our backend. Make sure to create this parameter store in the same region as your ECS cluster. We've included a sample cloudformation template to deploy this parameter store to simplify this process. +Our config.yaml file includes a standard configuration that'll ensure proper ingestion of logs, metrics and traces by our backend. Make sure to create this parameter store in the same region as your ECS cluster. We've included a sample cloudformation template to deploy this parameter store to simplify this process. -Once the Parameter Store has been created, you'll need to add the container to your existing Task Definition. +Once the Parameter Store has been created, you'll need to add the OTEL container to your existing Task Definition(s). Example container declaration within a Task Definition: @@ -62,38 +64,33 @@ Example container declaration within a Task Definition: "valueFrom": "/CX_OTEL/config.yaml" } ], + "user": "0", "logConfiguration": { "logDriver": "awsfirelens", "options": { - "Format": "json_lines", - "Header": "Authorization Bearer ", - "Host": "ingress.", - "Name": "http", - "Port": "443", - "Retry_Limit": "10", - "TLS": "On", - "URI": "/logs/v1/singles", - "compress": "gzip" + "Name": "OpenTelemetry" } + }, + "systemControls": [], + "firelensConfiguration": { + "type": "fluentbit" } } ] ``` -In the example above, you'll need to set two instances each of `` and ``. The logConfiguration section included in the example will forward OTEL logs to the Coralogix platform, as documented in our fluentbit log processing configuration instructions [here](../logs/fluent-bit/ecs-fargate/README.md). - -**NOTE:** If you wish to store your Coralogix Privatekey in Secret Manager, you can remove the `"Header"` from `"options"` and create one under `"secretOptions"` and reference the Secret's ARN. Create the secret as plaintext with the same format as above. You will also need to add the secretsmanager:GetSecretValue permission to your ecs Task Execution Role. +In the example above, you'll need to set `` and ``. The logConfiguration section included in the example will forward OTEL logs to the Coralogix platform. Make sure you set all your existing containers' logConfiguration to the same. ``` -"secretOptions": [ - { - "name": "Header", - "valueFrom": "arn:aws:secretsmanager:us-east-1::secret:" +"logConfiguration": { + "logDriver": "awsfirelens", + "options": { + "Name": "OpenTelemetry" } -] +}, ``` -If you don't want to have them submitted to the Coralogix platform, you can replace the logConfiguration with whichever logDriver configuration you would prefer. To submit to Cloudwatch, you can configure as so: +If you don't want to have logs submitted to the Coralogix platform, you can replace the logConfiguration with whichever logDriver configuration you would prefer. To submit to Cloudwatch, you can configure as so: ``` "logConfiguration": { @@ -107,6 +104,17 @@ If you don't want to have them submitted to the Coralogix platform, you can repl } ``` +**NOTE:** If you wish to store your Coralogix Privatekey in Secret Manager, you can remove the `"Header"` from `"options"` and create one under `"secretOptions"` and reference the Secret's ARN. Create the secret as plaintext with the same format as above. You will also need to add the secretsmanager:GetSecretValue permission to your ecs Task Execution Role. + +``` +"secretOptions": [ + { + "name": "Header", + "valueFrom": "arn:aws:secretsmanager:us-east-1::secret:" + } +] +``` + In order to allow container access to the Systems Manager Parameter Store, you'll need to provide the ssm:GetParameters action permissions to the task execution role: ``` @@ -126,6 +134,6 @@ In order to allow container access to the Systems Manager Parameter Store, you'l } ``` -Note: Don't confuse Task Role for Task Execution Role, this permission needs to be added to the Task Execution Role. (Contrary to the fluentbit Logs integration) +Note: This permission needs to be added to the Task Execution Role. After adding the above container to your existing Task Definition, your applications can submit their traces and metrics exports to http://localhost:4318/v1/traces and /v1/metrics. It will also collect container metrics from all containers in the Task Definition. diff --git a/otel-ecs-fargate/config.yaml b/otel-ecs-fargate/config.yaml index 18087c26..b6267ee4 100644 --- a/otel-ecs-fargate/config.yaml +++ b/otel-ecs-fargate/config.yaml @@ -16,11 +16,42 @@ exporters: subsystem_name_attributes: - service.name - aws.ecs.docker.name + - container_name timeout: 30s traces: headers: X-Coralogix-Distribution: ecs-fargate-integration/0.0.1 processors: + transform/firelens: + log_statements: + - context: log + statements: + # parse json logs + - merge_maps(cache, ParseJSON(body), "insert") where IsMatch(body, "^\\{") + # set message + - set(body, cache["message"]) where cache["message"] != nil + + # set trace/span id + - set(trace_id.string, cache["trace_id"]) where cache["trace_id"] != nil + - set(span_id.string, cache["span_id"]) where cache["span_id"] != nil + + # set severity when available + - set(severity_number, SEVERITY_NUMBER_INFO) where IsMatch(cache["level"], "(?i)info") + - set(severity_number, SEVERITY_NUMBER_WARN) where IsMatch(cache["level"], "(?i)warn") + - set(severity_number, SEVERITY_NUMBER_ERROR) where IsMatch(cache["level"], "(?i)err") + - set(severity_number, SEVERITY_NUMBER_DEBUG) where IsMatch(cache["level"], "(?i)debug") + - set(severity_number, SEVERITY_NUMBER_TRACE) where IsMatch(cache["level"], "(?i)trace") + - set(severity_number, cache["severity_number"]) where cache["severity_number"] != nil + + # move log_record attributes to resource + - set(resource.attributes["container_name"], attributes["container_name"]) + - set(resource.attributes["container_id"], attributes["container_id"]) + - delete_key(attributes, "container_id") + - delete_key(attributes, "container_name") + + - delete_matching_keys(cache, "^(message|trace_id|span_id|severity_number)$") + + - merge_maps(attributes,cache, "insert") batch: send_batch_max_size: 2048 send_batch_size: 1024 @@ -38,6 +69,9 @@ processors: override: true timeout: 2s receivers: + fluentforward/socket: + # ECS will send logs to this socket + endpoint: unix:///var/run/fluent.sock awsecscontainermetrics: collection_interval: 10s otlp: @@ -56,6 +90,16 @@ receivers: - 127.0.0.1:8888 service: pipelines: + logs: + exporters: + - coralogix + processors: + - transform/firelens + - resource/metadata + - resourcedetection + - batch + receivers: + - fluentforward/socket metrics: exporters: - coralogix diff --git a/otel-ecs-fargate/parameter_store.cf b/otel-ecs-fargate/parameter_store.cf index c98af726..4cc65f1b 100644 --- a/otel-ecs-fargate/parameter_store.cf +++ b/otel-ecs-fargate/parameter_store.cf @@ -27,11 +27,42 @@ Resources: subsystem_name_attributes: - service.name - aws.ecs.docker.name + - container_name timeout: 30s traces: headers: X-Coralogix-Distribution: ecs-fargate-integration/0.0.1 processors: + transform/firelens: + log_statements: + - context: log + statements: + # parse json logs + - merge_maps(cache, ParseJSON(body), "insert") where IsMatch(body, "^\\{") + # set message + - set(body, cache["message"]) where cache["message"] != nil + + # set trace/span id + - set(trace_id.string, cache["trace_id"]) where cache["trace_id"] != nil + - set(span_id.string, cache["span_id"]) where cache["span_id"] != nil + + # set severity when available + - set(severity_number, SEVERITY_NUMBER_INFO) where IsMatch(cache["level"], "(?i)info") + - set(severity_number, SEVERITY_NUMBER_WARN) where IsMatch(cache["level"], "(?i)warn") + - set(severity_number, SEVERITY_NUMBER_ERROR) where IsMatch(cache["level"], "(?i)err") + - set(severity_number, SEVERITY_NUMBER_DEBUG) where IsMatch(cache["level"], "(?i)debug") + - set(severity_number, SEVERITY_NUMBER_TRACE) where IsMatch(cache["level"], "(?i)trace") + - set(severity_number, cache["severity_number"]) where cache["severity_number"] != nil + + # move log_record attributes to resource + - set(resource.attributes["container_name"], attributes["container_name"]) + - set(resource.attributes["container_id"], attributes["container_id"]) + - delete_key(attributes, "container_id") + - delete_key(attributes, "container_name") + + - delete_matching_keys(cache, "^(message|trace_id|span_id|severity_number)$") + + - merge_maps(attributes,cache, "insert") batch: send_batch_max_size: 2048 send_batch_size: 1024 @@ -49,6 +80,9 @@ Resources: override: true timeout: 2s receivers: + fluentforward/socket: + # ECS will send logs to this socket + endpoint: unix:///var/run/fluent.sock awsecscontainermetrics: collection_interval: 10s otlp: @@ -67,6 +101,16 @@ Resources: - 127.0.0.1:8888 service: pipelines: + logs: + exporters: + - coralogix + processors: + - transform/firelens + - resource/metadata + - resourcedetection + - batch + receivers: + - fluentforward/socket metrics: exporters: - coralogix