Skip to content

philips-software/fluent-bit-out-hsdp

Repository files navigation

fluent bit HSDP logging output plugin

This plugin outputs your logs to the HSDP Host Logging service. This is useful when your workloads are not running on Cloud foundry, but you still want to utilize the central logging facilities of HSDP.

flow

Fluent bit supports parser and filter plugin which can convert unstructured data gathered from the log Input interface into a structured one and to alter existing structured data before ingestion.

More on fluent-bit

Cloud and On-Premise

The plugin supports deployment to both Cloud and On-Premise environments. Depending on the deployment type you can either specify the Cloud Region and Environment or the On-Premise IamUrl and IdmUrl values.

API Signing, Service Identities or Logdrainer

The plugin supports both the API Signing authorization mechanism or the use of a IAM Service Identity with the LOG.CREATE scope. It also supports using a CF Logdrain endpoint although that limits the log format output (e.g. no support for the custom field)

Configuration options

Your fluent-bit.conf file should include an entry like below to enable the plugin:

[output]
    Name hsdp
    Match *

Configuring the authorization mechanism and HSDP Logging endpoints should ideally be done by setting the right Environment variables:

Settings

General

These keys are relevant when using either SigningKey or Service identities

Key Description Environment variable Required
ProductKey The Product key of your proposition HSDP_PRODUCT_KEY Required
Region The HSP Region (Cloud) HSDP_REGION Required
Environment THE HSP Environment (Cloud) HSDP_ENVIRONMENT Required
IamUrl The IAM URL (On-Premise) HSDP_IAM_URL Optional
IdmUrl The IDM URL (On-Premise) HSDP_IDM_URL Optional
IngestorHost The HSDP ingestor host HSDP_INGESTOR_HOST Optional
Debug Shows request details when set to true HSDP_DEBUG Optional
CustomField Adds the field hash to custom field when set to true HSDP_CUSTOM_FIELD Optional
InsecureSkipVerify Skip checking HSDP ingestor TLS cert. Insecure! HSDP_INSECURE_SKIP_VERIFY Optional
SynchronousFlush Flushes log messages synchronously without batching. By default this is set to false Optional
RetryOnError Returns retry to FLB if flush fails. Applicable only when SynchronousFlush option is set. By default this is set to false Optional
Proxy HTTP Proxy URL in case there is a proxy redirection required Optional

Signing keys

Key Description Environment variable Required
SharedKey The Shared key for signing requests HSDP_SHARED_KEY Optional
SecretKey The Secret key for signing requests HSDP_SECRET_KEY Optional

Service identities

Key Description Environment variable Required
ServiceId The Service ID to use for authentication HSDP_SERVICE_ID Optional
ServicePrivateKey The Service private key HSDP_SERVICE_PRIVATE_KEY Optional

Logdrain

You can reuse an existing Cloud foundry logdrainer endpoint to ship your logs to HSDP logging. The advantage is that you only need the Logdrain URL itself, no other configuration. This URL is considered a credential so care should be taken to protect it though.

Key Description Environment variable Required
LogdrainUrl The HSP Logdrain URL HSDP_LOGDRAIN_URL Optional
LogdrainApplicationName The Application name to use HSDP_LOGDRAIN_APPLICATION_NAME Optional
LogdrainServerName The Server name to use HSDP_LOGDRAIN_SERVER_NAME Optional

Environment variable values take precedence over those in configuration files.

Record field mapping to HSDP logging resource

The plugin supports full pass-through of the native LogEvent JSON message type, example:

{
  "resourceType": "LogEvent",
  "id": "7f4c85a8-e472-479f-b772-2916353d02a4",
  "applicationName": "OPS",
  "eventId": "110114",
  "category": "TRACELOG",
  "component": "TEST",
  "transactionId": "2abd7355-cbdd-43e1-b32a-43ec19cd98f0",
  "serviceName": "OPS",
  "applicationInstance": "INST‐00002",
  "applicationVersion": "1.0.0",
  "originatingUser": "SomeUsr",
  "serverName": "ops-dev.cloud.pcftest.com",
  "logTime": "2017-01-31T08:00:00Z",
  "severity": "INFO",
  "logData": {
    "message": "VGVzdCBNZXNzYWdl"
  },
  "custom": {
    "foo": "bar"
  }
}

NOTE: the logData.message field must be base64 encoded

field mapping

Alternatively, the plugin maps certain record fields to defined HSDP logging resource fields. The below table shows the mapping, and the default value.

Record field HSDP logging field Default value Details
server_name serverName fluent-bit
app_name applicationName fluent-bit
app_instance applicationInstance fluent-bit
app_version applicationVersion 1.0
category category TraceLog
severity severity informational
service_name service_name fluent-bit
originating_user originating_user fluent-bit
event_id event_id 1
transaction_id transaction_id random UUID if original input is not a valid UUID a new one will be generated
trace_id trace_id
span_id span_id
logdata_message logData.Message field hash will replace the default field hash dump went present

Fields mapped to a HSDP logging resource field will be removed from the log message dump

The below filter definition shows an example of assigning fields

[filter]
    Name record_modifier
    Match *
    Record server_name ${HOSTNAME}
    Record service_name Awesome_Tool
[filter]
    Name modify
    Match *
    Rename container_name app_name
    Rename container_name service_name
    Rename component_name component
    Rename container_id app_instance

Remaining fields will be rendered to a JSON hash and assigned to logData.Message

Building

docker build -t fluent-bit-out-hsdp .

Testing with Docker

docker run --rm \
    -p 127.0.0.1:24224:24224 \
    -e HSDP_PRODUCT_KEY=product-key-here \
    -e HSDP_REGION=us-east \
    -e HSDP_ENVIRONMENT=client-test \
    -e [email protected] \
    -e HSDP_SERVICE_PRIVATE_KEY="$(cat service_private_key.pem)" \
    -it ghcr.io/philips-software/fluent-bit-out-hsdp:latest

Once the above is running you can start other Docker containers and use fluentd log driver to start logging to HSDP logging:

docker run --rm -it --log-driver fluentd alpine echo "hello world"

Helm Chart

You can deploy fluent-bit and the HSDP plugin using a Helm chart

Secret

The chart will attempt to read credentials from an hsdp-logging Kubernetes secret which should reside in the namespace. An example hsdp-logging-secret.yaml is included below. Make sure you replace the values accordingly.

NOTE: All the values should be in the base64 encoded.

apiVersion: v1
kind: Secret
metadata:
  name: hsdp-logging
type: Opaque
data:
  shared_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  secret_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  product_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  ingestor_host: aHR0cHM6Ly9sb2dpbmdlc3RvcjItY2xpZW50LXRlc3QuZXUtd2VzdC5waGlsaXBzLWhlYWx0aHN1aXRlLmNvbQ==

Apply the secret to the right namepace (for Fiesta this is kube-system):

kubectl apply -f hsdp-logging-secret.yaml -n logging

Add helm repo:

helm repo add philips-software https://philips-software.github.io/helm-charts/

Download the helm chart

helm pull philips-software/fluent-bit-out-hsdp

Navigate to the download Helm chart and unzip.

After unzip, open fluent-bit-out-hsdp\values.yml and add the enviroment details like:

fluent-bit:
  env:
   - name: HSDP_REGION
     valueFrom:
       secretKeyRef:
         key: region
         name: hsdp-logging
         optional: true
   - name: HSDP_ENVIRONMENT
     valueFrom:
       secretKeyRef:
         key: environment
         name: hsdp-logging
         optional: true
   - name: HSDP_PRODUCT_KEY
     valueFrom:
       secretKeyRef:
         key: product_key
         name: hsdp-logging
         optional: false
   - name: HSDP_INGESTOR_HOST
     valueFrom:
       secretKeyRef:
         key: ingestor_host
         name: hsdp-logging
         optional: true
   - name: HSDP_SHARED_KEY
     valueFrom:
       secretKeyRef:
         key: shared_key
         name: hsdp-logging
         optional: true
   - name: HSDP_SECRET_KEY
     valueFrom:
       secretKeyRef:
         key: secret_key
         name: hsdp-logging
         optional: true
   - name: HSDP_SERVICE_ID
     valueFrom:
       secretKeyRef:
         key: service_id
         name: hsdp-logging
         optional: true
   - name: HSDP_SERVICE_PRIVATE_KEY
     valueFrom:
       secretKeyRef:
         key: service_private_key
         name: hsdp-logging
         optional: true

You can skip the unwanted fields from above yaml. For e.g, if you are going to use secret_key and shared_key based authentication, then only fields required are:

fluent-bit:
  env:
   - name: HSDP_PRODUCT_KEY
     valueFrom:
       secretKeyRef:
         key: product_key
         name: hsdp-logging
         optional: false
   - name: HSDP_INGESTOR_HOST
     valueFrom:
       secretKeyRef:
         key: ingestor_host
         name: hsdp-logging
         optional: true
   - name: HSDP_SHARED_KEY
     valueFrom:
       secretKeyRef:
         key: shared_key
         name: hsdp-logging
         optional: true
   - name: HSDP_SECRET_KEY
     valueFrom:
       secretKeyRef:
         key: secret_key
         name: hsdp-logging
         optional: true

Install the Helm chart

helm install my-fluent-bit-out-hsdp fluent-bit-out-hsdp -n logging

If the credentials are correct you should now see your Kubernetes cluster logs in the HSDP Logging system.

Contact / Getting help

Andy Lo-A-Foe [email protected]

License

License is MIT