Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kcg on k8s #20

Open
networkop opened this issue Jul 16, 2021 · 2 comments
Open

kcg on k8s #20

networkop opened this issue Jul 16, 2021 · 2 comments
Labels
enhancement New feature or request question Further information is requested

Comments

@networkop
Copy link

Hey @lspgn 👋
I've done a POC recently of one of your example pipelines (Flows + Kafka + Clicklhouse + Grafana) deployed on top of k8s.
Would you be interested in that? I still have all of the commands and files available.

@lspgn
Copy link
Member

lspgn commented Aug 2, 2021

Hi @networkop, apologies for the delay. This is great!
Kafka and Clickhouse are big deployments and require operators: it might be interesting to have a Deployment/Helm for GoFlow.
Do you have a repository with the files?

@lspgn lspgn added question Further information is requested enhancement New feature or request labels Aug 2, 2021
@networkop
Copy link
Author

so far I have a bunch of interactive steps. with a bit of effort they can be turned into some flux manifests to deploy all this stuff in one go. Here's how it looks like

Install kafka

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install sflow bitnami/kafka \
--set persistence.enabled=false \
--set zookeeper.persistence.enabled=false

# The resulting kafka API endpoint will be sflow-kafka.sflow.svc.cluster.local:9092

Install clickhouse

curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/clickhouse/create.sh
curl -LO  https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/clickhouse/protocols.csv
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/pb/flow.proto

sed -i 's/kafka:9092/sflow-kafka.sflow.svc.cluster.local:9092/' create.sh

# this is needed because configmaps are mounted as read-only and clickhouse expects 
# to chown files in these directories and will crash if it can't do it
sed -i '1 a cp /tmp/flow.proto /var/lib/clickhouse/format_schemas/flow.proto' create.sh
sed -i '1 a cp /tmp/protocols.csv /var/lib/clickhouse/user_files/protocols.csv' create.sh

kubectl -n sflow create cm initdb --from-file=create.sh
kubectl -n sflow create cm proto --from-file=flow.proto
kubectl -n sflow create cm csv --from-file=protocols.csv

helm repo add slamdev https://slamdev.github.io/helm-charts/
helm install sflow-db slamdev/altinity-clickhouse-operator --version 0.0.5 
cat << EOF > db.yml
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "sflow-db"
spec:
  configuration:
    users:
      grafana/password: grafana
      grafana/profile: default
      grafana/quota: default
      grafana/networks/ip: "::/0"
    clusters:
      - name: "clickhouse"
        templates:
          podTemplate: db-with-goflow
          serviceTemplates: chi-service-template
        layout:
          shardsCount: 1
          replicasCount: 1
  templates:
    serviceTemplates:
    - name: chi-service-template
      spec:
        type: ClusterIP
    podTemplates:
    - name: db-with-goflow
      spec:
          containers:
            - name: clickhouse
              image: yandex/clickhouse-server:21.5.6.6
              ports:
                - name: http
                  containerPort: 8123
              volumeMounts:
                - name: volume-initdb
                  mountPath: /docker-entrypoint-initdb.d/
                - name: volume-proto
                  mountPath: /tmp/flow.proto
                  subPath: flow.proto
                  readOnly: true
                - name: volume-csv
                  mountPath: /tmp/protocols.csv
                  subPath: protocols.csv
                  readOnly: false
          volumes:
                - name: volume-initdb
                  configMap:
                    name: initdb
                - name: volume-proto
                  configMap:
                    name: proto
                - name: volume-csv
                  configMap:
                    name: csv
EOF

Install goflow2

cat << EOF > deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: goflow
  name: goflow
  namespace: sflow
spec:
  selector:
    matchLabels:
      app: goflow
  template:
    metadata:
      labels:
        app: goflow
    spec:
      containers:
      - image: netsampler/goflow2:latest
        name: goflow
        args:
        - -transport.kafka.brokers=sflow-kafka.sflow.svc.cluster.local:9092
        - -transport=kafka
        - -transport.kafka.topic=flows
        - -format=pb
        - -format.protobuf.fixedlen=true
EOF

k expose -n sflow deploy goflow --type NodePort --port 6343 --protocol UDP
k -n sflow get svc goflow
NAME     TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
goflow   NodePort   10.107.183.222   <none>        6343:30323/UDP   30s

Install Grafana

curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/dashboards/viz-ch.json
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/datasources-ch.yml
curl -LO https://raw.githubusercontent.com/netsampler/goflow2/main/compose/kcg/grafana/dashboards.yml

sed -i 's/db:8123/clickhouse-sflow-db:8123/' datasources-ch.yml
sed -i "s/user: ''/user: 'grafana'/" datasources-ch.yml
sed -i "s/password: ''/password: 'grafana'/" datasources-ch.yml

kubectl -n sflow create cm grafana-ch --from-file=viz-ch.json
kubectl -n sflow create cm grafana-source --from-file=datasources-ch.yml
kubectl -n sflow create cm grafana-dashboards --from-file=dashboards.yml

kubectl label cm grafana-source grafana_datasource=include
kubectl label cm grafana-ch  grafana_dashboard=include

helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana \
--set sidecar.datasources.enabled=true \
--set sidecar.datasources.labelValue=include \
--set sidecar.dashboards.enabled=true \
--set sidecar.dashboards.labelValue=include \
--set service.type=NodePort \
--set adminPassword=grafana \
--set plugins=vertamedia-clickhouse-datasource \
--set env.GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=vertamedia-clickhouse-datasource

k get svc grafana
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
grafana   NodePort   10.102.189.108   <none>        80:31174/TCP   37s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants