Sandbox on Cloud

[HasnainRaz]

Going through the deployment guide, I came across the part:

Flyte Sandbox on the Cloud
Sometimes it’s also helpful to be able to install a sandboxed environment on a cloud provider. That is, you have access to an EKS or GKE cluster, but provisioning a separate database or blob storage bucket is harder because of a lack of infrastructure support. Instructions for how to do this will be forthcoming.

This sounds like exactly what I want to do. I want it on the cloud but with ingress setup such that my team can try it out. I have an EKS cluster, but getting S3 and RDS access is not feasible. Does anyone know when these instructions will be available? OR if anyone can point me to how I can use the helm charts to do it myself, what would be needed?

Am I correct in thinking I:

1. Spin up postgres using the bitnami chart
2. Spin up minio
3. Sse the flyte-binary helm chart, but provide the minio and postgres values instead of s3 and rds.
4. Get rid of the service accounts
5. Add ingress (This one I'm not sure about how to do)

[kumare3]

I would recommend just run the single binary chart, it will spin up everything? And expose the ports somehow - ingress, port-forward etc

[HasnainRaz]

@kumare3 https://github.com/flyteorg/flyte/blob/master/charts/flyte-binary/values.yaml#L60

Seems like the binary chart only supports s3 or gcs, how do I get around that? Will it also spin up it's own postgres?

[davidmirror-ops]

@HasnainRaz I'm working on getting this guide out.
The process I'm following (which is similar to what other users have done) is to use the flyte-deps chart to bootstrap minio and postgres (with a few tweaks to run Postgres on a PersistentVolume) and then use flyte-binary (with some adjustments) to actually install Flyte

I'll let you know once the guide is ready.

[HasnainRaz]

Amazing, is there an ETA on this guide?

[davidmirror-ops]

I hope this is out by EOW next week

[HasnainRaz]

@davidmirror-ops Hey David, any updates now?

[davidmirror-ops]

@HasnainRaz I've a mostly working setup that goes like this:

1. Start a k3d cluster (with some args if you're on Mac OS M1)
2. Create the flyte namespace
3. Submit the following manifest:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-persistent
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgresql-pvc
  namespace: flyte
spec:
  storageClassName: local-persistent
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: flyte
  labels: 
    app.kubernetes.io/name: postgres
    app.kubernetes.io/instance: release-name
    helm.sh/chart: flyte-deps-v0.1.10
    app.kubernetes.io/managed-by: Helm
spec:
  type: NodePort
  ports:
  - name: postgres
    port: 5432
    nodePort: 30089
    protocol: TCP
    targetPort: postgres
  selector: 
    app.kubernetes.io/name: postgres
    app.kubernetes.io/instance: release-name
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: flyte
  labels: 
    app.kubernetes.io/name: postgres
    app.kubernetes.io/instance: release-name
    helm.sh/chart: flyte-deps-v0.1.10
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels: 
      app.kubernetes.io/name: postgres
      app.kubernetes.io/instance: release-name
  template:
    metadata:
      labels: 
        app.kubernetes.io/name: postgres
        app.kubernetes.io/instance: release-name
        helm.sh/chart: flyte-deps-v0.1.10
        app.kubernetes.io/managed-by: Helm
    spec:
      containers:
      - image: "ecr.flyte.org/ubuntu/postgres:13-21.04_beta"
        imagePullPolicy: "IfNotPresent"
        name: postgres
        env:
        - name: POSTGRES_HOST_AUTH_METHOD
          value: trust
        - name: POSTGRES_DB
          value: flyteadmin
        ports:
        - containerPort: 5432
          name: postgres
        resources: 
          limits:
            cpu: 1000m
            memory: 512Mi
          requests:
            cpu: 10m
            memory: 128Mi
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgres-storage
          persistentVolumeClaim:
            claimName: postgresql-pvc

4. Install the flyte-deps chart making sure to disable postrgres DB creation:

postgres:
  # --- enable or disable Postgres deployment installation
  enabled: false

5. Install flyte-binary chart adjusting the values to the local DB and minio instances:

configuration:
  database:
    username: postgres
    password: <db-password>
    host: postgres.flyte.svc.cluster.local #uses K8s service discovery
    dbname: flyteadmin  
  storage:
    type: minio
    metadataContainer: my-s3-bucket 
    userDataContainer: my-s3-bucket
    provider: s3
    providerConfig:
      s3:
        region: "us-east-1"  #irrelevant but needed
        authType: "accesskey"
        endpoint: "http://minio.flyte.svc.cluster.local:9000"
        accessKey: "minio"
        secretKey: "miniostorage"
        disableSSL: "true"
        secure: "false"

  inline:
    plugins:
      k8s:
        inject-finalizer: true
        default-env-vars:
          - FLYTE_AWS_ENDPOINT: "http://minio.flyte.svc.cluster.local:9000"
          - FLYTE_AWS_ACCESS_KEY_ID: "minio"
          - FLYTE_AWS_SECRET_ACCESS_KEY: "miniostorage"

serviceAccount:
  create: true

6. Port forward the minio service to the local 9000 port and write a local DNS entry to match minio's endpoint URL:

cat /etc/hosts                                                                                                                                                     ✔
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       minio.flyte.svc.cluster.local
127.0.0.1	localhost

7. Port forward both flyte-binary services:
   kubectl -n flyte port-forward service/flyte-binary-http 8088:8088
kubectl -n flyte port-forward service/flyte-binary-grpc 8089:8089

8. You can move to using Flyte.
   BUT, I'm struggling right now with a totally non-Flyte related issue where my K3d node runs out of storage so it's unable to actually finish running a workflow task. Other than that, from the platform perspective, seems to be working.

I'll be posting an improved version of this first stage as a guide, then adding an Ingress (and circumventing the Docker networking structure) to potentially avoid some of the port forwards.

[HasnainRaz]

Hey @davidmirror-ops awesome thanks so much! I would be very much interested in the ingress as well, as I would like to deploy it to my kubernetes cluster and have my team try it out for testing.