Extracted common logic to nirmata/kyverno-notation-verifier #7

vishal-chdhry · 2023-06-22T11:53:43Z

Closes: #5
Closes: #2

Common logic has been moved to: https://pkg.go.dev/github.com/nirmata/[email protected]

To pass AWS specific Auth Config (IRSA), we now pass a resolver in option WithProviderAuthConfigResolver.
- Since every image can be from a different reason, We might need a different auth config per image, so I changed from passing authn.Config to a resolver of type func (ctx context.Context, ref registry.Reference) (authn.AuthConfig, error)

Signed-off-by: Vishal Choudhary <[email protected]>

vishal-chdhry · 2023-07-10T17:51:35Z

@realshuting @JimBugwadia Tested in local env, using image: ghcr.io/vishal-chdhry/kyverno-notation-aws:verifier-lib

EDIT: Policy used: https://github.com/nirmata/kyverno-notation-aws/blob/464802557d0f3754becb004e537987ee9b3e2aa8/configs/samples/kyverno-policy.yaml

Created a policy with validate enabled
Pod was created
The image of the pod has tag and not digest
Updated the policy to use mutate
Pod was created and the digest of the pod was mutated
Created a pod with unsigned image and it was rejected

$ kubectl apply -f configs/samples/kyverno-policy.yaml
clusterpolicy.kyverno.io/check-images created

$ kubectl -n test-notation run test --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
pod/test created

$ kubectl describe pod test -n test-notation
Name:             test
Namespace:        test-notation
Priority:         0
Service Account:  default
Node:             ip-10-20-10-23.us-west-1.compute.internal/10.20.10.23
Start Time:       Mon, 10 Jul 2023 23:12:39 +0530
Labels:           run=test
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Running
IP:               10.20.10.174
IPs:
  IP:  10.20.10.174
Containers:
  test:
    Container ID:   containerd://2454a766c1ef21f7389136fef774055738ecee92ea375ad1a9ce9a48bb078745
    Image:          844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
    Image ID:       844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 10 Jul 2023 23:12:40 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ggqld (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-ggqld:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  8s    default-scheduler  Successfully assigned test-notation/test to ip-10-20-10-23.us-west-1.compute.internal
  Normal  Pulled     7s    kubelet            Container image "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1" already present on machine
  Normal  Created    7s    kubelet            Created container test
  Normal  Started    7s    kubelet            Started container test

$ kubectl delete cpol check-images
clusterpolicy.kyverno.io "check-images" deleted

$ kubectl delete pod test -n test-notation
pod "test" deleted

$ kubectl apply -f configs/samples/kyverno-policy.yaml
clusterpolicy.kyverno.io/check-images created

$ kubectl -n test-notation run test --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
pod/test created


$ kubectl describe pod test -n test-notation
Name:             test
Namespace:        test-notation
Priority:         0
Service Account:  default
Node:             ip-10-20-10-23.us-west-1.compute.internal/10.20.10.23
Start Time:       Mon, 10 Jul 2023 23:13:43 +0530
Labels:           run=test
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Running
IP:               10.20.10.132
IPs:
  IP:  10.20.10.132
Containers:
  test:
    Container ID:   containerd://adfb3df1c88802138d63721f30d8dd46d9b2e2cc27aa82e0748a66cb7301b11e
    Image:          844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
    Image ID:       844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 10 Jul 2023 23:13:44 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gpbrb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-gpbrb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  5s    default-scheduler  Successfully assigned test-notation/test to ip-10-20-10-23.us-west-1.compute.internal
  Normal  Pulled     5s    kubelet            Container image "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108" already present on machine
  Normal  Created    5s    kubelet            Created container test
  Normal  Started    4s    kubelet            Started container test

  
$ kubectl -n test-notation run test1 --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned --dry-run=server
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request: mutation policy check-images error: failed to apply policy check-images rules [call-aws-signer-extension: failed to evaluate list response.results: JMESPath query failed: Unknown key "response" in path: JMESPath query failed: Unknown key "response" in path]

$ kubectl logs -f deployment/kyverno-notation-aws -n kyverno-notation-aws
Defaulted container "kyverno-notation-aws" out of: kyverno-notation-aws, kube-notation
2023-07-10T17:41:39.890Z        INFO    setup/setup.go:23       configuring notation    {"dir.UserConfigDir": "/notation", "dir.UserLibexecDir": "/notation"}
2023-07-10T17:41:39.891Z        ERROR   verifier/verify.go:99   initialization error: trust policy is not present. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy
2023-07-10T17:41:40.444Z        ERROR   verifier/verify.go:99   initialization error: trust policy is not present. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy
2023-07-10T17:41:41.526Z        ERROR   verifier/verify.go:99   initialization error: trust policy is not present. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy
2023-07-10T17:41:42.837Z        ERROR   verifier/verify.go:99   initialization error: trust policy is not present. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy
2023-07-10T17:41:44.420Z        ERROR   verifier/verify.go:99   initialization error: trust policy is not present. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy
2023-07-10T17:41:46.763Z        ERROR   verifier/verify.go:99   initialization error: trust policy is not present. To create a trust policy, see: https://notaryproject.dev/docs/quickstart/#create-a-trust-policy
2023-07-10T17:41:51.273Z        INFO    verifier/verify.go:112  initialized     {"namespace": "kyverno-notation-aws", "secrets": "", "insecureRegistry": false}
2023-07-10T17:41:51.273Z        INFO    /main.go:71     Listening...
2023-07-10T17:42:37.408Z        INFO    verifier/verify.go:171  verifying image infos 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
2023-07-10T17:42:37.408Z        INFO    verifier/verify.go:188  verifying image 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
2023-07-10T17:42:39.266Z        INFO    verifier/verify.go:140  verified map[%!d(string=test):{{%!d(string=844333597536.dkr.ecr.us-west-2.amazonaws.com) %!d(string=kyverno-demo) %!d(string=kyverno-demo) %!d(string=v1) %!d(string=)} %!d(string=/spec/containers/0/image)}] containers 
2023-07-10T17:42:39.266Z        INFO    verifier/verify.go:149  verified map[] initContainers
2023-07-10T17:42:39.266Z        INFO    verifier/verify.go:158  verified map[] ephemeralContainers
2023-07-10T17:43:41.427Z        INFO    verifier/verify.go:171  verifying image infos 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
2023-07-10T17:43:41.427Z        INFO    verifier/verify.go:188  verifying image 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1
2023-07-10T17:43:43.150Z        INFO    verifier/verify.go:140  verified map[%!d(string=test):{{%!d(string=844333597536.dkr.ecr.us-west-2.amazonaws.com) %!d(string=kyverno-demo) %!d(string=kyverno-demo) %!d(string=v1) %!d(string=)} %!d(string=/spec/containers/0/image)}] containers 
2023-07-10T17:43:43.150Z        INFO    verifier/verify.go:149  verified map[] initContainers
2023-07-10T17:43:43.150Z        INFO    verifier/verify.go:158  verified map[] ephemeralContainers
2023-07-10T17:44:20.421Z        INFO    verifier/verify.go:171  verifying image infos 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned
2023-07-10T17:44:20.421Z        INFO    verifier/verify.go:188  verifying image 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned
2023-07-10T17:44:21.205Z        ERROR   verifier/verify.go:174  verification failed for image 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned: no signature is associated with "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5", make sure the artifact was signed successfully

realshuting · 2023-07-11T02:37:29Z

@realshuting @JimBugwadia Tested in local env, using image: ghcr.io/vishal-chdhry/kyverno-notation-aws:verifier-lib

Created a policy with validate enabled

Pod was created

The image of the pod has tag and not digest

Updated the policy to use mutate

Pod was created and the digest of the pod was mutated

Created a pod with unsigned image and it was rejected

Thanks Vishal, this looks good.

Can you also attach policy manifests for reference?

Regarding 6, do we not have any response for unsigned images? Can we improve this?

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request: mutation policy check-images error: failed to apply policy check-images rules [call-aws-signer-extension: failed to evaluate list response.results: JMESPath query failed: Unknown key "response" in path: JMESPath query failed: Unknown key "response" in path]

vishal-chdhry · 2023-07-11T08:38:05Z

@realshuting updated the error message to this

$ kubectl -n test-notation run test --image=844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned --dry-run=server
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/test-notation/test was blocked due to the following policies 

check-images:
  call-aws-signer-extension: 'failed to verify container kyverno-demo: failed to verify
    image 844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo:v1-unsigned: no
    signature is associated with "844333597536.dkr.ecr.us-west-2.amazonaws.com/kyverno-demo@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5",
    make sure the artifact was signed successfully'

Signed-off-by: Vishal Choudhary <[email protected]>

realshuting

This PR looks good.

Is this the response structure? Can you clarify how the response is built, for failure/success scenario respectively?

vishal-chdhry · 2023-07-18T08:36:02Z

@realshuting I have updated the response structure on main, the library will be updated with the next release of verifier library

Signed-off-by: Vishal Choudhary <[email protected]>

realshuting · 2023-07-20T08:43:47Z

@realshuting I have updated the response structure on main, the library will be updated with the next release of verifier library

Great! Is there any pending item?

vishal-chdhry · 2023-07-20T08:51:40Z

@realshuting Nope, I just updated everything and created a new image with improved logging, validation and comments ghcr.io/vishal-chdhry/kyverno-notation-aws:v1-rc3

removed common logic

c665cb4

Signed-off-by: Vishal Choudhary <[email protected]>

vishal-chdhry requested a review from JimBugwadia June 24, 2023 10:02

vishal-chdhry added 3 commits June 28, 2023 18:22

Merge branch 'main' into common-library

60fc8f7

Signed-off-by: Vishal Choudhary <[email protected]>

chore: update kyverno-notation-verifier to v0.3.1

eeb065c

Signed-off-by: Vishal Choudhary <[email protected]>

fix: update kyverno policy

4648025

Signed-off-by: Vishal Choudhary <[email protected]>

vishal-chdhry marked this pull request as ready for review July 10, 2023 17:51

vishal-chdhry mentioned this pull request Jul 10, 2023

Updated policy to using images variable and mutate patches #6

Closed

vishal-chdhry requested a review from realshuting July 10, 2023 17:56

chore: update verifier to v0.3.2

ff79e28

Signed-off-by: Vishal Choudhary <[email protected]>

realshuting approved these changes Jul 17, 2023

View reviewed changes

merge: branch 'main' into common-library

faaaef0

chore: update verifier to v0.3.3

5e2eb3f

Signed-off-by: Vishal Choudhary <[email protected]>

realshuting merged commit c394e43 into main Jul 20, 2023
2 checks passed

realshuting deleted the common-library branch August 9, 2023 05:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extracted common logic to nirmata/kyverno-notation-verifier #7

Extracted common logic to nirmata/kyverno-notation-verifier #7

vishal-chdhry commented Jun 22, 2023 •

edited

Loading

vishal-chdhry commented Jul 10, 2023 •

edited

Loading

realshuting commented Jul 11, 2023 •

edited

Loading

vishal-chdhry commented Jul 11, 2023

realshuting left a comment

vishal-chdhry commented Jul 18, 2023

realshuting commented Jul 20, 2023

vishal-chdhry commented Jul 20, 2023

Extracted common logic to nirmata/kyverno-notation-verifier #7

Extracted common logic to nirmata/kyverno-notation-verifier #7

Conversation

vishal-chdhry commented Jun 22, 2023 • edited Loading

vishal-chdhry commented Jul 10, 2023 • edited Loading

realshuting commented Jul 11, 2023 • edited Loading

vishal-chdhry commented Jul 11, 2023

realshuting left a comment

Choose a reason for hiding this comment

vishal-chdhry commented Jul 18, 2023

realshuting commented Jul 20, 2023

vishal-chdhry commented Jul 20, 2023

vishal-chdhry commented Jun 22, 2023 •

edited

Loading

vishal-chdhry commented Jul 10, 2023 •

edited

Loading

realshuting commented Jul 11, 2023 •

edited

Loading