Hacking Helm

A presentation at Kubecon EU in May 2019 in by Paul Czarkowski

Slide 1

Hacking Helm @pczarkowski @r6by

Slide 2

hello! Paul Czarkowski @pczarkowski pczarkowski@pivotal.io Scott Rigby @r6by scott@r6by.com 2

Slide 3

Agenda ✘ ✘ ✘ ✘ What is Helm ? Security Concerns Lock down Tiller Secure Values 3 @pczarkowski @r6by

Slide 4

�� What Is Helm? 4 @pczarkowski @r6by

Slide 5

“ Helm is the best way to find, share, and use software built for Kubernetes 5 @pczarkowski @r6by

Slide 6

Components ✘ ✘ ✘ ✘ Helm Client Helm Chart [package] Chart Repository Tiller Server 6 @pczarkowski @r6by

Slide 7

Helm Client 7 @pczarkowski @r6by

Slide 8

$ helm create $ helm install $ helm repository 8 @pczarkowski @r6by

Slide 9

Helm Chart [Package] 9 @pczarkowski @r6by

Slide 10

services db custom ci load balancer Chart.yaml Values.yaml templates/ 10 @pczarkowski @r6by

Slide 11

Helm Repository 11 @pczarkowski @r6by

Slide 12

A Helm repository is just a web server. $ ls /var/www/html index.yaml chart-0.0.1.tgz chart-0.0.2.tgz chart-0.0.3.tgz https://github.com/helm/chart-releaser 12 @pczarkowski @r6by

Slide 13

Secure | https://hub.helm.sh Discover & launch great Kubernetes-ready apps Search charts Wordpress, Jenkins, Kubeless… 666 charts ready to deploy 🤘 13 @pczarkowski @r6by

Slide 14

Tiller Server 14 @pczarkowski @r6by

Slide 15

Tiller server runs in the Kubernetes Cluster Kubernetes Cluster Tiller 15 @pczarkowski @r6by

Slide 16

$ helm init Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. $ k get deployments NAME READY UP-TO-DATE AVAILABLE tiller-deploy 1/1 1 63s 16 AGE 1 @pczarkowski @r6by

Slide 17

�� Helm Chart Workflow 17 @pczarkowski @r6by

Slide 18

Helm Repository Kubernetes Cluster Helm Client Tiller 18 @pczarkowski @r6by

Slide 19

Helm Repository Kubernetes Cluster Helm Client Tiller 19 @pczarkowski @r6by

Slide 20

Helm Repository Kubernetes Cluster Helm Client Tiller 20 @pczarkowski @r6by

Slide 21

Helm Repository Kubernetes Cluster Helm Client Deployment Service Tiller 21 @pczarkowski @r6by

Slide 22

�� Security Concerns 22 @pczarkowski @r6by

Slide 23

Helm Repository Kubernetes Cluster Helm Client Deployment Service Tiller 23 @pczarkowski @r6by

Slide 24

Tenancy 24 @pczarkowski @r6by

Slide 25

k8s Tenancy Models Model Tenancy Everyone is Cluster Admin none Every User has their own NS soft Each Team/Project have a shared NS soft Each Team has their own Cluster hard 25 @pczarkowski @r6by

Slide 26

k8s Tenancy Models Model Tenancy Everyone is Cluster Admin none Every User has their own NS soft Each Team/Project have a shared NS soft Each Team has their own Cluster hard 26 @pczarkowski @r6by

Slide 27

k8s Tenancy Models Model Tenancy Everyone is Cluster Admin none Every User has their own NS soft Each Team/Project have a shared NS soft Each Team has their own Cluster hard 27 @pczarkowski @r6by

Slide 28

k8s Tenancy Models Model Tenancy Everyone is Cluster Admin none Every User has their own NS soft Each Team/Project have a shared NS soft Each Team has their own Cluster hard 28 @pczarkowski @r6by

Slide 29

RBAC vs ABAC 29 @pczarkowski @r6by

Slide 30

ABAC ✘ Don’t use ABAC 30 @pczarkowski @r6by

Slide 31

RBAC ✘ Service Accounts - for “robots” ✘ User Accounts - for “humans” 31 @pczarkowski @r6by

Slide 32

RBAC ✘ Service Accounts - for “robots” ✘ User Accounts - for “humans” 32 @pczarkowski @r6by

Slide 33

RBAC ✘ Service Accounts - for “robots” ✘ User Accounts - for “humans” 33 @pczarkowski @r6by

Slide 34

Lock Down Tiller 34 @pczarkowski @r6by

Slide 35

Tiller Attack Vectors ✘ Privilege Escalation ✘ In Cluster ✘ Naughty Chart Author 35 @pczarkowski @r6by

Slide 36

Tiller Attack Vectors ✘ Privilege Escalation ✘ In Cluster ✘ Naughty Chart Author 36 @pczarkowski @r6by

Slide 37

$ kubectl -n kube-system create \ serviceaccount tiller $ kubectl create clusterrolebinding tiller \ —clusterrole cluster-admin \ —serviceaccount=kube-system:tiller $ helm init —service-account=tiller 37 @pczarkowski @r6by

Slide 38

$ kubectl -n paul create \ serviceaccount tiller $ kubectl -n paul create role tiller \ —verb ‘*’ \ —resources=’services,deployments,…’ $ kubectl -n paul create rolebinding \ —role tiller —service-account=paul:tiller $ helm init —service-account=tiller —tiller-namespace=paul 38 @pczarkowski @r6by

Slide 39

Tiller Attack Vectors ✘ Privilege Escalation ✘ In Cluster ✘ Naughty Chart Author 39 @pczarkowski @r6by

Slide 40

$ kubectl -n kube-system get service tiller-deploy NAME tiller-deploy TYPE ClusterIP CLUSTER-IP 10.110.112.63 40 EXTERNAL-IP <none> PORT(S) 44134/TCP AGE 20h @pczarkowski @r6by

Slide 41

$ kubectl run -n default —quiet —rm —restart=Never \ -ti —image=alpine/helm helm —command — /bin/sh helm# helm —host tiller-deploy.kube-system:44134 version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Server: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} helm# helm —host tiller-deploy.kube-system:44134 install bitcoin 41 @pczarkowski @r6by

Slide 42

$ kubectl -n kube-system delete service tiller-deploy service “tiller-deploy” deleted helm# helm —host tiller-deploy.kube-system:44134 version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Timed out. Helm# helm —host 172.17.0.4:44134 version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Server: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} 42 @pczarkowski @r6by

Slide 43

$ kubectl -n kube-system patch deployment \ tiller-deploy —patch ’ spec: template: spec: containers: - name: tiller ports: [] command: [“/tiller”] args: [“—listen=localhost:44134”] ’ 43 @pczarkowski @r6by

Slide 44

Helm# helm —host 172.17.0.4:44134 version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Timed out. 44 @pczarkowski @r6by

Slide 45

? But how do my Helm commands Still work 45 @pczarkowski @r6by

Slide 46

? What if I need Helm access from Inside my cluster? 46 @pczarkowski @r6by

Slide 47

$ helm init —tiller-tls —tiller-tls-cert ./tiller.crt \ —tiller-tls-key ./tiller.key —tiller-tls-verify \ —tls-ca-cert ca.crt $ helm —tls —tls-ca-cert ca.crt —tls-cert myclient.crt \ —tls-key myclient.key version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Server: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} 47 @pczarkowski @r6by

Slide 48

Tiller Attack Vectors ✘ Privilege Escalation ✘ In Cluster ✘ Naughty Chart Author 48 @pczarkowski @r6by

Slide 49

? Helm Install is the new Curl Bash 49 @pczarkowski @r6by

Slide 50

Naughty Chart Authors ✘ PodSecurityPolicy ✘ AdmissionController Webhooks ( Open Policy Agent! ) ✘ ConfTest - shift OPA left ○ https://github.com/instrumenta/conftest 50 @pczarkowski @r6by

Slide 51

Just Don’t Install Tiller! Can’t Hack What Doesn’t Exist 51 @pczarkowski @r6by

Slide 52

$ helm template -f prod.yaml stable/wordpress > deploy.yaml $ kubectl apply -f deploy.yaml 52 @pczarkowski @r6by

Slide 53

$ helm tiller start my-team-namespace $ helm install … $ helm tiller stop https://github.com/rimusz/helm-tiller 53 @pczarkowski @r6by

Slide 54

Secrets & Value Leakage 54 @pczarkowski @r6by

Slide 55

Leaky Value Syndrome 😭 ✘ Helm values leak… ✘ Version control is forever! ✘ Tips for using k8s secrets with Helm 55 @pczarkowski @r6by

Slide 56

$ helm install stable/grafana —name grafana \ —namespace grafana —set “adminPassword=z3roC00l!” $ kubectl -n kube-system get configmap grafana.v1 -o yaml 56 @pczarkowski @r6by

Slide 57

$ for i in $(helm ls | awk ‘{print $1}’ | tail -n +2) do helm get values $i | grep -i pass done adminPassword: dsadRGR password: flkdjsfdsk3r3r3 harborAdminPassword: sadkljejfewreDDf33 57 @pczarkowski @r6by

Slide 58

$ git clone http://github.com/e-corp/infrastructure-as-code $ cd infrastructure-as-code/envs/production $ cat values.yaml | grep -i pass 58 @pczarkowski @r6by

Slide 59

How to Handle Secrets then ? ✘ Encrypt them before pushing to github ○ Sealed Secrets ( bitnami tool ) ✘ Store them in a separate encrypted volume ○ I’ve even seen a shared keepass/onepass used for this ✘ YOLO them into git 59 @pczarkowski @r6by

Slide 60

HELM 3 60 @pczarkowski @r6by

Slide 61

�� Forget Everything We Just Told you. 61 @pczarkowski @r6by

Slide 62

Summary ✘ ✘ ✘ ✘ ✘ ✘ First know your tenancy model Secure your tiller install based on tenancy Tiller is “insecure” by default We still have a “how to keep secrets secret” problem Helm is awesome, and can used securely! Helm 3 will might be more secure. 62 @pczarkowski @r6by

Slide 63

References and Further Reading ✘ ✘ ✘ ✘ ✘ ✘ https://hub.helm.sh https://engineering.bitnami.com/articles/helm-security.html https://github.com/instrumenta/conftest https://github.com/helm/chart-releaser https://github.com/helm/chart-testing https://github.com/rimusz/helm-tiller 63 @pczarkowski @r6by

Slide 64

thanks! Any questions? You can find us at Paul Czarkowski @pczarkowski pczarkowski@pivotal.io Scott Rigby @r6by scott@r6by.com 64