Hacking Helm
A presentation at Kubecon EU by Paul Czarkowski
Hacking Helm @pczarkowski @r6by
hello! Paul Czarkowski @pczarkowski pczarkowski@pivotal.io Scott Rigby @r6by scott@r6by.com 2
Agenda ✘ ✘ ✘ ✘ What is Helm ? Security Concerns Lock down Tiller Secure Values 3 @pczarkowski @r6by
�� What Is Helm? 4 @pczarkowski @r6by
“ Helm is the best way to find, share, and use software built for Kubernetes 5 @pczarkowski @r6by
Components ✘ ✘ ✘ ✘ Helm Client Helm Chart [package] Chart Repository Tiller Server 6 @pczarkowski @r6by
- Helm Client 7 @pczarkowski @r6by
$ helm create $ helm install $ helm repository 8 @pczarkowski @r6by
- Helm Chart [Package] 9 @pczarkowski @r6by
services db custom ci load balancer Chart.yaml Values.yaml templates/ 10 @pczarkowski @r6by
- Helm Repository 11 @pczarkowski @r6by
A Helm repository is just a web server. $ ls /var/www/html index.yaml chart-0.0.1.tgz chart-0.0.2.tgz chart-0.0.3.tgz https://github.com/helm/chart-releaser 12 @pczarkowski @r6by
Secure | https://hub.helm.sh Discover & launch great Kubernetes-ready apps Search charts Wordpress, Jenkins, Kubeless… 666 charts ready to deploy 🤘 13 @pczarkowski @r6by
- Tiller Server 14 @pczarkowski @r6by
Tiller server runs in the Kubernetes Cluster Kubernetes Cluster Tiller 15 @pczarkowski @r6by
$ helm init Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. $ k get deployments NAME READY UP-TO-DATE AVAILABLE tiller-deploy 1/1 1 63s 16 AGE 1 @pczarkowski @r6by
�� Helm Chart Workflow 17 @pczarkowski @r6by
Helm Repository Kubernetes Cluster Helm Client Tiller 18 @pczarkowski @r6by
Helm Repository Kubernetes Cluster Helm Client Tiller 19 @pczarkowski @r6by
Helm Repository Kubernetes Cluster Helm Client Tiller 20 @pczarkowski @r6by
Helm Repository Kubernetes Cluster Helm Client Deployment Service Tiller 21 @pczarkowski @r6by
�� Security Concerns 22 @pczarkowski @r6by
Helm Repository Kubernetes Cluster Helm Client Deployment Service Tiller 23 @pczarkowski @r6by
- Tenancy 24 @pczarkowski @r6by
k8s Tenancy Models Model Tenancy Everyone is Cluster Admin none Every User has their own NS soft Each Team/Project have a shared NS soft Each Team has their own Cluster hard 25 @pczarkowski @r6by
k8s Tenancy Models Model Tenancy Everyone is Cluster Admin none Every User has their own NS soft Each Team/Project have a shared NS soft Each Team has their own Cluster hard 26 @pczarkowski @r6by
k8s Tenancy Models Model Tenancy Everyone is Cluster Admin none Every User has their own NS soft Each Team/Project have a shared NS soft Each Team has their own Cluster hard 27 @pczarkowski @r6by
k8s Tenancy Models Model Tenancy Everyone is Cluster Admin none Every User has their own NS soft Each Team/Project have a shared NS soft Each Team has their own Cluster hard 28 @pczarkowski @r6by
- RBAC vs ABAC 29 @pczarkowski @r6by
ABAC ✘ Don’t use ABAC 30 @pczarkowski @r6by
RBAC ✘ Service Accounts - for “robots” ✘ User Accounts - for “humans” 31 @pczarkowski @r6by
RBAC ✘ Service Accounts - for “robots” ✘ User Accounts - for “humans” 32 @pczarkowski @r6by
RBAC ✘ Service Accounts - for “robots” ✘ User Accounts - for “humans” 33 @pczarkowski @r6by
- Lock Down Tiller 34 @pczarkowski @r6by
Tiller Attack Vectors ✘ Privilege Escalation ✘ In Cluster ✘ Naughty Chart Author 35 @pczarkowski @r6by
Tiller Attack Vectors ✘ Privilege Escalation ✘ In Cluster ✘ Naughty Chart Author 36 @pczarkowski @r6by
$ kubectl -n kube-system create \ serviceaccount tiller $ kubectl create clusterrolebinding tiller \ —clusterrole cluster-admin \ —serviceaccount=kube-system:tiller $ helm init —service-account=tiller 37 @pczarkowski @r6by
$ kubectl -n paul create \ serviceaccount tiller $ kubectl -n paul create role tiller \ —verb ‘*’ \ —resources=’services,deployments,…’ $ kubectl -n paul create rolebinding \ —role tiller —service-account=paul:tiller $ helm init —service-account=tiller —tiller-namespace=paul 38 @pczarkowski @r6by
Tiller Attack Vectors ✘ Privilege Escalation ✘ In Cluster ✘ Naughty Chart Author 39 @pczarkowski @r6by
$ kubectl -n kube-system get service tiller-deploy NAME tiller-deploy TYPE ClusterIP CLUSTER-IP 10.110.112.63 40 EXTERNAL-IP <none> PORT(S) 44134/TCP AGE 20h @pczarkowski @r6by
$ kubectl run -n default —quiet —rm —restart=Never \ -ti —image=alpine/helm helm —command — /bin/sh helm# helm —host tiller-deploy.kube-system:44134 version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Server: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} helm# helm —host tiller-deploy.kube-system:44134 install bitcoin 41 @pczarkowski @r6by
$ kubectl -n kube-system delete service tiller-deploy service “tiller-deploy” deleted helm# helm —host tiller-deploy.kube-system:44134 version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Timed out. Helm# helm —host 172.17.0.4:44134 version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Server: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} 42 @pczarkowski @r6by
$ kubectl -n kube-system patch deployment \ tiller-deploy —patch ’ spec: template: spec: containers: - name: tiller ports: [] command: [“/tiller”] args: [“—listen=localhost:44134”] ’ 43 @pczarkowski @r6by
Helm# helm —host 172.17.0.4:44134 version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Timed out. 44 @pczarkowski @r6by
? But how do my Helm commands Still work 45 @pczarkowski @r6by
? What if I need Helm access from Inside my cluster? 46 @pczarkowski @r6by
$ helm init —tiller-tls —tiller-tls-cert ./tiller.crt \ —tiller-tls-key ./tiller.key —tiller-tls-verify \ —tls-ca-cert ca.crt $ helm —tls —tls-ca-cert ca.crt —tls-cert myclient.crt \ —tls-key myclient.key version Client: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} Server: &version.Version{SemVer:”v2.13.1”, GitCommit:”618447cbf203d147601b4b9bd7f8c37a5d39fbb4”, GitTreeState:”clean”} 47 @pczarkowski @r6by
Tiller Attack Vectors ✘ Privilege Escalation ✘ In Cluster ✘ Naughty Chart Author 48 @pczarkowski @r6by
? Helm Install is the new Curl Bash 49 @pczarkowski @r6by
Naughty Chart Authors ✘ PodSecurityPolicy ✘ AdmissionController Webhooks ( Open Policy Agent! ) ✘ ConfTest - shift OPA left ○ https://github.com/instrumenta/conftest 50 @pczarkowski @r6by
Just Don’t Install Tiller! Can’t Hack What Doesn’t Exist 51 @pczarkowski @r6by
$ helm template -f prod.yaml stable/wordpress > deploy.yaml $ kubectl apply -f deploy.yaml 52 @pczarkowski @r6by
$ helm tiller start my-team-namespace $ helm install … $ helm tiller stop https://github.com/rimusz/helm-tiller 53 @pczarkowski @r6by
- Secrets & Value Leakage 54 @pczarkowski @r6by
Leaky Value Syndrome 😭 ✘ Helm values leak… ✘ Version control is forever! ✘ Tips for using k8s secrets with Helm 55 @pczarkowski @r6by
$ helm install stable/grafana —name grafana \ —namespace grafana —set “adminPassword=z3roC00l!” $ kubectl -n kube-system get configmap grafana.v1 -o yaml 56 @pczarkowski @r6by
$ for i in $(helm ls | awk ‘{print $1}’ | tail -n +2) do helm get values $i | grep -i pass done adminPassword: dsadRGR password: flkdjsfdsk3r3r3 harborAdminPassword: sadkljejfewreDDf33 57 @pczarkowski @r6by
$ git clone http://github.com/e-corp/infrastructure-as-code $ cd infrastructure-as-code/envs/production $ cat values.yaml | grep -i pass 58 @pczarkowski @r6by
How to Handle Secrets then ? ✘ Encrypt them before pushing to github ○ Sealed Secrets ( bitnami tool ) ✘ Store them in a separate encrypted volume ○ I’ve even seen a shared keepass/onepass used for this ✘ YOLO them into git 59 @pczarkowski @r6by
HELM 3 60 @pczarkowski @r6by
�� Forget Everything We Just Told you. 61 @pczarkowski @r6by
Summary ✘ ✘ ✘ ✘ ✘ ✘ First know your tenancy model Secure your tiller install based on tenancy Tiller is “insecure” by default We still have a “how to keep secrets secret” problem Helm is awesome, and can used securely! Helm 3 will might be more secure. 62 @pczarkowski @r6by
References and Further Reading ✘ ✘ ✘ ✘ ✘ ✘ https://hub.helm.sh https://engineering.bitnami.com/articles/helm-security.html https://github.com/instrumenta/conftest https://github.com/helm/chart-releaser https://github.com/helm/chart-testing https://github.com/rimusz/helm-tiller 63 @pczarkowski @r6by
thanks! Any questions? You can find us at Paul Czarkowski @pczarkowski pczarkowski@pivotal.io Scott Rigby @r6by scott@r6by.com 64
Helm is the best way to build, package, and run Kubernetes manifests. However it has been considered by some as a fairly insecure way to deploy software, mostly due to its server component Tiller. Let’s put that to the test and hack (and then protect from those hacks) Helm.
Paul and Scott will talk about the architecture behind Helm (v2) and how it is seen as particularly vulnerable to hacks designed to either gather information about deployed applications and even access the kubernetes cluster itself. They will then demonstrate several hacks in growing sophistication that do exactly that and then show how to protect yourself from those attacks. They will finish with a summary of how you can use Helm as securely as possible.
for free. You
can too.
