BOSH: running platforms so you can run platforms on your platforms Paul Czarkowski / @pczarkowski Principal Technologist Pivotal software © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0

Sucessful Patterns for Deploying [and Operating] Platforms. Paul Czarkowski / @pczarkowski Principal Technologist Pivotal software © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0

Do you know Cloud Foundry?

Do you know Kubernetes ?

Do you know BOSH ?

What is a platform ? https://en.wikipedia.org/wiki/Computing_platform

A modern software platform provides API driven compute resources.

​ ​ ​“Every IT team ​is a ​Platform Team.” ​

http://slides.eightypercent.net/platform-platform/index.html#1

https://twitter.com/kelseyhightower/status/935252923721793536

Generic Platform Users API Artifacts Database Storage Compute Network Access

Enterprise IT Users API Storage Admin DBA Systems Admin Network Engineer Security QA

“You don’t have a platform problem, you have a culture problem.”

Evolve your IT teams! Take business requirements and turn them into features Application Team App1 Messaging Build common services for App Teams Platform Team App Platform ML ? App2 Creds/Certs App3 Middleware ELK Container Services DBaaS Container Hosts | Kubernetes ? Infrastructure Team Abstract infrastructure complexity with easy consumption IaaS Infra Services Virtual Infrastructure Physical Infrastructure ?

People build Platforms People build Apps Apps run on Platforms People are the most important component of any platform.

What Abstraction is the Right One for You ? More Control Traditional Ticket Based Human Toil Less Complexity Build App Artifact Build App Container(s) App → to the Platform Config Management Build Application Dependencies CF API Infrastructure As Code IaaS API PXE boot ? PaaS PaaS Application Platform K8s API CaaS Container CaaS Orchestrator Container Runtime IaaSHosts Container IaaS Hardware 18

FaaS PaaS Higher flexibility and less enforcement of standards CaaS IaaS Hardware Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Lower development complexity and higher operational efficiency

Consumers of the abstraction App Team Platform Team Infrastructure Team FaaS PaaS Platform Team Infrastructure Team App Team CaaS IaaS Infrastructure Team App Team Platform Team Hardware Infrastructure Team Platform Team

​“In general, taking something that’s already working somewhere and expanding its usage (capabilities) is far more likely to succeed than building these capabilities from scratch”

WE build the container YOU build the container Application Code & Frameworks Continuous delivery Github Pivotal Network Concourse Pivotal Application Service (PAS) Pivotal Container Service (PKS)

cf push Java | .NET | NodeJS Marketplace Pivotal and Partner Products kubectl run v1 CVEs Product Updates Pivotal Services Open Service Broker API Buildpacks | Spring Boot | Spring Cloud | Steeltoe Public Cloud Services Customer Managed Services Elastic | Packaged Software | Spark v2 “3Rs” v3 Embedded OS ... NSX-T (Windows & Linux) Concourse CPI (15 methods) vSphere Openstack AWS Google Cloud Azure & Azure Stack Repair — CVEs Repave Rotate — Credhub

The Google Problem x 1,000,000

K8s Contr Master oller Kube Mana Scheduler ger API Server K8s Worker C Docker Kube-pro xy CNI Pod Pod Pod K8s Worker K8s Worker Kubelet Kubelet Etcd Kubelet Pod Pod Pod Docker Kube-pro xy CNI Pod Pod Pod Docker Kube-pro xy CNI

K8s Master Google Compute Engine (hidden from user) Controller Manager Etcd Kube Scheduler API Server Google Compute Engine K8s Worker Kubelet Kubelet Docker K8s Worker K8s Worker C Pod Pod Pod Kubelet Pod Pod Pod Docker Kube-proxy Kube-proxy CNI Pod Pod Pod Docker Kube-proxy CNI CNI

What Abstraction is the Right One for You ? More Control Traditional Ticket Based Human Toil Less Complexity Build App Artifact Build App Container(s) App → to the Platform Config Management Build Application Dependencies CF API Infrastructure As Code IaaS API PXE boot ? PaaS PaaS Application Platform K8s API CaaS Container CaaS Orchestrator Container Runtime IaaSHosts Container IaaS Hardware 32

BOSH - Component Architecture Cloud Provider Interface IaaS CLI IaaS API Registry BOSH provides the means to go from deployment configuration to VM creation and management. It includes interfaces for Azure, vSphere, AWS, GCP, and OpenStack. Additional CPI can be written for alternative IaaS providers. Director Health Monitor Ag NATS Postgres DB Blob Store en t Ag en t en tA ge nt Ag

BOSH - Service Deployment Deployment Manifest Release w Stemcell

BOSH - Stemcell ▪ ▪ ▪ Stemcell ▪ ▪ ▪ ▪ Secured, Hardened, and Versioned Operating System image wrapped with IaaS specific packaging Contains a bare minimum OS skeleton with a few common utilities pre-installed, a BOSH Agent, and a few configuration files to securely configure the OS by default. Images come in two flavors Ubuntu 14.04 and CentOS 7 for all IaaS’ supported Maintained by BOSH team and available at http://bosh.io/stemcells

BOSH - Release Elements: Jobs - Pieces of the service or application you are releasing, including how to compile & run them Packages - Provide source code and dependencies to jobs Organization: Dependency Tree: Job(s) Monit Spec Src - Non-binary files which is provided to packages Blobs - Provide binary files (other than those checked into a source code repository) to packages Monit - Script utilized to start/stop/restart the job Packaging - Script utilized to compile the source needed by a job Spec - Key/Value file which stores all configuration properties which can be set externally Green - Script (erb or bash) Orange - Properties (yml) Blobs Package(s) Packaging Spec Src

BOSH - Manifest Provides the ability to customize BOSH releases (your service) YAML - Primer found at end of presentation Required Blocks: ▪ Deployment Identification: A name for the deployment and the UUID of the Director managing the deployment ▪ Releases Block: Name and version of each release in a deployment ▪ Stemcells Block: Name and version of each stemcell in a deployment ▪ Update Block: Defines how BOSH updates instances during deployment ▪ Instance Groups Block: Configuration and resource information for instance groups (Jobs) ▪ Properties Block: Describes global properties and generalized configuration information ▪

BOSH - Service Deployment Cloud Provider Interface CLI IaaS API Stemcell Registry Manifest Director Release Health Monitor NATS Postgres DB * * Blob Store Config Ag Aegnt en t Ag en t A ge nt

BOSH - Process High Availability IaaS Alert Sent! Director Health Monitor NATS Pro c-1 Restart! Ag ent Pro Ag e nt c-1

BOSH - VM High Availability Manifest - Desired State Cloud Provider Interface IaaS IaaS API Alert Sent! Director Health Monitor NATS c-1 Pro Ag ent c-1 Pro Ag ent c-1 Pro Ag e nt

-1 c Pro Ag Ag ent ent -1 c Pro

BOSH - Canary Upgrades Manifest - Any update error causes the deployment to stop. Since only canaries are affected before an update stops, problem jobs and packages are prevented from taking over all instances. Code Update 4.04! Ag en t Ag en Ag t en t Ag en t

BOSH - Day 2 Ops Consistent, Reliable, Scalable, Secure Deploy and Configure Provision Current State Desired State Re-mediate Monitor and Detect ▪ ▪ Checks against “desired state to return consistency No ad hoc automation burden Manage services, not servers ▪ 4 layers of Self Healing ▪

PKS “How it Works” #pks K8s-1 K8s-2 -nsmall 3 # pkscreate-cluster create-clusterK8s-3 --plan Cluster3 The value of BOSH API Node V M Node V M Node V M Node V M Node V M Node V M API Node V M Kubernetes Cluster Services Node V V M M Cluster3 Node Node V V M M Node V M Node V M API PKS Control Plane Kubernetes Cluster Services PKS includes: Cluster1 • PKS Control Plane, CFCR • NSX-T, Harbor, GCP Broker Kubernetes • BOSH Release for Kubernetes (As a Bosh Release) • Configures Day 1 of - • CFCR vSphere NSX Integration Harbor Manages Day 2 of Kubernetes Clusters - Scaling Patching Upgrades Failures CNI NSX-T API Kubernetes Cluster Services BOSH CPI (Deploys/Manages VMs) Kubo CFCR Harbor GCP Private Container Registry Service Broker vSphere IaaS Cluster2

AWS Lambda, Azure Functions, OpenWhisk, kubeless, PFS FaaS PaaS PKS, GKE, OpenShift, AWS Fargate, Kubernetes CaaS IaaS HPE, Dell, IBM, Lenovo PCF, Azure App Service, Heroku, Deis Hardware AWS, Azure, GCP, Openstack, VMWare

http://www.bsielearning.com.au/keep-simple-stupid/

Full Opensource DIY ● ● ● ● ??? pxe Openstack Ansible Kubespray Ansible Hardening https://github.com/openstack/openstack-ansible https://github.com/openstack/ansible-hardening https://github.com/kubernetes-incubator/kubespray

!!!

How We Think about the Business Case SPEED & AGILITY -90%* Time to Scale SCALABILITY STABILITY $ Millions Annual Savings on HW, SW and Support SAVINGS PLATFORM VALUE STREAM AND METRICS $ SECURITY Fewer Support Incidents REPLATFORM > MODERNIZE > OPTIMIZE % Faster Patching Delivery @ Zero Downtime 25-50%* More Projects With Same Staff $ 40%* 40-60%* ESTABLISH, MEASURE AND UPDATE KEY OBJECTIVES AND RESULTS (OKRs)

THE END Questions ?