Skip to main content

CockroachDB on Kubernetes | Kubernetes Deployment Explained | Multi-Region Kubernetes

CockroachDB chose to run CockroachCloud, our database as a service, on Kubermetes. It was not an easy decision for a lot of reasons and this video will expain how we deploy Kubernetes across single and multiple regions. 00:00 1:28 The origin of CockroachCloud 4:10 Kubernetes vs Borg 5:40 Kubernetes Skepticism (the networking stack is scary!) 6:50 Fitting Kubernetes into our use case 7:50 Powerful automation primitives of Kubernetes 9:13 Single region Kubernetes vs Multi-Region Kubernetes 14:08 Why CockroachDB chose Kubernetes Let’s summarize the situation. Google has used container orchestration for a long time and is very successful with the technology. There are two main benefits: 1. Efficient use of compute, when you are running a suite of heterogeneous services, via binpacking. 2. Solid automation primitives, which allow software engineers and SREs to think in terms of applications, not machines. 1 doesn’t benefit Cockroach Labs at all. "2" benefits Cockroach Labs a lot. On the other hand, Kubernetes is complicated, increasing cognitive load. Specifically, the networking stack is the main piece of complexity the SRE team worries about. What to do? We feel right now that the benefits of using Kubernetes outweigh the complexity costs. By using Kubernetes, we hope to create a highly orchestrated managed service, which will free us up to do interesting work on the reliability and efficiency of CockroachDB itself. There are also some benefits to Kubernetes that we haven’t covered in detail here, such as the fact that it provides a cross-cloud abstraction (Managed CockroachDB is already available for GCP and AWS), and the fact that many of our existing enterprise customers (not Managed CockroachDB customers) use it to deploy CockroachDB to their private datacenters. We are building out Managed CockroachDB on Kubernetes now and plan to launch it later this year. The SRE team at Cockroach Labs has an open headcount. If working at a startup that builds a super cool distributed database interests you, consider working with us! A parting thought Why is Kubernetes networking so complicated? One reason is the requirement that each pod has its own IP address, which is needed for efficient binpacking (without requiring application level changes from Kubernetes users). Funnily enough, this requirement doesn’t do Managed CockroachDB any good, since we always run a single CockroachDB instance on each VM. This makes me wonder how many users choose Kubernetes for binpacking vs. for automation primitives vs. for both. In this brave new world of cheap public cloud elastic compute and managed Kubernetes offerings, do many users choose Kubernetes only for the automation primitives? If so, could the networking stack be made even simpler for those users? --------------------------------------------------------------------------------------------------------------------------- What makes a database Kubernetes native? How to orchestrate a single regionKubernetes cluster: How to orchestrate a multi-region Kubernetes cluster: Free Kubernetes Tutorials: CockroachDB on Kubernetes: Community Slack: