Mastering Kubernetes

Table of Contents

Mastering Kubernetes

Credits

About the Author

About the Reviewer

www.PacktPub.com

eBooks, discount offers, and more

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the color images of this book

Errata

Piracy

Questions

1. Understanding Kubernetes Architecture

Understanding container orchestration

Physical machines, virtual machines, and containers

Containers in the cloud

Cattle versus pets

Kubernetes concepts

Cluster

Node

Master

Pod

Label

Annotation

Label selector

Replication controller and replica set

Service

Volume

StatefulSet

Secret

Name

Namespace

Diving into Kubernetes architecture in depth

Distributed systems design patterns

Sidecar pattern

Ambassador pattern

Adapter pattern

Multi-node patterns

The Kubernetes APIs

Kubernetes API

Autoscaling API

Batch API

Kubernetes components

Master components

API server

Etcd

Controller manager

Scheduler

DNS

Node components

Proxy

Kubelet

Kubernetes runtimes

The runtime interface

Docker

Rkt

App container

Rktnetes

Is rkt ready for production usage?

Hyper containers

Hypernetes

Continuous integration and deployment

What is a CI/CD pipeline?

Designing a CI/CD pipeline for Kubernetes

Summary

2. Creating Kubernetes Clusters

Quick single-node cluster with Minikube

Getting ready

Creating the cluster

Troubleshooting

Checking out the cluster

Doing work

Examining the cluster with the dashboard

Creating a multi-node cluster using kubeadm

Getting ready

Preparing a cluster of vagrant VMs

Installing the required software

The hosts file

The vars.yml file

The playbook.yml file

Creating the cluster

Initializing the master

Setting up the pod network

Adding the worker nodes

Creating clusters in the cloud (GCP, AWS, Azure)

The cloud-provider interface

GCP

AWS

Azure

Creating a bare-metal cluster from scratch

Use cases for bare-metal

When should you consider creating a bare-metal cluster?

The process

Using virtual private cloud infrastructure

Summary

3. Monitoring, Logging, and Troubleshooting

Monitoring Kubernetes with Heapster

cAdvisor

InfluxDB backend

The storage schema

CPU

Filesystem

Memory

Network

Uptime

Grafana visualization

Performance analysis with the dashboard

Top-level view

Admin view

Workloads

Services and discovery

Adding central logging

Planning central logging

Fluentd

Elasticsearch

Kibana

Detecting node problems

Node problem detector

DaemonSet

Problem Daemons

Troubleshooting scenarios

Designing robust systems

Hardware failure

Quotas, shares, and limits

Bad configuration

Cost versus performance

Managing cost on the cloud

Managing cost on bare metal

Managing cost on hybrid clusters

Summary

4. High Availability and Reliability

High-availability concepts

Redundancy

Hot swapping

Leader election

Smart load balancing

Idempotency

Self-healing

High-availability best practices

Creating highly available clusters

Making your nodes reliable

Protecting your cluster state

Clustering etcd

Static discovery

Etcd discovery

DNS discovery

The etcd.yaml file

Verifying the etcd cluster

etcd 2 versus etcd 3

Protecting your data

Running redundant API servers

Running leader election with Kubernetes

Leader election for your application

Making your staging environment highly available

Testing high-availability

Live cluster upgrades

Rolling upgrades

Complex deployments

Blue-green upgrades

Managing data-contract changes

Migrating data

Deprecating APIs

Large-cluster performance, cost, and design trade-offs

Availability requirements

Best effort

Maintenance windows

Quick recovery

Zero-downtime

Performance and data consistency

Summary

5. Configuring Kubernetes Security, Limits, and Accounts

Understanding Kubernetes security challenges

Node challenges

Network challenges

Image challenges

Configuration and deployment challenges

Pod and container challenges

Organisational, cultural, and process challenges

Hardening Kubernetes

Understanding service accounts in Kubernetes

How does Kubernetes manage service accounts?

Accessing the API server

Authenticating users

Authorizing requests

Using admission control plugins

Securing pods

Using a private image repository

ImagePullSecrets

Specifying a security context

Protecting your cluster with AppArmor

Requirements

Securing a pod with AppArmor

Writing AppArmor profiles

Pod security policies

Managing network policies

Choosing a supported networking solution

Defining a network policy

Using secrets

Storing secrets in Kubernetes

Creating secrets

Decoding secrets

Using secrets in a container

Running a multi-user cluster

The case for a multi-user cluster

Using namespaces for safe multi-tenancy

Avoiding namespace pitfalls

Summary

6. Using Critical Kubernetes Resources

Designing the Hue platform

Defining the scope of Hue

Hue components

User profile

User graph

Identity

Authorizer

External service

Generic sensor

Generic actuator

User learner

Hue microservices

Plugins

Data stores

Stateless microservices

Queue-based interactions

Planning workflows

Automatic workflows

Human workflows

Budget-aware workflows

Using Kubernetes to build the Hue platform

Using Kubectl effectively

Understanding Kubectl resource configuration files

ApiVersion

Kind

Metadata

Spec

Container spec

Deploying long-running microservices in pods

Creating pods

Decorating pods with labels

Deploying long- running processes with deployments

Updating a deployment

Separating internal and external services

Deploying an internal service

Creating the Hue-reminders service

Exposing a service externally

Ingress

Using namespace to limit access

Launching jobs

Running jobs in parallel

Cleaning up completed jobs

Scheduling cron jobs

Kubectl get pods

Mixing non-cluster components

Outside-the-cluster-network components

Inside-the-cluster-network components

Managing the Hue platform with Kubernetes

Using liveness probes to ensure your containers are alive

Using readiness probes to manage dependencies

Employing init containers for orderly pod bring-up

Sharing with DaemonSet pods

Evolving the Hue platform with Kubernetes

Utilizing Hue in the enterprise

Advancing science with Hue

Educating the kids of the future with hue

Summary

7. Handling Kubernetes Storage

Persistent volumes walkthrough

Volumes

Using emptyDir for intra-pod communication

Using HostPath for intra-node communication

Provisioning persistent volumes

Provisioning persistent volumes statically

Provisioning persistent volumes dynamically

Creating persistent volumes

Capacity

Access modes

Reclaim policy

Volume type

Making persistent volume claims

Mounting claims as volumes

Storage classes

Default storage class

Demonstrating persistent volume storage end to end

Public storage volume types - GCE, AWS, and Azure

AWS Elastic Block Store (EBS)

AWS Elastic File System (EFS)

GCE persistent disk

Azure data disk

Azure file storage

GlusterFS and Ceph volumes in Kubernetes

Using GlusterFS

Creating endpoints

Adding a GlusterFS Kubernetes service

Creating pods

Using Ceph

Connecting to Ceph using RBD

Connecting to Ceph using CephFS

Flocker as a clustered container data volume manager

Integrating enterprise storage into Kubernetes

Torus – the new kid on the block

Summary

8. Running Stateful Applications with Kubernetes

Stateful versus stateless applications in Kubernetes

Understanding the nature of distributed data-intensive apps

Why manage states in Kubernetes?

Why manage states outside of Kubernetes?

Shared environment variables versus DNS records for discovery

Accessing external data stores via DNS

Accessing external data stores via environment variables

Creating a ConfigMap

Consuming a ConfigMap as an environment variable

Using a redundant in-memory state

Using DaemonSet for redundant persistent storage

Applying persistent volume claims

Utilizing StatefulSet

When to use StatefulSet

The components of StatefulSet

Running a Cassandra cluster in Kubernetes

Quick introduction to Cassandra

The Cassandra Docker image

Exploring the run.sh script

Hooking up Kubernetes and Cassandra

Digging into the Cassandra configuration

The custom seed provider

Creating a Cassandra headless service

Using statefulSet to create the Cassandra cluster

Dissecting the stateful set configuration file

Using a replication controller to distribute Cassandra

Dissecting the replication controller configuration file

Assigning pods to nodes

Using DaemonSet to distribute Cassandra

Summary

9. Rolling Updates, Scalability, and Quotas

Horizontal pod autoscaling

Declaring horizontal pod autoscaler

Custom metrics

Autoscaling with Kubectl

Performing rolling updates with autoscaling

Handling scarce resources with limits and quotas

Enabling resource quotas

Resource quota types

Compute resource quota

Storage resource quota

Object count quota

Quota scopes

Requests and limits

Working with quotas

Using namespace-specific context

Creating quotas

Using limit ranges for default compute quotas

Choosing and managing the cluster capacity

Choosing your node types

Choosing your storage solutions

Trading off cost and response time

Using effectively multiple node configurations

Benefiting from elastic cloud resources

Autoscaling instances

Mind your cloud quotas

Manage regions carefully

Considering Hyper.sh

Pushing the envelope with Kubernetes

Improving the performance and scalability of Kubernetes

Caching reads in the API server

The pod lifecycle event generator

Serializing API objects with protocol buffers

Measuring the performance and scalability of Kubernetes

The Kubernetes SLOs

Measuring API responsiveness

Measuring end to end pod startup time

Testing Kubernetes at scale

Introducing the Kubemark tool

Setting up a Kubemark cluster

Comparing a Kubemark cluster to a real-world cluster

Summary

10. Advanced Kubernetes Networking

Understanding the Kubernetes networking model

Intra-pod communication (container to container)

Inter-pod communication (pod to pod)

Pod to service communication

External access

Kubernetes networking versus Docker networking

Lookup and discovery

Self-registration

Services and endpoints

Loosely coupled connectivity with queues

Loosely coupled connectivity with data stores

Kubernetes ingress

Kubernetes network plugins

Basic Linux networking

IP addresses and ports

Network namespaces

Virtual Ethernet devices

Bridges

Routing

Maximum transmission unit

Pod networking

Kubenet

Requirements

Setting the MTU

Container networking interface

Container runtime

CNI plugin

Kubernetes networking solutions

Bridging on bare metal clusters

Contiv

Open vSwitch

Nuage networks VCS

Canal

Flannel

Calico project

Romana

Weave net

Using network policies effectively

Understanding the Kubernetes network policy design

Network policies and CNI plugins

Configuring network policies

Implementing network policies

Load balancing options

External load balancer

Configuring an external load balancer

Via configuration file

Via Kubectl

Finding the load balancer IP addresses

Identifying client IP addresses

Annotating the load balancer for client IP address preservation

Understanding potential in even external load balancing

Service load balancer

Ingress

HAProxy

Utilizing the NodePort

Custom load balancer provider using HAProxy

Running HAProxy Inside the Kubernetes cluster

Keepalived VIP

Writing your own CNI plugin

First look at the loopback plugin

Building on the CNI plugin skeleton

Reviewing the bridge plugin

Summary

11. Running Kubernetes on Multiple Clouds and Cluster Federation

Understanding cluster federation

Important use cases for cluster federation

Capacity overflow

Sensitive workloads

Avoiding vendor lock-in

Geo-distributing high availability

The federation control plane

Federation API server

Federation controller manager

Federated resources

Federated ConfigMap

Creating a federated ConfigMap

Viewing a federated ConfigMap

Updating a federated ConfigMap

Deleting a federated ConfigMap

Federated DaemonSet

Federated deployment

Federated events

Federated ingress

Creating a federated ingress

Request routing with a federated ingress

Handling failures with federated ingress

Federated namespace

Federated ReplicaSet

Federated secrets

The hard parts

Federated unit of work

Location affinity

Strictly-coupled

Loosely-coupled

Preferentially-coupled

Strictly-decoupled

Uniformly-spread

Cross-cluster scheduling

Federated data access

Federated auto-scaling

Managing a Kubernetes cluster federation

Setting up cluster federation from the ground up

Initial setup

Using the official hyperkube image

Running the federation control plane

Registering Kubernetes clusters with federation

Updating KubeDNS

Shutting down the federation

Setting up cluster federation with Kubefed

Getting Kubefed

Choosing a host cluster

Deploying a federation control plane

Adding a cluster to a federation

Naming rules and customization

Secret name

Removing a cluster from a federation

Shutting down the federation

Cascading delete of resources

Load balancing across multiple clusters

Failing over across multiple clusters

Federated service discovery

Federated migration

Running federated workloads

Creating a federated service

Adding backend pods

Verifying public DNS records

Discovering a federated service

DNS expansion

Handling failures of backend pods and whole clusters

Troubleshooting

Unable to connect to federation API server

Summary

12. Customizing Kubernetes - API and Plugins

Working with the Kubernetes API

Understanding OpenAPI

Setting up a proxy

Exploring the Kubernetes API directly

Using Postman to explore the Kubernetes API

Filtering the output with httpie and jq

Creating a pod via the Kubernetes API

Accessing the Kubernetes API via the Python client

Dissecting the CoreV1API group

Listing objects

Creating objects

Watching objects

Invoking Kubectl programmatically

Using Python subprocess to run Kubectl

Extending the Kubernetes API

Understanding the structure of a third-party-resource

Developing third-party-resources

Integrating third party resources

Writing Kubernetes plugins

Writing a custom scheduler plugin

Understanding the design of the Kubernetes scheduler

The scheduler

Registering an algorithm provider

Configuring the scheduler

Packaging the scheduler

Deploying the custom scheduler

Running another custom scheduler in the cluster

Assigning pods to the custom scheduler

Verifying that the pods were scheduled using custom scheduler

Writing an authorization plugin

Writing an admission control plugin

Implementing an admission control plugin

Registering an admission control plugin

Linking your custom admission control plugin

Writing a custom metrics plugin

Configuring the pod for custom metrics

Specifying a target metric value

Writing a volume plugin

Implementing a volume plugin

Registering a volume plugin

Linking a volume plugin

Summary

13. Handling the Kubernetes Package Manager

Understanding Helm

The motivation for Helm

The Helm architecture

Helm components

The Tiller server

The Helm client

Helm versus. Helm-classic

Using Helm

Installing Helm

Installing the Helm client

Installing the Tiller server

Installing Tiller in-cluster

Installing Tiller locally

Finding charts

Installing packages

Checking installation status

Customizing a chart

Additional installation options

Upgrading and rolling back a release

Deleting a release

Working with repositories

Managing charts with Helm

Taking advantage of starter packs

Creating your own charts

The Chart.yaml file

Versioning charts