Kubernetes Extension Interfaces

Introduction

This tutorial explores Kubernetes extension interfaces: Container Networking Interface (CNI), Container Storage Interface (CSI), and Container Runtime Interface (CRI). Understanding these interfaces is crucial for customizing and extending Kubernetes to suit specific environment needs. We assume a basic understanding of Kubernetes concepts like pods, deployments, and services. Familiarity with containerization (Docker) and networking fundamentals is also helpful. This tutorial focuses on the conceptual understanding and practical configuration aspects rather than deep-dive development of custom implementations.

CNI: Container Networking Interface

CNI is a specification for writing plugins to configure network interfaces for Linux containers. Kubernetes uses CNI to manage networking for pods, allowing different networking solutions to be integrated without modifying the core Kubernetes codebase.

CNI Responsibilities:
- Allocate IP addresses to containers.
- Configure network namespaces to connect containers to the network.
- Implement network policies.
Common CNI Plugins:
- Calico: Provides network policy enforcement and advanced networking features.
- Flannel: Simple overlay network solution.
- Weave Net: Another popular overlay network with encryption options.
- kube-router: Network provider and policy engine that operates at layer 3.
Inspecting CNI Configuration:

On a Kubernetes node, CNI configurations are typically located in /etc/cni/net.d. Let’s examine a sample configuration file (e.g., for Calico):
NODE_TYPE // bash
```
cat /etc/cni/net.d/10-calico.conflist
```
NODE_TYPE // output
```
{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "datastore_type": "kubernetes",
      "nodename": "__K8S_NODE_NAME__",
      "ipam": {
        "type": "calico-ipam"
      },
      "policy": {
        "type": "k8s"
      },
      "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      },
      "snat": true
    },
    {
      "type": "bandwidth",
      "capabilities": {
        "bandwidth": true
      }
    }
  ]
}
```
This configuration file specifies the CNI plugins to be used, including Calico, portmap, and bandwidth. type indicates the plugin name, and other fields configure its behavior.
CNI Interaction with Pods:

When a pod is created, Kubernetes invokes the configured CNI plugin on the node where the pod is scheduled. The CNI plugin then:
- Creates a network namespace for the pod.
- Creates a virtual Ethernet pair (veth).
- Attaches one end of the veth pair to the pod’s network namespace.
- Attaches the other end to the node’s network.
- Assigns an IP address to the pod’s interface.
- Configures routing rules.
Verifying Pod Networking:

Let’s create a simple pod and verify its network configuration.
NODE_TYPE // kubernetes
```
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: busybox:latest
    command: ["sleep", "3600"]
```
NODE_TYPE // bash
```
kubectl apply -f pod.yaml
```
NODE_TYPE // bash
```
kubectl exec -it test-pod -- sh
```
Inside the pod, you can use tools like ip addr to inspect the network interface and IP address.
NODE_TYPE // bash
```
ip addr
```
NODE_TYPE // output
```
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1440 qdisc noqueue
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
```
This output shows the pod’s network interface (eth0) and its IP address (e.g., 172.17.0.2).

CSI: Container Storage Interface

CSI is a standard for exposing arbitrary storage systems to containerized workloads. It enables Kubernetes to work with various storage solutions (e.g., cloud providers’ block storage, NFS, iSCSI) without needing to include storage-specific code in the core Kubernetes codebase.

CSI Components:
- CSI Driver: A vendor-provided component that implements the CSI specification for a particular storage system. It typically consists of two containers:
  - Controller Plugin: Handles provisioning, deletion, and snapshotting of volumes. Typically runs on a dedicated node.
  - Node Plugin: Handles mounting and unmounting volumes on the nodes where pods are running. Typically runs as a DaemonSet.
- Kubernetes CSI Proxy: (Windows only) Allows CSI drivers running in containers to perform privileged operations on the host.
CSI Workflow:
- A user creates a PersistentVolumeClaim (PVC) in Kubernetes, requesting storage.
- Kubernetes’ external provisioner detects the PVC.
- The external provisioner calls the CSI driver’s controller plugin to provision a volume.
- The CSI driver provisions the volume on the underlying storage system.
- The CSI driver returns the volume details (e.g., volume ID) to Kubernetes.
- Kubernetes creates a PersistentVolume (PV) object representing the provisioned volume.
- The PV is bound to the PVC.
- When a pod needs to use the volume, Kubernetes calls the CSI driver’s node plugin to mount the volume on the node where the pod is running.
Deploying a CSI Driver:

Deploying a CSI driver typically involves applying several Kubernetes manifests. Let’s consider the example of the csi-hostpath driver, which uses local directories on the host as storage. (This is primarily for testing and development purposes).
NODE_TYPE // bash
```
kubectl apply -k github.com/kubernetes-csi/csi-driver-host-path/deploy/kubernetes/?ref=v1.17.0
```
This command deploys the CSI driver using kubectl apply -k, which applies all the manifests in the specified directory. The manifests include deployments for the controller plugin and daemonsets for the node plugin.
Creating a StorageClass:

A StorageClass defines how volumes should be provisioned.
NODE_TYPE // kubernetes
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-hostpath-sc
provisioner: hostpath.csi.k8s.io
reclaimPolicy: Delete
volumeBindingMode: Immediate
```
NODE_TYPE // bash
```
kubectl apply -f storageclass.yaml
```
The provisioner field specifies the CSI driver to use. reclaimPolicy: Delete means that the volume will be deleted when the PV is deleted. volumeBindingMode: Immediate means that the PV will be provisioned as soon as the PVC is created.
Creating a PersistentVolumeClaim:
NODE_TYPE // kubernetes
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-hostpath-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: csi-hostpath-sc
  resources:
    requests:
      storage: 1Gi
```
NODE_TYPE // bash
```
kubectl apply -f pvc.yaml
```
The storageClassName field specifies the StorageClass to use. accessModes: ReadWriteOnce means that the volume can be mounted by a single node at a time. resources: requests: storage: 1Gi requests a volume with 1 GiB of storage.
Using the PersistentVolumeClaim in a Pod:
NODE_TYPE // kubernetes
```
apiVersion: v1
kind: Pod
metadata:
  name: csi-hostpath-pod
spec:
  containers:
  - name: test-container
    image: busybox:latest
    command: ["sleep", "3600"]
    volumeMounts:
    - mountPath: /data
      name: my-volume
  volumes:
  - name: my-volume
    persistentVolumeClaim:
      claimName: csi-hostpath-pvc
```
NODE_TYPE // bash
```
kubectl apply -f pod.yaml
```
This pod will mount the provisioned volume at /data.
NODE_TYPE // bash
```
kubectl exec -it csi-hostpath-pod -- sh
```
Inside the pod, you can write data to /data, which will be stored on the host’s filesystem.

CRI: Container Runtime Interface

CRI is an API that enables Kubernetes to use different container runtimes without recompiling. It decouples Kubernetes from specific container runtime implementations like Docker, containerd, and CRI-O.

CRI Components:
- kubelet: The Kubernetes node agent that interacts with the container runtime.
- CRI Shim: An adapter that translates Kubernetes’ CRI API calls into calls understood by the specific container runtime.
- Container Runtime: The software that runs containers (e.g., Docker, containerd, CRI-O).
CRI Workflow:
- The kubelet receives a request to create a pod.
- The kubelet uses the CRI API to request the container runtime to create the containers for the pod.
- The CRI shim translates the CRI API calls into calls understood by the container runtime.
- The container runtime creates the containers.
- The kubelet monitors the containers using the CRI API.
Common Container Runtimes:
- Docker: A widely used containerization platform. Kubernetes uses the dockershim CRI implementation to interface with Docker (deprecated in recent Kubernetes versions, but important to understand historically).
- containerd: A container runtime that is designed to be embedded into larger systems. It’s a CNCF project and a popular choice for Kubernetes.
- CRI-O: A container runtime specifically designed for Kubernetes. It’s also a CNCF project.
Checking the Container Runtime:

You can check the container runtime used by Kubernetes by inspecting the kubelet configuration or by querying the node status.
NODE_TYPE // bash
```
kubectl get node <node-name> -o yaml | grep containerRuntimeVersion
```
Replace <node-name> with the actual name of the node.
NODE_TYPE // output
```
    containerRuntimeVersion: docker://19.03.12
```
(The output will vary depending on the container runtime in use.)
Configuring the Container Runtime:

The container runtime is typically configured during the Kubernetes installation process. For example, when using kubeadm, you can specify the container runtime using the --container-runtime flag.
NODE_TYPE // bash
```
kubeadm init --container-runtime=containerd
```
The exact configuration steps vary depending on the container runtime and the Kubernetes distribution.
Switching Container Runtimes:

Switching container runtimes is a complex process that typically involves:
- Draining the node (evicting all pods).
- Stopping the kubelet.
- Installing and configuring the new container runtime.
- Configuring the kubelet to use the new container runtime (e.g., by modifying the kubelet configuration file).
- Starting the kubelet.
- Uncordoning the node (allowing pods to be scheduled on it again).
Switching container runtimes should be done with caution, as it can disrupt running workloads. It’s recommended to test the process in a non-production environment first.

Conclusion

This tutorial provided an overview of Kubernetes extension interfaces: CNI, CSI, and CRI. Understanding these interfaces is essential for customizing and extending Kubernetes to meet specific needs. We covered the responsibilities of each interface, common implementations, and basic configuration steps. While this tutorial focused on the conceptual understanding and practical configuration aspects, further exploration of these interfaces might involve developing custom CNI, CSI, or CRI implementations to integrate with specific networking, storage, or container runtime solutions.

Kubernetes Extension Interfaces

Introduction

CNI: Container Networking Interface

CSI: Container Storage Interface

CRI: Container Runtime Interface

Conclusion

Next Topic

Creating Kubernetes Clusters with Kubeadm

Deploying with Helm and Kustomize

Highly Available Kubernetes Control Plane