Stephan Fabel
on 10 December 2018
Using GPGPUs with Kubernetes
This post walks through the use of GPGPUs with Kubernetes and DevicePlugins. We’ll use MicroK8s for a developer workstation example and charmed K8s for a cluster since that’s a consistent multi-cloud Kubernetes approach. The various cloud CAAS offerings like GKE are also enabling GPGPU facilities so you may want to try those too.
We’ll use Ubuntu as the OS because the underlying enablement for GPGPUs ‘Just Works’ in all the clouds and with all the local hardware, and making docker images on Ubuntu ensures that the CUDA libraries line up with the drivers properly.
In order for this all to work, the correct (and matching) driver needs to be installed on the worker node to make the device accessible from the OS; and typically it also requires some userland libraries in order to work. With NVIDIA GPUs this enablement further depends on using the right Docker runtime (nvidia-docker2) which requires additional host-level configuration and post-deployment installation.
All of that is automated on Ubuntu with MicroK8s and the charmed Kubernetes charms, across all the public clouds where GPUs are available. It’s also currently activated in GKE, other cloud CAAS offerings will follow.
Workstation GPGPU containers with Microk8s
Microk8s is a snap of upstream Kubernetes that is designed for development purposes. It’s not a cluster but it gives you a small zero-ops kubernetes environment that is compatible with all the major multi-cloud K8s offerings. For our purposes the important thing is that it includes GPGPU enablement in the box.
To install MicroK8s:
$ snap install microk8s --classicThis will give you the latest stable version of MicroK8s which tracks upstream releases closely.
You can select a particular version using ‘snap channels’, see ‘snap info microk8s’ for the available tracks. By selecting a particular track you can lock yourself to a particular version of Kubernetes. By default you will be on the ‘latest’ track, and get upgrades when upstream Kubernetes releases a new stable version. Select a particular track with --channel=track/stability from the available channels. ‘Stable’ maps to ‘latest/stable’.
$ snap info microk8s
[...]
channels:
stable: v1.13.0 (340) 204MB classic
candidate: v1.13.0 (340) 204MB classic
beta: v1.13.0 (340) 204MB classic
edge: v1.13.0 (340) 204MB classic
1.13/stable: v1.13.0 (340) 204MB classic
1.13/candidate: v1.13.0 (340) 204MB classic
1.13/beta: v1.13.0 (340) 204MB classic
1.13/edge: v1.13.0 (341) 204MB classic
1.12/stable: v1.12.3 (336) 226MB classic
1.12/candidate: v1.12.3 (336) 226MB classic
1.12/beta: v1.12.3 (336) 226MB classic
1.12/edge: v1.12.3 (336) 226MB classic
1.11/stable: v1.11.5 (322) 219MB classic
1.11/candidate: v1.11.5 (322) 219MB classic
1.11/beta: v1.11.5 (322) 219MB classic
1.11/edge: v1.11.5 (322) 219MB classic
1.10/stable: v1.10.11 (321) 175MB classic
1.10/candidate: v1.10.11 (321) 175MB classic
1.10/beta: v1.10.11 (321) 175MB classic
1.10/edge: v1.10.11 (321) 175MB classic
Assuming you have an Nvidia GPU with a current driver installed, you can activate Kubernetes support for it with the “enable” subcommand:
$ microk8s.enable gpuYou can confirm that the GPU is available to Microk8s with this command:
$ microk8s.status
microk8s is running
addons:
gpu: enabled
storage: disabled
registry: disabled
ingress: disabled
dns: disabled
metrics-server: disabled
istio: disabled
dashboard: disabledRunning GPGPU-accelerated containers on Kubernetes
Now that you have GPGPU capacity available to Kubernetes you can deploy containers there that get access to the special hardware they need.
Your container needs to have the right userspace pieces, so again we suggest that you build the OCI images on Ubuntu with the CUDA libraries provided; those will be most portable across all the different cloud CAAS offerings as well as offerings from Canonical, VMware, Pivotal, Cisco and others that also use Ubuntu for K8s.
Your workloads can now use something like this to select appropriate worker nodes (example taken from here):
Listing 1: nvidia-pod-example.yaml
apiVersion: v1
kind: Pod
metadata:
name: cuda-vector-add
spec:
restartPolicy: OnFailure
containers:
- name: cuda-vector-add
image: "k8s.gcr.io/cuda-vector-add:v0.1"
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPUKubernetes cluster deployment with GPGPUs
A compelling feature of the Charmed Distribution of Kubernetes (CDK) is that it will automatically enable GPGPU resources which are present on the worker node for use by K8s pods.
GPU resources are enabled through the use of Device Plugins which are deployed as DaemonSets. This ensures that each GPU-enabled worker node is allowed access to the GPU and sets the right paths to the driver plugins on the host.
With the DaemonSet deployed, the Kubernetes scheduler can leverage the NodeSelector to filter worker node candidates advertising the nvidia.com/gpu feature when scheduling workloads.
Charms fully automate the deployment of Kubernetes in a way that is model-driven and thus flexible for use on different kinds of cloud or cluster. We use charms successfully for HPC deployments of Kubernetes, for example, making the deployment of AI/ML pipelines on top of Kubernetes easier. GPU enablement is important for those sorts of workloads.
However, before deploying Kubeflow or similar frameworks, the Kubernetes layer needs to be fully automated and GPUs activated.
The charms of Kubernetes do all the work. As worker nodes get commissioned into the model, the Kubernetes charms auto-detect the presence of NVIDIA hardware, install the right driver and host libraries, replace the container runtime with the NVIDIA supported one, deploy the DaemonSet for the DevicePlugin and labels the nodes automatically.
The K8s cluster is best deployed with conjure-up which will walk you through the entire process. You can use conjure-up on a public cloud with GPU-enabled instance types, or on MAAS for bare metal clusters with servers that contain GPUs. In both cases, the deployment process is exactly the same.
For example, you can use p2.xlarge instances on AWS. In order to make that happen, we need to pass a constraint into the conjure-up command line so that we force the usage of the GPU enabled instance types when deploying workers.
Listing 2: cdk-gpu-worker.yaml
services:
"kubernetes-worker":
charm: "cs:~containers/kubernetes-worker"
num_units: 1
options:
channel: 1.13/stable
expose: true
constraints: "instance-type=p2.xlarge root-disk=32768"Pass this to conjure-up:
$ conjure-up canonical-kubernetes --bundle-add cdk-gpu-worker.yamlThis will launch the conjure-up wizard interface and allow you to select additional add-ons to be deployed, for example, Kubeflow can be selected here. On the controller selection screen, you can either deploy a dedicated Juju controller (one more VM) or you can take advantage of JAAS, which provides Juju-as-a-service on the major public clouds.
Once the installation is kicked off, you see a status screen as shown below:
The status can also be shown using the Juju command directly. If you use JAAS, locate the model name using:
$ juju models -c jaas
Controller: jaas
Model Cloud/Region Status Machines Cores Access Last connection
conjure-canonical-kubern-9dc aws/us-east-1 available 0 0 admin never connectedThen, inspect the status of the model:
$ juju status -m jaas:conjure-canonical-kubern-9dc
Model Controller Cloud/Region Version SLA Timestamp
conjure-canonical-kubern-9dc jaas aws/us-east-1 2.4.5 unsupported 16:13:37-08:00
App Version Status Scale Charm Store Rev OS Notes
aws-integrator 1.15.71 active 1 aws-integrator jujucharms 7 ubuntu
easyrsa 3.0.1 maintenance 1 easyrsa jujucharms 117 ubuntu
etcd maintenance 3 etcd jujucharms 209 ubuntu
flannel waiting 0 flannel jujucharms 146 ubuntu
kubeapi-load-balancer maintenance 1 kubeapi-load-balancer jujucharms 162 ubuntu exposed
kubernetes-master maintenance 2 kubernetes-master jujucharms 219 ubuntu
kubernetes-worker waiting 0/1 kubernetes-worker jujucharms 239 ubuntu exposed
Unit Workload Agent Machine Public address Ports Message
aws-integrator/0* active idle 0 54.165.35.94 ready
easyrsa/0* maintenance executing 1 34.234.207.232 (install) installing charm
etcd/0* maintenance executing 2 54.208.163.252 (install) installing charm
etcd/1 maintenance executing 3 34.201.210.154 (install) installing charm
etcd/2 maintenance executing 4 54.235.228.45 (install) installing charm
kubeapi-load-balancer/0* maintenance executing 5 34.228.169.37 (install) installing charm
kubernetes-master/0 maintenance executing 6 18.207.179.122 (install) installing charm
kubernetes-master/1* maintenance executing 7 18.212.150.203 (install) installing charm
kubernetes-worker/0 waiting allocating 8 35.175.104.2 waiting for machine
Machine State DNS Inst id Series AZ Message
0 started 54.165.35.94 i-083ce279733998d59 bionic us-east-1a running
1 started 34.234.207.232 i-04828688ddfdb0c6c bionic us-east-1b running
2 started 54.208.163.252 i-03d910e892e7c09f6 bionic us-east-1a running
3 started 34.201.210.154 i-00adeecd668174ee0 bionic us-east-1b running
4 started 54.235.228.45 i-032875fd24a1c1e78 bionic us-east-1c running
5 started 34.228.169.37 i-0008405049b9bed6d bionic us-east-1d running
6 started 18.207.179.122 i-003abf7f3612a2f18 bionic us-east-1b running
7 started 18.212.150.203 i-0abe01060e8179618 bionic us-east-1a running
8 pending 35.175.104.2 i-0d493a35776b9217d bionic us-east-1e runningOnce the installation has finished all GPGPU resources are properly configured and available to the Kubernetes operator. You can check this with:
$ kubectl get no -o wide -L cuda,gpuConclusion
Leveraging GPGPU resources in your Kubernetes cluster is automatic and easy to do when using the Charmed Distribution of Kubernetes or Microk8s.
What do you think? We’d love to hear about your use cases and how CDK and Microk8s helped with your GPGPU-sensitive workloads.
