CSCWiki - User contributions [en]

Kubernetes

2026-09-11T03:26:26Z

A2lianto: Changed user documentation to point to CSClub Wiki

We are running a [https://kubernetes.io/ Kubernetes] cluster on top of [[CloudStack]].

User documentation is here: [[Kubernetes User Guide]]

== CloudStack setup ==
Enable the [https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-service.html Kubernetes plugin] from the CloudStack UI. This will require a restart of the management servers.

We currently have one control node and 3 worker nodes. Each node is using the same Compute offering (8 CPUs, 16GB of RAM). Autoscaling is enabled, so CloudStack will automatically create more worker nodes if necessary.

The admin kubeconfig has been installed on biloba and chamomile.

Note that we cannot use LoadBalancers because we are basically running our own load balancer (NGINX) outside of Kubernetes which accepts external traffic. To expose services, use Ingresses or NodePorts instead.

== NGINX Ingress ==
Read this first: https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal-clusters

Then run:
<pre>
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/baremetal/deploy.yaml
</pre>

Get the NodePort:
<pre>
kubectl -n ingress-nginx get svc
</pre>
Create an upstream in /etc/nginx/nginx.conf which points to the IPs of one or more Kubernetes VMs, with the HTTP port from the NodePort.
Then, reload NGINX on biloba and chamomile.

Mark the NGINX IngressClass as the default IngressClass:
<pre>
kubectl edit ingressclass nginx
</pre>
This will open up Vim; add the annotation <code>ingressclass.kubernetes.io/is-default-class: "true"</code> to the annotations section.

Edit the global ConfigMap:
<pre>
kubectl -n ingress-nginx edit configmap ingress-nginx-controller
</pre>
Add the following to the 'data' section:
<pre>
allow-backend-server-header: "true"
use-forwarded-headers: "true"
proxy-buffer-size: 128k
server-snippet: |
proxy_http_version 1.1;
proxy_pass_header Connection;
proxy_pass_header Upgrade;
</pre>

== CSI Driver ==
We are using a [https://github.com/apalia/cloudstack-csi-driver CSI driver] for PersistentVolume storage.

Installation: 
UPDATE: don't apply the manifest directly; you'll need to download and edit it first. It seems like the labels on the control plane node changed starting from v1.24. 
After downloading the manifest, open it in an editor and change <code>node-role.kubernetes.io/master: ""</code> to <code>node-role.kubernetes.io/control-plane: ""</code>. 
If you already applied the manifest and need to edit it, just run <code>kubectl -n kube-system edit deployment cloudstack-csi-controller</code>.
<pre>
wget https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml
# Make necessary edits
vim manifest.yaml
kubectl apply -f manifest.yaml
</pre>

To make this the default StorageClass, clone the [https://github.com/apalia/cloudstack-csi-driver repo], and edit examples/k8s/0-storageclass.yml so that it looks like this:
<pre>
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cloudstack-storage
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.cloudstack.apache.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
parameters:
csi.cloudstack.apache.org/disk-offering-id: 0da1f706-fd2e-4203-8bae-1b740aef9886
</pre>
Change the disk-offering-id to the ID of the 'custom' disk size offering in CloudStack. Apply the YAML file once you are done editing it.

Note: only a single writer is allowed, so do NOT use ReadWriteMany on any PersistentVolumeClaims.

=== Testing ===
Create a PersistentVolumeClaim and bind it to a Pod just to make sure that everything's working:
<pre>
kubectl apply -f ./examples/k8s/pvc.yaml
kubectl apply -f ./examples/k8s/pod.yaml
</pre>
Run <code>kubectl get pv</code> to make sure that a PersistentVolume was dynamically provisioned.

Once you're done testing, delete the resources:
<pre>
kubectl delete -f ./examples/k8s/pvc.yaml
kubectl delete -f ./examples/k8s/pod.yaml
</pre>

== SSH'ing into a node ==
If you need to SSH into one of the Kubernetes nodes, get the IP from the CloudStack UI, and run e.g.
<pre>
ssh -i /var/lib/cloudstack/management/.ssh/id_rsa core@172.19.134.149
</pre>
(Do this from biloba or chamomile.)

== Docker shim ==
The original CloudStack Kubernetes ISO which we used (v1.22) used Docker as the container engine, which is no longer supported; after upgrading to v1.24, all hell broke loose because kubelet tried to use containerd instead. As a workaround, we are using [https://github.com/Mirantis/cri-dockerd cri-dockerd] on the control plane and the worker nodes. Each VM should have this in /var/lib/kubelet/kubeadm-flags.env:
<pre>
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///run/cri-dockerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.7"
</pre>
The solution was found [https://discuss.kubernetes.io/t/unable-to-determine-runtime-api-version-rpc-error/20736 here].

== Enabling a feature gate ==
UPDATE: starting from v1.23, the PodSecurity feature gate is [https://v1-23.docs.kubernetes.io/docs/concepts/security/pod-security-admission/#enabling-the-podsecurity-admission-plugin enabled by default], so there is no need to manually enable it. The rest of this section was kept for historical purposes only.

In v1.22, the PodSecurity feature gate is an Alpha feature and must be enabled in kube-apiserver (https://kubernetes.io/docs/concepts/security/pod-security-admission/).

SSH into the control node, and edit /etc/kubernetes/manifests/kube-apiserver.yaml so that the 'command' list has the following flag:
<pre>
--feature-gates=PodSecurity=true
</pre>
(If the flag is already present, add the gate after a comma, e.g. <code>--feature-gates=Feature1=true,PodSecurity=true</code>.)

This will automatically restart the kube-apiserver; wait a minute and run <code>kubectl -n kube-system get pods</code> to check.

== Certificates ==
=== Admin kubeconfig ===
The kubeconfig in the CloudStack UI will only last one year (as of this writing, it is expired, so don't use it). If it expires again, here's how you can renew it:
<ol>
<li>SSH into the control plane VM (see instructions above)</li>
<li>
Create a file called e.g. kubeadm-config.yaml with this content:
<pre>
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
clusterName: "kubernetes"
controlPlaneEndpoint: "172.19.134.149:6443"
certificatesDir: "/etc/kubernetes/pki"
</pre>
</li>
<li>
Generate a new admin kubeconfig which will last ten years:
<pre>
kubeadm kubeconfig user --config=kubeadm-config.yaml --client-name=kubernetes-admin --org=system:masters --validity-period=87600h0m0s
</pre>
Reference: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubeconfig-additional-users
</li>
<li>
Copy the output into <code>/root/.kube/config</code> on biloba and chamomile.
</li>
</ol>

=== Kubelet certificate rotation ===
See https://kubernetes.io/docs/tasks/tls/certificate-rotation/.

SSH into the control plane and make sure that <code>rotateCertificates: true</code> is set in /var/lib/kubelet/config.yaml.

=== What to do if the certificates expire ===
SSH into the control plane VM. To check the expired certs, run <code>kubeadm certs check-expiration</code>. From biloba or chamomile, you can also run <code>kubectl -n kube-system get cm kubeadm-config -o yaml</code> if the admin kubeconfig has not expired.

To renew the expired certificates, run
<pre>
kubeadm certs renew all
</pre>
You now need to restart kube-apiserver, kube-controller-manager, kube-scheduler and etcd. Unfortunately I was never able to figure out how to do this. Deleting the pods doesn't seem to work. We might need to restart all of the Docker containers running on the control plane. You can check the logs of the kube-apiserver pod to see if it's still having certificate expiry issues.

Anywho, the safe but slow option is to just restart all of the Kubernetes VMs from the CloudStack web UI.

To be sure that everything is working again, make sure that you can create a temporary pod successfully.

== Members ==
[[ceo]] manages the creation of new Kubernetes namespaces for members. See [https://git.csclub.uwaterloo.ca/public/pyceo/src/branch/master/ceod/model/KubernetesService.py here] to see how this works.

We are also using [https://open-policy-agent.github.io/gatekeeper/website/docs/ OPA Gatekeeper] to restrict the Ingresses which members can create. See [https://git.csclub.uwaterloo.ca/cloud/manifests/src/branch/master/cscingressconstraint-template.yaml here] and [https://git.csclub.uwaterloo.ca/cloud/manifests/src/branch/master/cscingressconstraint-constraint.yaml here] for details.

=== Certificate Signing Requests ===
We're going to set the max. CSR signing duration to 10 years so that members don't have to worry about their kubeconfig cert expiring (at least, not for a long time).

SSH into the control node and edit /etc/kubernetes/manifests/kube-controller-manager.yaml so that it has the following CLI flag:
<pre>
--cluster-signing-duration=87600h
</pre>
The controller will automatically restart after you save and close the file.

Kubernetes User Guide

2026-09-11T03:24:47Z

A2lianto: removed "template:info"

= Kubernetes =
We are running a [https://kubernetes.io/ Kubernetes] cluster on-prem on Tahini and Teriyaki servers. Each member gets their own namespace in which they can deploy their applications.

== Account resource limits ==
As of August 23, 2026, the per-namespace resource limits for each member are:

* 40 pods
* 10 jobs
* 10 cron jobs
* 25 GB of Persistent Volume Claims
* 5 NodePort services

If you wish to acquire more resources, please send an email to the Systems Committee with a brief justification.

== Create a new namespace ==
Log into a general-use machine and run the following:<pre>
ceo k8s account activate
</pre>This will create a new Kubernetes namespace with the name <code>csc-username</code>. A new kubeconfig file will be placed into <code>~/.kube/config</code>. Your kubeconfig points at your namespace by default, so all commands you run will land there unless you specify otherwise.

To verify that everything is working, run:<pre>
kubectl cluster-info
</pre>The output should look something like this:<pre>
Kubernetes control plane is running at https://129.97.134.5:6443
CoreDNS is running at https://129.97.134.5:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
</pre>

== Quickstart ==
Say you have a Dockerized app and the image is available on a public registry such as [https://hub.docker.com/ Docker Hub] or [https://quay.io/ Quay.io]. Here is the minimum you need to get your app running in the cluster.

First, create a Deployment:<pre>
kubectl create deployment demo --image=ctdalek/myapp --port=80
</pre>Replace <code>--image</code> and <code>--port</code> as needed.

If your app does not need to be publicly accessible, you are done. Otherwise, create a Service to expose your Deployment:<pre>
kubectl expose deployment demo
</pre>Then create an Ingress to expose the Service to the outside world:<pre>
kubectl create ingress demo --rule='ctdalek.k8s.csclub.cloud/*=demo:80'
</pre>Your app will be accessible at <code>https://ctdalek.k8s.csclub.cloud</code>. If you wish to use a custom domain, please contact the Systems Committee.

For non-web apps, use a NodePort instead of an Ingress. NodePorts are only accessible from on-campus.

== Further reading ==
Kubernetes is well documented, so we will not repeat it here:

* [https://kubernetes.io/docs/home/ Kubernetes documentation]
* [https://kubernetes.io/docs/tutorials/ Kubernetes tutorials]
* [https://kubectl.docs.kubernetes.io/guides/ kubectl guides]

You may also contact the [mailto:syscom@csclub.uwaterloo.ca Systems Committee] if something is not working the way you expect.

Kubernetes User Guide

2026-09-11T03:24:17Z

A2lianto: Created Kubernetes User Guide for new k3s cluster

= Kubernetes =
We are running a [https://kubernetes.io/ Kubernetes] cluster on-prem on Tahini and Teriyaki servers. Each member gets their own namespace in which they can deploy their applications.

{{Info|If you wish to learn about how to operate a Kubernetes cluster, you may create your own cluster using software such as [https://k3s.io/ k3s]. If all you want to do is run some containerized apps, then we suggest that you use our cluster instead, since it has far more resources.}}

== Account resource limits ==
As of August 23, 2026, the per-namespace resource limits for each member are:

* 40 pods
* 10 jobs
* 10 cron jobs
* 25 GB of Persistent Volume Claims
* 5 NodePort services

If you wish to acquire more resources, please send an email to the Systems Committee with a brief justification.

{{Info|LoadBalancer services are disabled because we handle external traffic through Traefik, which we manage ourselves. See [[#Exposing your app|Exposing your app]] below.}}

== Create a new namespace ==
Log into a general-use machine and run the following:<pre>
ceo k8s account activate
</pre>This will create a new Kubernetes namespace with the name <code>csc-username</code>. A new kubeconfig file will be placed into <code>~/.kube/config</code>. Your kubeconfig points at your namespace by default, so all commands you run will land there unless you specify otherwise.

To verify that everything is working, run:<pre>
kubectl cluster-info
</pre>The output should look something like this:<pre>
Kubernetes control plane is running at https://129.97.134.5:6443
CoreDNS is running at https://129.97.134.5:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
</pre>

== Quickstart ==
Say you have a Dockerized app and the image is available on a public registry such as [https://hub.docker.com/ Docker Hub] or [https://quay.io/ Quay.io]. Here is the minimum you need to get your app running in the cluster.

First, create a Deployment:<pre>
kubectl create deployment demo --image=ctdalek/myapp --port=80
</pre>Replace <code>--image</code> and <code>--port</code> as needed.

If your app does not need to be publicly accessible, you are done. Otherwise, create a Service to expose your Deployment:<pre>
kubectl expose deployment demo
</pre>Then create an Ingress to expose the Service to the outside world:<pre>
kubectl create ingress demo --rule='ctdalek.k8s.csclub.cloud/*=demo:80'
</pre>Your app will be accessible at <code>https://ctdalek.k8s.csclub.cloud</code>. If you wish to use a custom domain, please contact the Systems Committee.

For non-web apps, use a NodePort instead of an Ingress. NodePorts are only accessible from on-campus.

== Further reading ==
Kubernetes is well documented, so we will not repeat it here:

* [https://kubernetes.io/docs/home/ Kubernetes documentation]
* [https://kubernetes.io/docs/tutorials/ Kubernetes tutorials]
* [https://kubectl.docs.kubernetes.io/guides/ kubectl guides]

You may also contact the [mailto:syscom@csclub.uwaterloo.ca Systems Committee] if something is not working the way you expect.