Kubernetes - Revision history

Merenber: /* Members */

2023-01-07T07:49:11Z

Members

← Older revision		Revision as of 02:49, 7 January 2023
Line 175:		Line 175:
	We're going to set the max. CSR signing duration to 10 years so that members don't have to worry about their kubeconfig cert expiring (at least, not for a long time).		We're going to set the max. CSR signing duration to 10 years so that members don't have to worry about their kubeconfig cert expiring (at least, not for a long time).

	SSH into the control node and edit /etc/kubernetes/manifests/kube-controller-manager.~~yml~~ so that it has the following CLI flag:		SSH into the control node and edit /etc/kubernetes/manifests/kube-controller-manager.yaml so that it has the following CLI flag:
	<pre>		<pre>
	--cluster-signing-duration=87600h		--cluster-signing-duration=87600h

Merenber at 07:24, 7 January 2023

2023-01-07T07:24:01Z

← Older revision		Revision as of 02:24, 7 January 2023
Line 101:		Line 101:
	</pre>		</pre>
	(Do this from biloba or chamomile.)		(Do this from biloba or chamomile.)

			== Docker shim ==
			The original CloudStack Kubernetes ISO which we used (v1.22) used Docker as the container engine, which is no longer supported; after upgrading to v1.24, all hell broke loose because kubelet tried to use containerd instead. As a workaround, we are using [https://github.com/Mirantis/cri-dockerd cri-dockerd] on the control plane and the worker nodes. Each VM should have this in /var/lib/kubelet/kubeadm-flags.env:
			<pre>
			KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///run/cri-dockerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.7"
			</pre>
			The solution was found [https://discuss.kubernetes.io/t/unable-to-determine-runtime-api-version-rpc-error/20736 here].

	== Enabling a feature gate ==		== Enabling a feature gate ==

Merenber: /* CSI Driver */

2023-01-07T06:53:49Z

CSI Driver

← Older revision		Revision as of 01:53, 7 January 2023
Line 51:		Line 51:
	We are using a [https://github.com/apalia/cloudstack-csi-driver CSI driver] for PersistentVolume storage.		We are using a [https://github.com/apalia/cloudstack-csi-driver CSI driver] for PersistentVolume storage.

	Installation:		Installation: <br>
			UPDATE: don't apply the manifest directly; you'll need to download and edit it first. It seems like the labels on the control plane node changed starting from v1.24. <br>
			After downloading the manifest, open it in an editor and change <code>node-role.kubernetes.io/master: ""</code> to <code>node-role.kubernetes.io/control-plane: ""</code>. <br>
			If you already applied the manifest and need to edit it, just run <code>kubectl -n kube-system edit deployment cloudstack-csi-controller</code>.
	<pre>		<pre>
	~~kubectl apply -f~~ https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml		wget https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml
			# Make necessary edits
			vim manifest.yaml
			kubectl apply -f manifest.yaml
	</pre>		</pre>

Merenber: /* Enabling a feature gate */

2023-01-07T03:51:56Z

Enabling a feature gate

← Older revision		Revision as of 22:51, 6 January 2023
Line 97:		Line 97:

	== Enabling a feature gate ==		== Enabling a feature gate ==
			UPDATE: starting from v1.23, the PodSecurity feature gate is [https://v1-23.docs.kubernetes.io/docs/concepts/security/pod-security-admission/#enabling-the-podsecurity-admission-plugin enabled by default], so there is no need to manually enable it. The rest of this section was kept for historical purposes only.

	In v1.22, the PodSecurity feature gate is an Alpha feature and must be enabled in kube-apiserver (https://kubernetes.io/docs/concepts/security/pod-security-admission/).		In v1.22, the PodSecurity feature gate is an Alpha feature and must be enabled in kube-apiserver (https://kubernetes.io/docs/concepts/security/pod-security-admission/).

Merenber: /* Kubelet certificate rotation */

2022-12-06T01:08:04Z

Kubelet certificate rotation

← Older revision		Revision as of 20:08, 5 December 2022
Line 137:		Line 137:
	See https://kubernetes.io/docs/tasks/tls/certificate-rotation/.		See https://kubernetes.io/docs/tasks/tls/certificate-rotation/.

	SSH into the control plane and make sure that `rotateCertificates: true` is set in /var/lib/kubelet/config.yaml.		SSH into the control plane and make sure that <code>rotateCertificates: true</code> is set in /var/lib/kubelet/config.yaml.

	=== What to do if the certificates expire ===		=== What to do if the certificates expire ===

Merenber: /* Admin kubeconfig */

2022-12-06T01:07:44Z

Admin kubeconfig

← Older revision		Revision as of 20:07, 5 December 2022
Line 130:		Line 130:
	</li>		</li>
	<li>		<li>
	Copy the output into <code>/root/.kube.config</code> on biloba and chamomile.		Copy the output into <code>/root/.kube/config</code> on biloba and chamomile.
	</li>		</li>
	</ol>		</ol>

Merenber at 01:07, 6 December 2022

2022-12-06T01:07:12Z

← Older revision		Revision as of 20:07, 5 December 2022
Line 106:		Line 106:

	This will <b>automatically</b> restart the kube-apiserver; wait a minute and run <code>kubectl -n kube-system get pods</code> to check.		This will <b>automatically</b> restart the kube-apiserver; wait a minute and run <code>kubectl -n kube-system get pods</code> to check.

			== Certificates ==
			=== Admin kubeconfig ===
			The kubeconfig in the CloudStack UI will only last one year (as of this writing, it is expired, so don't use it). If it expires again, here's how you can renew it:
			<ol>
			<li>SSH into the control plane VM (see instructions above)</li>
			<li>
			Create a file called e.g. kubeadm-config.yaml with this content:
			<pre>
			apiVersion: kubeadm.k8s.io/v1beta3
			kind: ClusterConfiguration
			clusterName: "kubernetes"
			controlPlaneEndpoint: "172.19.134.149:6443"
			certificatesDir: "/etc/kubernetes/pki"
			</pre>
			</li>
			<li>
			Generate a new admin kubeconfig which will last ten years:
			<pre>
			kubeadm kubeconfig user --config=kubeadm-config.yaml --client-name=kubernetes-admin --org=system:masters --validity-period=87600h0m0s
			</pre>
			Reference: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubeconfig-additional-users
			</li>
			<li>
			Copy the output into <code>/root/.kube.config</code> on biloba and chamomile.
			</li>
			</ol>

			=== Kubelet certificate rotation ===
			See https://kubernetes.io/docs/tasks/tls/certificate-rotation/.

			SSH into the control plane and make sure that `rotateCertificates: true` is set in /var/lib/kubelet/config.yaml.

			=== What to do if the certificates expire ===
			SSH into the control plane VM. To check the expired certs, run <code>kubeadm certs check-expiration</code>. From biloba or chamomile, you can also run <code>kubectl -n kube-system get cm kubeadm-config -o yaml</code> if the admin kubeconfig has not expired.

			To renew the expired certificates, run
			<pre>
			kubeadm certs renew all
			</pre>
			You now need to restart kube-apiserver, kube-controller-manager, kube-scheduler and etcd. Unfortunately I was never able to figure out how to do this. Deleting the pods doesn't seem to work. We might need to restart all of the Docker containers running on the control plane. You can check the logs of the kube-apiserver pod to see if it's still having certificate expiry issues.

			Anywho, the safe but slow option is to just restart all of the Kubernetes VMs from the CloudStack web UI.

			To be sure that everything is working again, make sure that you can create a temporary pod successfully.

	== Members ==		== Members ==

Merenber: /* CSI Driver */

2022-01-01T18:14:56Z

CSI Driver

← Older revision		Revision as of 13:14, 1 January 2022
Line 71:		Line 71:
	csi.cloudstack.apache.org/disk-offering-id: 0da1f706-fd2e-4203-8bae-1b740aef9886		csi.cloudstack.apache.org/disk-offering-id: 0da1f706-fd2e-4203-8bae-1b740aef9886
	</pre>		</pre>
	Change the disk-offering-id to the ID of the 'custom' disk size offering in CloudStack.		Change the disk-offering-id to the ID of the 'custom' disk size offering in CloudStack. Apply the YAML file once you are done editing it.

	<b>Note</b>: only a single writer is allowed, so do NOT use ReadWriteMany on any PersistentVolumeClaims.		<b>Note</b>: only a single writer is allowed, so do NOT use ReadWriteMany on any PersistentVolumeClaims.

Merenber: /* CSI Driver */

2022-01-01T18:14:17Z

CSI Driver

← Older revision		Revision as of 13:14, 1 January 2022
Line 53:		Line 53:
	Installation:		Installation:
	<pre>		<pre>
	https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml		kubectl apply -f https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml
	</pre>		</pre>

Merenber: /* Members */

2022-01-01T18:11:52Z

Members

← Older revision		Revision as of 13:11, 1 January 2022
Line 111:		Line 111:

	We are also using [https://open-policy-agent.github.io/gatekeeper/website/docs/ OPA Gatekeeper] to restrict the Ingresses which members can create. See [https://git.csclub.uwaterloo.ca/cloud/manifests/src/branch/master/cscingressconstraint-template.yaml here] and [https://git.csclub.uwaterloo.ca/cloud/manifests/src/branch/master/cscingressconstraint-constraint.yaml here] for details.		We are also using [https://open-policy-agent.github.io/gatekeeper/website/docs/ OPA Gatekeeper] to restrict the Ingresses which members can create. See [https://git.csclub.uwaterloo.ca/cloud/manifests/src/branch/master/cscingressconstraint-template.yaml here] and [https://git.csclub.uwaterloo.ca/cloud/manifests/src/branch/master/cscingressconstraint-constraint.yaml here] for details.

			=== Certificate Signing Requests ===
			We're going to set the max. CSR signing duration to 10 years so that members don't have to worry about their kubeconfig cert expiring (at least, not for a long time).

			SSH into the control node and edit /etc/kubernetes/manifests/kube-controller-manager.yml so that it has the following CLI flag:
			<pre>
			--cluster-signing-duration=87600h
			</pre>
			The controller will automatically restart after you save and close the file.