kubernetes deployment strategy notes

2026-10-30 17:18:18 -07:00
parent 504f81de76
commit 35c9d9fd36
57 changed files with 1677 additions and 0 deletions
@@ -0,0 +1 @@
+I used AI (Gemini) to take a long collection of commands required to prepare a node to run kubernetes and turned that into an Ansible playbook in order to make the process faster and easier to repeat in the future.
@@ -0,0 +1,2 @@
+[defaults]
+host_key_checking = False
@@ -0,0 +1,130 @@
+---
+- name: Prepare node for Kubernetes with CRI-O
+  hosts: all
+  become: true
+  tasks:
+    - name: Disable and stop firewalld
+      ansible.builtin.systemd:
+        name: firewalld
+        state: stopped
+        enabled: false
+
+    - name: Set SELinux to permissive mode in config
+      ansible.builtin.lineinfile:
+        path: /etc/selinux/config
+        regexp: '^SELINUX='
+        line: 'SELINUX=permissive'
+
+    - name: Set SELinux to permissive runtime immediately
+      ansible.posix.selinux:
+        policy: targeted
+        state: permissive
+
+    - name: Configure kernel modules to load on boot
+      ansible.builtin.copy:
+        dest: /etc/modules-load.d/k8s.conf
+        content: |
+          overlay
+          br_netfilter
+        mode: '0644'
+
+    - name: Load kernel modules immediately
+      community.general.modprobe:
+        name: "{{ item }}"
+        state: present
+      loop:
+        - overlay
+        - br_netfilter
+
+    - name: Configure sysctl parameters for Kubernetes networking
+      ansible.posix.sysctl:
+        name: "{{ item.key }}"
+        value: "{{ item.value }}"
+        sysctl_file: /etc/sysctl.d/k8s.conf
+        state: present
+        reload: true
+      loop:
+        - { key: 'net.bridge.bridge-nf-call-iptables', value: '1' }
+        - { key: 'net.bridge.bridge-nf-call-ip6tables', value: '1' }
+        - { key: 'net.ipv4.ip_forward', value: '1' }
+
+    - name: Add Kubernetes YUM repository
+      ansible.builtin.yum_repository:
+        name: kubernetes
+        description: Kubernetes
+        baseurl: https://pkgs.k8s.io/core:/stable:/v1.37/rpm/
+        enabled: true
+        gpgcheck: true
+        gpgkey: https://pkgs.k8s.io/core:/stable:/v1.37/rpm/repodata/repomd.xml.key
+
+    - name: Add CRI-O YUM repository
+      ansible.builtin.yum_repository:
+        name: cri-o
+        description: CRI-O
+        baseurl: https://download.opensuse.org/repositories/isv:/cri-o:/stable:/v1.37/rpm/
+        enabled: true
+        gpgcheck: true
+        gpgkey: https://download.opensuse.org/repositories/isv:/cri-o:/stable:/v1.37/rpm/repodata/repomd.xml.key
+
+    - name: Install CRI-O, Podman, and Kubernetes packages
+      ansible.builtin.dnf:
+        name:
+          - cri-o
+          - cri-tools
+          - podman
+          - kubelet
+          - kubeadm
+          - kubectl
+        state: present
+
+    - name: Ensure CRI-O configuration drop-in directory exists
+      ansible.builtin.file:
+        path: /etc/crio/crio.conf.d
+        state: directory
+        mode: '0755'
+
+    - name: Configure CRI-O cgroup manager
+      ansible.builtin.copy:
+        dest: /etc/crio/crio.conf.d/10-cgroup-manager.conf
+        content: |
+          [crio.runtime]
+          conmon_cgroup = "pod"
+          cgroup_manager = "systemd"
+        mode: '0644'
+      notify: Restart crio
+
+    - name: Configure kubelet runtime endpoint
+      ansible.builtin.copy:
+        dest: /etc/sysconfig/kubelet
+        content: |
+          KUBELET_EXTRA_ARGS="--container-runtime-endpoint=unix:///var/run/crio/crio.sock"
+        mode: '0644'
+      notify: Restart kubelet
+
+    - name: Enable and start CRI-O
+      ansible.builtin.systemd:
+        name: crio
+        state: started
+        enabled: true
+        daemon_reload: true
+
+    - name: Enable and start kubelet
+      ansible.builtin.systemd:
+        name: kubelet
+        state: started
+        enabled: true
+
+    - name: Reboot the server
+      ansible.builtin.reboot:
+        reboot_timeout: 600
+
+  handlers:
+    - name: Restart crio
+      ansible.builtin.systemd:
+        name: crio
+        state: restarted
+
+    - name: Restart kubelet
+      ansible.builtin.systemd:
+        name: kubelet
+        state: restarted
@@ -0,0 +1,7 @@
+[k8s_nodes]
+control-plane.k8s.lab.cudanet.org
+compute-node-1.k8s.lab.cudanet.org
+compute-node-2.k8s.lab.cudanet.org
+
+[k8s_nodes:vars]
+ansible_user=ansible
@@ -0,0 +1 @@
+the initial deployment and configuration of ArgoCD on my cluster was a pretty straightforward helm chart deployment.  However, I did run into some issues with how ArgoCD handles SSL that I leveraged AI (Gemini) to help quickly troubleshoot. Beyond that, creating a project, adding a repo, creating an application and deploying nginx was mostly handled referring to my personal notes.
@@ -0,0 +1,19 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  annotations:
+  name: letsencrypt
+  namespace: argocd
+spec:
+  commonName: 'argocd.k8s.lab.cudanet.org'
+  dnsNames:
+  - 'argocd.k8s.lab.cudanet.org'
+  duration: 2160h0m0s
+  issuerRef:
+    kind: ClusterIssuer
+    name: letsencrypt
+  renewBefore: 360h0m0s
+  secretName: letsencrypt
+  usages:
+  - server auth
+  - client auth
@@ -0,0 +1,24 @@
+NAME: argocd
+LAST DEPLOYED: Fri Oct 30 13:32:17 2026
+NAMESPACE: argocd
+STATUS: deployed
+REVISION: 1
+DESCRIPTION: Install complete
+TEST SUITE: None
+NOTES:
+In order to access the server UI you have the following options:
+
+1. kubectl port-forward service/argocd-server -n argocd 8080:443
+
+    and then open the browser on http://localhost:8080 and accept the certificate
+
+2. enable ingress in the values file `server.ingress.enabled` and either
+      - Add the annotation for ssl passthrough: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-1-ssl-passthrough
+      - Set the `configs.params."server.insecure"` in the values file and terminate SSL at your ingress: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-2-multiple-ingress-objects-and-hosts
+
+
+After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by running:
+
+kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
+
+(You should delete the initial secret afterwards as suggested by the Getting Started Guide: https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli)
@@ -0,0 +1,37 @@
+apiVersion: traefik.io/v1alpha1
+kind: ServersTransport
+metadata:
+  name: argocd-server-transport
+  namespace: argocd
+spec:
+  insecureSkipVerify: true
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: argocd-server
+  namespace: argocd
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    # 1. gRPC routing for Argo CD CLI and API clients
+    - match: Host(`argocd.k8s.lab.cudanet.org`) && Headers(`Content-Type`, `application/grpc`)
+      kind: Rule
+      priority: 11
+      services:
+        - name: argocd-server
+          port: 443
+          scheme: https
+          serversTransport: argocd-server-transport
+    # 2. Standard Web UI traffic
+    - match: Host(`argocd.k8s.lab.cudanet.org`)
+      kind: Rule
+      priority: 10
+      services:
+        - name: argocd-server
+          port: 443
+          scheme: https
+          serversTransport: argocd-server-transport
+  tls:
+    secretName: argocd-tls-cert # Replace with your TLS Secret, or omit if using Traefik default cert / cert-resolver
@@ -0,0 +1 @@
+Ci4Jz6xJVj9Zg3ER
@@ -0,0 +1 @@
+eyJhbGciOiJSUzI1NiIsImtpZCI6InJTeEp1ZTlwbzZnUkduNk5hN2Fhc0hMTHQtRTM1RlZlcGJZX2J4OS1GSFkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzkzMzk2MzcyLCJpYXQiOjE3OTMzOTI3NzIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiMDlmZThkZWMtNmZlMy00NzMwLWEzNGYtNTlkNGNiNTE5Yzg2Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJoZWFkbGFtcCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJoZWFkbGFtcCIsInVpZCI6IjMxZjYyYzRlLWRjNjMtNDQwZi05ZWU4LTBhNDY0YTU2MTNmMyJ9fSwibmJmIjoxNzkzMzkyNzcyLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6aGVhZGxhbXA6aGVhZGxhbXAifQ.t_zHac1qGD2ea3VasvoFRUN1KpN8JR-YenjOcUdTPVIVAY4E2RAveveQKUPVQ_kI-p0NEUfs_cW_p6zZMcOMicib40BS-fVhgCTvtYo2qkmUXRR-sjvNLnk69z-QF_1d-QghCvFm0kSZosyJad_HNqiFwtwhJ5w4fYd_uiAu7zHQuBXLQsLDhvuHkkp4E5zrmZomFx-XQSdAf6YoROuVpz0aEAuAK9dMRc5tmb7u21m11OhemP1DJ43efHu0vV--xLCG738jF_yqonLm7Aj4DzS9-vyNBbaji5Ola57UQLKNeDFdnzEvtvGIenuXMLwpfE9WRfsXYUngFFJKZ0M2sA
@@ -0,0 +1,5 @@
+no AI here.  Just a copy + paste from my notes 
+
+kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
+
+just make sure that the kubeadm-config.yaml specifies the corred pod network subnet 192.168.0.0/16 for Calico.
@@ -0,0 +1 @@
+no AI here - just a quick google search.  Once cert-manager was installed via helm, adding a cluster-issuer and requesting certs came from my personal notes having used cert-manager for years.
@@ -0,0 +1,19 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  annotations:
+  name: letsencrypt
+  namespace: cert-manager
+spec:
+  commonName: '*.k8s.lab.cudanet.org'
+  dnsNames:
+  - '*.k8s.lab.cudanet.org'
+  duration: 2160h0m0s
+  issuerRef:
+    kind: ClusterIssuer
+    name: letsencrypt
+  renewBefore: 360h0m0s
+  secretName: letsencrypt
+  usages:
+  - server auth
+  - client auth
@@ -0,0 +1,19 @@
+apiVersion: v1
+data:
+  api-key: OWVjMTdmYzA0MjJlN2VlZmQxNDdmMmU2ZGE5MmQxMjY2OGZiNQ==
+kind: Secret
+metadata:
+  annotations:
+  name: cloudflare-api-key-secret
+  namespace: cert-manager
+type: Opaque
+---
+apiVersion: v1
+data:
+  api-token: NEpmdEMxRm5acGN4U2t2OWFWNk1QWEQ4SEN4eVlIblAyWDhCcTNxZg==
+kind: Secret
+metadata:
+  annotations:
+  name: cloudflare-api-token-secret
+  namespace: cert-manager
+type: Opaque
@@ -0,0 +1,22 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  annotations:
+  name: letsencrypt
+spec:
+  acme:
+    email: ryancuda@protonmail.com
+    privateKeySecretRef:
+      name: acme-key
+    server: https://acme-v02.api.letsencrypt.org/directory
+    solvers:
+    - dns01:
+        cloudflare:
+          apiKeySecretRef:
+            key: api-key
+            name: cloudflare-api-key-secret
+          email: ryancuda@protonmail.com
+        cnameStrategy: Follow
+      selector:
+        dnsZones:
+        - lab.cudanet.org
@@ -0,0 +1,11 @@
+# 1. Add the official Jetstack Helm repository
+helm repo add jetstack https://charts.jetstack.io
+
+# 2. Update your local Helm chart repository cache
+helm repo update
+
+# 3. Install cert-manager and automatically create its Custom Resource Definitions (CRDs)
+helm install cert-manager jetstack/cert-manager \
+  --namespace cert-manager \
+  --create-namespace \
+  --set crds.enabled=true
@@ -0,0 +1 @@
+no AI used here.  I've been using democratic-csi in my lab for years.  Just a simple, straightforward helm deployment.
@@ -0,0 +1,61 @@
+csiDriver:
+  name: "org.democratic-csi.iscsi"
+storageClasses:
+- name: iscsi
+  defaultClass: false
+  reclaimPolicy: Delete
+  volumeBindingMode: Immediate
+  allowVolumeExpansion: true
+  parameters:
+    fsType: ext4
+  mountOptions: []
+  secrets:
+    provisioner-secret:
+    controller-publish-secret:
+    node-stage-secret:
+    node-publish-secret:
+    controller-expand-secret:
+volumeSnapshotClasses:
+- name: iscsi-snapclass
+  parameters:
+   detachedSnapshots: "true"
+   secrets:
+     snapshotter-secret:
+driver:
+  config:
+    driver: zfs-generic-iscsi
+    driver: zfs-generic-iscsi
+    sshConnection:
+      host: 10.4.0.100
+      port: 22
+      username: root
+      password: "<changeme>"
+    zfs:
+      cli:
+        paths:
+          zfs: /usr/sbin/zfs
+          zpool: /usr/sbin/zpool
+          sudo: /usr/bin/sudo
+          chroot: /usr/sbin/chroot
+      datasetParentName: pool0/vols
+      detachedSnapshotsDatasetParentName: pool0/snaps
+      zvolCompression:
+      zvolDedup:
+      zvolEnableReservation: false
+      zvolBlocksize:
+    iscsi:
+      shareStrategy: "targetCli"
+      shareStrategyTargetCli:
+        #sudoEnabled: true
+        basename: "iqn.1994-05.com.redhat:d9e2323cca59"
+        tpg:
+          attributes:
+            authentication: 0
+            generate_node_acls: 1
+            cache_dynamic_acls: 1
+            demo_mode_write_protect: 0
+        auth:
+      targetPortal: "10.4.0.100:3260"
+      interface: ""
+      namePrefix:
+      nameSuffix:
@@ -0,0 +1,53 @@
+csiDriver:
+  name: "org.democratic-csi.nfs"
+storageClasses:
+- name: nfs
+  defaultClass: false
+  reclaimPolicy: Delete
+  volumeBindingMode: Immediate
+  allowVolumeExpansion: true
+  parameters:
+    fsType: nfs
+  mountOptions: []
+  secrets:
+    provisioner-secret:
+    controller-publish-secret:
+    node-stage-secret:
+    node-publish-secret:
+    controller-expand-secret:
+volumeSnapshotClasses:
+- name: nfs-snapclass
+  parameters:
+   detachedSnapshots: "true"
+   secrets:
+     snapshotter-secret:
+driver:
+  config:
+    driver: zfs-generic-nfs
+    sshConnection:
+      host: 10.4.0.100
+      port: 22
+      username: root
+      password: "<changeme>"
+    zfs:
+      cli:
+        paths:
+          zfs: /usr/sbin/zfs
+          zpool: /usr/sbin/zpool
+          sudo: /usr/bin/sudo
+          chroot: /usr/sbin/chroot
+      datasetParentName: pool0/k8s/nfs/volumes
+      detachedSnapshotsDatasetParentName: pool0/k8s/nfs/snapshots
+      datasetEnableQuotas: true
+      datasetEnableReservation: false
+      datasetPermissionsMode: "0777"
+      datasetPermissionsUser: 0
+      datasetPermissionsGroup: 0
+    nfs:
+      shareStrategy: "setDatasetProperties"
+      shareStrategySetDatasetProperties:
+        properties:
+          sharenfs: "rw,no_subtree_check,no_root_squash"
+          #sharenfs: "on"
+          # share: ""
+      shareHost: "10.4.0.100"
@@ -0,0 +1,16 @@
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: test-nfs
+  namespace: default
+  annotations:
+    volume.beta.kubernetes.io/storage-provisioner: org.democratic-csi.nfs
+    volume.kubernetes.io/storage-provisioner: org.democratic-csi.nfs
+spec:
+  accessModes:
+    - ReadWriteMany
+  resources:
+    requests:
+      storage: 1Gi
+  storageClassName: nfs
+  volumeMode: Filesystem
@@ -0,0 +1,31 @@
+firewall-cmd --permanent --add-masquerade
+firewall-cmd --permanent --add-port=6443/tcp        # Kubernetes API Server
+firewall-cmd --permanent --add-port=2379-2380/tcp   # etcd server client API
+firewall-cmd --permanent --add-port=10250/tcp       # Kubelet API
+firewall-cmd --permanent --add-port=10259/tcp       # kube-scheduler
+firewall-cmd --permanent --add-port=10257/tcp       # kube-controller-manager
+firewall-cmd --permanent --add-port=8472/udp        # Flannel VXLAN overlay
+firewall-cmd --permanent --add-port=51820/udp       # Flannel WireGuard (optional/fallback)
+firewall-cmd --permanent --zone=trusted --add-source=10.3.0.0/24      # Lab Node Subnet
+firewall-cmd --permanent --zone=trusted --add-source=10.244.0.0/16    # Pod CIDR
+firewall-cmd --permanent --zone=trusted --add-source=10.96.0.0/12     # Service CIDR
+firewall-cmd --permanent --zone=trusted --add-interface=cni0
+firewall-cmd --permanent --zone=trusted --add-interface=flannel.1
+firewall-cmd --reload
+firewall-cmd --permanent --add-masquerade
+firewall-cmd --permanent --add-port=6443/tcp        # API Server
+firewall-cmd --permanent --add-port=2379-2380/tcp   # etcd
+firewall-cmd --permanent --add-port=10250/tcp       # Kubelet API
+firewall-cmd --permanent --add-port=10259/tcp       # kube-scheduler
+firewall-cmd --permanent --add-port=10257/tcp       # kube-controller-manager
+firewall-cmd --permanent --add-port=179/tcp         # Calico BGP
+firewall-cmd --permanent --add-port=4789/udp        # Calico VXLAN (if used instead of IPIP)
+firewall-cmd --permanent --add-port=5473/tcp        # Typha (for large clusters)
+firewall-cmd --permanent --add-protocol=ipip        # Calico IP-in-IP encapsulation
+firewall-cmd --permanent --zone=trusted --add-source=10.4.0.0/24      # Node Network
+firewall-cmd --permanent --zone=trusted --add-source=192.168.0.0/16   # Calico Pod CIDR
+firewall-cmd --permanent --zone=trusted --add-source=10.96.0.0/12     # Service CIDR
+firewall-cmd --permanent --zone=trusted --add-interface=cali+
+firewall-cmd --permanent --zone=trusted --add-interface=tunl0
+firewall-cmd --permanent --zone=trusted --add-interface=vxlan.calico
+firewall-cmd --reload
@@ -0,0 +1,350 @@
+#!/usr/bin/env bash
+
+# Copyright The Helm Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The install script is based off of the MIT-licensed script from glide,
+# the package manager for Go: https://github.com/Masterminds/glide.sh/blob/master/get
+
+: ${BINARY_NAME:="helm"}
+: ${USE_SUDO:="true"}
+: ${DEBUG:="false"}
+: ${VERIFY_CHECKSUM:="true"}
+: ${VERIFY_SIGNATURES:="false"}
+: ${HELM_INSTALL_DIR:="/usr/local/bin"}
+: ${GPG_PUBRING:="pubring.kbx"}
+
+HAS_CURL="$(type "curl" &> /dev/null && echo true || echo false)"
+HAS_WGET="$(type "wget" &> /dev/null && echo true || echo false)"
+HAS_OPENSSL="$(type "openssl" &> /dev/null && echo true || echo false)"
+HAS_GPG="$(type "gpg" &> /dev/null && echo true || echo false)"
+HAS_GIT="$(type "git" &> /dev/null && echo true || echo false)"
+HAS_TAR="$(type "tar" &> /dev/null && echo true || echo false)"
+
+# initArch discovers the architecture for this system.
+initArch() {
+  ARCH=$(uname -m)
+  case $ARCH in
+    armv5*) ARCH="armv5";;
+    armv6*) ARCH="armv6";;
+    armv7*) ARCH="arm";;
+    aarch64) ARCH="arm64";;
+    x86) ARCH="386";;
+    x86_64) ARCH="amd64";;
+    i686) ARCH="386";;
+    i386) ARCH="386";;
+  esac
+}
+
+# initOS discovers the operating system for this system.
+initOS() {
+  OS=$(echo `uname`|tr '[:upper:]' '[:lower:]')
+
+  case "$OS" in
+    # Minimalist GNU for Windows
+    mingw*|cygwin*) OS='windows';;
+  esac
+}
+
+# runs the given command as root (detects if we are root already)
+runAsRoot() {
+  if [ $EUID -ne 0 -a "$USE_SUDO" = "true" ]; then
+    sudo "${@}"
+  else
+    "${@}"
+  fi
+}
+
+# verifySupported checks that the os/arch combination is supported for
+# binary builds, as well whether or not necessary tools are present.
+verifySupported() {
+  local supported="darwin-amd64\ndarwin-arm64\nlinux-386\nlinux-amd64\nlinux-arm\nlinux-arm64\nlinux-loong64\nlinux-ppc64le\nlinux-s390x\nlinux-riscv64\nwindows-amd64\nwindows-arm64"
+  if ! echo "${supported}" | grep -q "${OS}-${ARCH}"; then
+    echo "No prebuilt binary for ${OS}-${ARCH}."
+    echo "To build from source, go to https://github.com/helm/helm"
+    exit 1
+  fi
+
+  if [ "${HAS_CURL}" != "true" ] && [ "${HAS_WGET}" != "true" ]; then
+    echo "Either curl or wget is required"
+    exit 1
+  fi
+
+  if [ "${VERIFY_CHECKSUM}" == "true" ] && [ "${HAS_OPENSSL}" != "true" ]; then
+    echo "In order to verify checksum, openssl must first be installed."
+    echo "Please install openssl or set VERIFY_CHECKSUM=false in your environment."
+    exit 1
+  fi
+
+  if [ "${VERIFY_SIGNATURES}" == "true" ]; then
+    if [ "${HAS_GPG}" != "true" ]; then
+      echo "In order to verify signatures, gpg must first be installed."
+      echo "Please install gpg or set VERIFY_SIGNATURES=false in your environment."
+      exit 1
+    fi
+    if [ "${OS}" != "linux" ]; then
+      echo "Signature verification is currently only supported on Linux."
+      echo "Please set VERIFY_SIGNATURES=false or verify the signatures manually."
+      exit 1
+    fi
+  fi
+
+  if [ "${HAS_GIT}" != "true" ]; then
+    echo "[WARNING] Could not find git. It is required for plugin installation."
+  fi
+
+  if [ "${HAS_TAR}" != "true" ]; then
+    echo "[ERROR] Could not find tar. It is required to extract the helm binary archive."
+    exit 1
+  fi
+}
+
+# checkDesiredVersion checks if the desired version is available.
+checkDesiredVersion() {
+  if [ "x$DESIRED_VERSION" == "x" ]; then
+    # Get tag from release URL
+    # Cache behavior is provider-specific, so this mitigation is best effort.
+    # The current CDN does not revalidate on the request no-cache directive,
+    # so use a unique query while retaining no-cache for compliant intermediaries.
+    local latest_release_url="https://get.helm.sh/helm4-latest-version?ts=$(date +%s)"
+    local latest_release_response=""
+    if [ "${HAS_CURL}" == "true" ]; then
+      latest_release_response=$( curl -L --silent --show-error --fail --header "Cache-Control: no-cache" "$latest_release_url" 2>&1 || true )
+    elif [ "${HAS_WGET}" == "true" ]; then
+      latest_release_response=$( wget "$latest_release_url" --header="Cache-Control: no-cache" -q -O - 2>&1 || true )
+    fi
+    TAG=$( echo "$latest_release_response" | grep '^v[0-9]' )
+    if [ "x$TAG" == "x" ]; then
+      printf "Could not retrieve the latest release tag information from %s: %s\n" "${latest_release_url}" "${latest_release_response}"
+      exit 1
+    fi
+  else
+    TAG=$DESIRED_VERSION
+  fi
+}
+
+# checkHelmInstalledVersion checks which version of helm is installed and
+# if it needs to be changed.
+checkHelmInstalledVersion() {
+  if [[ -f "${HELM_INSTALL_DIR}/${BINARY_NAME}" ]]; then
+    local version=$("${HELM_INSTALL_DIR}/${BINARY_NAME}" version --template="{{ .Version }}")
+    if [[ "$version" == "$TAG" ]]; then
+      echo "Helm ${version} is already ${DESIRED_VERSION:-latest}"
+      return 0
+    else
+      echo "Helm ${TAG} is available. Changing from version ${version}."
+      return 1
+    fi
+  else
+    return 1
+  fi
+}
+
+# downloadFile downloads the latest binary package and also the checksum
+# for that binary.
+downloadFile() {
+  HELM_DIST="helm-$TAG-$OS-$ARCH.tar.gz"
+  DOWNLOAD_URL="https://get.helm.sh/$HELM_DIST"
+  CHECKSUM_URL="$DOWNLOAD_URL.sha256"
+  HELM_TMP_ROOT="$(mktemp -dt helm-installer-XXXXXX)"
+  HELM_TMP_FILE="$HELM_TMP_ROOT/$HELM_DIST"
+  HELM_SUM_FILE="$HELM_TMP_ROOT/$HELM_DIST.sha256"
+  echo "Downloading $DOWNLOAD_URL"
+  if [ "${HAS_CURL}" == "true" ]; then
+    curl -SsL "$CHECKSUM_URL" -o "$HELM_SUM_FILE"
+    curl -SsL "$DOWNLOAD_URL" -o "$HELM_TMP_FILE"
+  elif [ "${HAS_WGET}" == "true" ]; then
+    wget -q -O "$HELM_SUM_FILE" "$CHECKSUM_URL"
+    wget -q -O "$HELM_TMP_FILE" "$DOWNLOAD_URL"
+  fi
+}
+
+# verifyFile verifies the SHA256 checksum of the binary package
+# and the GPG signatures for both the package and checksum file
+# (depending on settings in environment).
+verifyFile() {
+  if [ "${VERIFY_CHECKSUM}" == "true" ]; then
+    verifyChecksum
+  fi
+  if [ "${VERIFY_SIGNATURES}" == "true" ]; then
+    verifySignatures
+  fi
+}
+
+# installFile installs the Helm binary.
+installFile() {
+  HELM_TMP="$HELM_TMP_ROOT/$BINARY_NAME"
+  mkdir -p "$HELM_TMP"
+  tar xf "$HELM_TMP_FILE" -C "$HELM_TMP"
+  HELM_TMP_BIN="$HELM_TMP/$OS-$ARCH/helm"
+  echo "Preparing to install $BINARY_NAME into ${HELM_INSTALL_DIR}"
+  runAsRoot cp "$HELM_TMP_BIN" "$HELM_INSTALL_DIR/$BINARY_NAME"
+  echo "$BINARY_NAME installed into $HELM_INSTALL_DIR/$BINARY_NAME"
+}
+
+# verifyChecksum verifies the SHA256 checksum of the binary package.
+verifyChecksum() {
+  printf "Verifying checksum... "
+  local sum=$(openssl sha1 -sha256 ${HELM_TMP_FILE} | awk '{print $2}')
+  local expected_sum=$(cat ${HELM_SUM_FILE})
+  if [ "$sum" != "$expected_sum" ]; then
+    echo "SHA sum of ${HELM_TMP_FILE} does not match. Aborting."
+    exit 1
+  fi
+  echo "Done."
+}
+
+# verifySignatures obtains the latest KEYS file from GitHub main branch
+# as well as the signature .asc files from the specific GitHub release,
+# then verifies that the release artifacts were signed by a maintainer's key.
+verifySignatures() {
+  printf "Verifying signatures... "
+  local keys_filename="KEYS"
+  local github_keys_url="https://raw.githubusercontent.com/helm/helm/main/${keys_filename}"
+  if [ "${HAS_CURL}" == "true" ]; then
+    curl -SsL "${github_keys_url}" -o "${HELM_TMP_ROOT}/${keys_filename}"
+  elif [ "${HAS_WGET}" == "true" ]; then
+    wget -q -O "${HELM_TMP_ROOT}/${keys_filename}" "${github_keys_url}"
+  fi
+  local gpg_keyring="${HELM_TMP_ROOT}/keyring.gpg"
+  local gpg_homedir="${HELM_TMP_ROOT}/gnupg"
+  mkdir -p -m 0700 "${gpg_homedir}"
+  local gpg_stderr_device="/dev/null"
+  if [ "${DEBUG}" == "true" ]; then
+    gpg_stderr_device="/dev/stderr"
+  fi
+  gpg --batch --quiet --homedir="${gpg_homedir}" --import "${HELM_TMP_ROOT}/${keys_filename}" 2> "${gpg_stderr_device}"
+  gpg --batch --no-default-keyring --keyring "${gpg_homedir}/${GPG_PUBRING}" --export > "${gpg_keyring}"
+  local github_release_url="https://github.com/helm/helm/releases/download/${TAG}"
+  if [ "${HAS_CURL}" == "true" ]; then
+    curl -SsL "${github_release_url}/helm-${TAG}-${OS}-${ARCH}.tar.gz.sha256.asc" -o "${HELM_TMP_ROOT}/helm-${TAG}-${OS}-${ARCH}.tar.gz.sha256.asc"
+    curl -SsL "${github_release_url}/helm-${TAG}-${OS}-${ARCH}.tar.gz.asc" -o "${HELM_TMP_ROOT}/helm-${TAG}-${OS}-${ARCH}.tar.gz.asc"
+  elif [ "${HAS_WGET}" == "true" ]; then
+    wget -q -O "${HELM_TMP_ROOT}/helm-${TAG}-${OS}-${ARCH}.tar.gz.sha256.asc" "${github_release_url}/helm-${TAG}-${OS}-${ARCH}.tar.gz.sha256.asc"
+    wget -q -O "${HELM_TMP_ROOT}/helm-${TAG}-${OS}-${ARCH}.tar.gz.asc" "${github_release_url}/helm-${TAG}-${OS}-${ARCH}.tar.gz.asc"
+  fi
+  local error_text="If you think this might be a potential security issue,"
+  error_text="${error_text}\nplease see here: https://github.com/helm/community/blob/master/SECURITY.md"
+  local num_goodlines_sha=$(gpg --verify --keyring="${gpg_keyring}" --status-fd=1 "${HELM_TMP_ROOT}/helm-${TAG}-${OS}-${ARCH}.tar.gz.sha256.asc" 2> "${gpg_stderr_device}" | grep -c -E '^\[GNUPG:\] (GOODSIG|VALIDSIG)')
+  if [[ ${num_goodlines_sha} -lt 2 ]]; then
+    echo "Unable to verify the signature of helm-${TAG}-${OS}-${ARCH}.tar.gz.sha256!"
+    echo -e "${error_text}"
+    exit 1
+  fi
+  local num_goodlines_tar=$(gpg --verify --keyring="${gpg_keyring}" --status-fd=1 "${HELM_TMP_ROOT}/helm-${TAG}-${OS}-${ARCH}.tar.gz.asc" 2> "${gpg_stderr_device}" | grep -c -E '^\[GNUPG:\] (GOODSIG|VALIDSIG)')
+  if [[ ${num_goodlines_tar} -lt 2 ]]; then
+    echo "Unable to verify the signature of helm-${TAG}-${OS}-${ARCH}.tar.gz!"
+    echo -e "${error_text}"
+    exit 1
+  fi
+  echo "Done."
+}
+
+# fail_trap is executed if an error occurs.
+fail_trap() {
+  result=$?
+  if [ "$result" != "0" ]; then
+    if [[ -n "$INPUT_ARGUMENTS" ]]; then
+      echo "Failed to install $BINARY_NAME with the arguments provided: $INPUT_ARGUMENTS"
+      help
+    else
+      echo "Failed to install $BINARY_NAME"
+    fi
+    echo -e "\tFor support, go to https://github.com/helm/helm."
+  fi
+  cleanup
+  exit $result
+}
+
+# testVersion tests the installed client to make sure it is working.
+testVersion() {
+  set +e
+  HELM="$(command -v $BINARY_NAME)"
+  if [ "$?" = "1" ]; then
+    echo "$BINARY_NAME not found. Is $HELM_INSTALL_DIR on your "'$PATH?'
+    exit 1
+  fi
+  set -e
+}
+
+# help provides possible cli installation arguments
+help () {
+  echo "Accepted cli arguments are:"
+  echo -e "\t[--help|-h ] ->> prints this help"
+  echo -e "\t[--version|-v <desired_version>] . When not defined it fetches the latest release tag from the Helm CDN"
+  echo -e "\te.g. --version v4.0.0 or -v canary"
+  echo -e "\t[--no-sudo]  ->> install without sudo"
+}
+
+# cleanup temporary files to avoid https://github.com/helm/helm/issues/2977
+cleanup() {
+  if [[ -d "${HELM_TMP_ROOT:-}" ]]; then
+    rm -rf "$HELM_TMP_ROOT"
+  fi
+}
+
+# Execution
+
+#Stop execution on any error
+trap "fail_trap" EXIT
+set -e
+
+# Set debug if desired
+if [ "${DEBUG}" == "true" ]; then
+  set -x
+fi
+
+# Parsing input arguments (if any)
+export INPUT_ARGUMENTS="${@}"
+set -u
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    '--version'|-v)
+       shift
+       if [[ $# -ne 0 ]]; then
+           export DESIRED_VERSION="${1}"
+           if [[ "$1" != "v"* ]]; then
+               echo "Expected version arg ('${DESIRED_VERSION}') to begin with 'v', fixing..."
+               export DESIRED_VERSION="v${1}"
+           fi
+       else
+           echo -e "Please provide the desired version. e.g. --version v4.0.0 or -v canary"
+           exit 0
+       fi
+       ;;
+    '--no-sudo')
+       USE_SUDO="false"
+       ;;
+    '--help'|-h)
+       help
+       exit 0
+       ;;
+    *) exit 1
+       ;;
+  esac
+  shift
+done
+set +u
+
+initArch
+initOS
+verifySupported
+checkDesiredVersion
+if ! checkHelmInstalledVersion; then
+  downloadFile
+  verifyFile
+  installFile
+fi
+testVersion
+cleanup
@@ -0,0 +1 @@
+no AI used here.  Pretty much just a copy+paste of my Gitea deployment on Openshift.  Worth noting - I actually started out with a helm chart deployment, but ran into issues with pods not spinning up, so I went back to basics and did it fully manually and worked perfectly.
@@ -0,0 +1,19 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  annotations:
+  name: letsencrypt
+  namespace: gitea
+spec:
+  commonName: 'gitea.k8s.lab.cudanet.org'
+  dnsNames:
+  - 'gitea.k8s.lab.cudanet.org'
+  duration: 2160h0m0s
+  issuerRef:
+    kind: ClusterIssuer
+    name: letsencrypt
+  renewBefore: 360h0m0s
+  secretName: letsencrypt
+  usages:
+  - server auth
+  - client auth
@@ -0,0 +1,70 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+  labels:
+    app: gitea
+    app.kubernetes.io/component: gitea
+    app.kubernetes.io/instance: gitea
+  name: gitea
+  namespace: gitea
+spec:
+  progressDeadlineSeconds: 600
+  replicas: 1
+  revisionHistoryLimit: 10
+  selector:
+    matchLabels:
+      deployment: gitea
+  strategy:
+    rollingUpdate:
+      maxSurge: 25%
+      maxUnavailable: 25%
+    type: RollingUpdate
+  template:
+    metadata:
+      annotations:
+      creationTimestamp: null
+      labels:
+        deployment: gitea
+    spec:
+      containers:
+      - env:
+        - name: GITEA__database__DB_TYPE
+          value: mysql
+        - name: GITEA__database__HOST
+          value: mariadb
+        - name: GITEA__database__NAME
+          value: gitea
+        - name: GITEA__database__USER
+          value: gitea
+        - name: GITEA__database__PASSWD
+          value: gitea
+        image: docker.gitea.com/gitea@sha256:3a54ec4b271f0479eb1f40624cbf90f2245306aab8533dd14bb5f49f4c580ccf
+        imagePullPolicy: IfNotPresent
+        name: gitea
+        ports:
+        - containerPort: 2222
+          protocol: TCP
+        - containerPort: 3000
+          protocol: TCP
+        resources: {}
+        terminationMessagePath: /dev/termination-log
+        terminationMessagePolicy: File
+        volumeMounts:
+        - mountPath: /etc/gitea
+          name: gitea-config
+        - mountPath: /var/lib/gitea
+          name: gitea-data
+      dnsPolicy: ClusterFirst
+      restartPolicy: Always
+      schedulerName: default-scheduler
+      securityContext: {}
+      terminationGracePeriodSeconds: 30
+      volumes:
+      - name: gitea-config
+        persistentVolumeClaim:
+          claimName: gitea-config
+      - name: gitea-data
+        persistentVolumeClaim:
+          claimName: gitea-data
@@ -0,0 +1,33 @@
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  annotations:
+  name: gitea-config
+  namespace: gitea
+spec:
+  accessModes:
+  - ReadWriteMany
+  resources:
+    requests:
+      storage: 100Gi
+  storageClassName: nfs
+  volumeMode: Filesystem
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  annotations:
+  finalizers:
+  - kubernetes.io/pvc-protection
+  name: gitea-data
+  namespace: gitea
+spec:
+  accessModes:
+  - ReadWriteMany
+  resources:
+    requests:
+      storage: 100Gi
+  storageClassName: nfs
+  volumeMode: Filesystem
+---
@@ -0,0 +1,28 @@
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app: gitea
+    app.kubernetes.io/component: gitea
+    app.kubernetes.io/instance: gitea
+  name: gitea
+  namespace: gitea
+spec:
+  clusterIP:
+  internalTrafficPolicy: Cluster
+  ipFamilies:
+  - IPv4
+  ipFamilyPolicy: SingleStack
+  ports:
+  - name: web
+    port: 3000
+    protocol: TCP
+    targetPort: 3000
+  - name: ssh
+    port: 2222
+    protocol: TCP
+    targetPort: 2222
+  selector:
+    deployment: gitea
+  sessionAffinity: None
+  type: ClusterIP
@@ -0,0 +1,24 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gitea-ingress
+  namespace: gitea
+  annotations:
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - gitea.k8s.lab.cudanet.org
+      secretName: letsencrypt
+  rules:
+    - host: gitea.k8s.lab.cudanet.org
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: gitea
+                port:
+                  number: 3000
@@ -0,0 +1,14 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: gitea-web
+  namespace: gitea
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`gitea.k8s.lab.cudanet.org`)
+      kind: Rule
+      services:
+        - name: gitea
+          port: 3000
@@ -0,0 +1,60 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+  labels:
+    app: mariadb
+    app.kubernetes.io/component: mariadb
+    app.kubernetes.io/instance: mariadb
+  name: mariadb
+  namespace: gitea
+spec:
+  progressDeadlineSeconds: 600
+  replicas: 1
+  revisionHistoryLimit: 10
+  selector:
+    matchLabels:
+      deployment: mariadb
+  strategy:
+    rollingUpdate:
+      maxSurge: 25%
+      maxUnavailable: 25%
+    type: RollingUpdate
+  template:
+    metadata:
+      annotations:
+      labels:
+        deployment: mariadb
+    spec:
+      containers:
+      - env:
+        - name: MYSQL_ROOT_PASSWORD
+          value: gitea
+        - name: MYSQL_USER
+          value: gitea
+        - name: MYSQL_PASSWORD
+          value: gitea
+        - name: MYSQL_DATABASE
+          value: gitea
+        image: docker.io/library/mariadb@sha256:659a15e38f4ccb2bfa2c7cf824a8a5955fdc5f63cfd01537bc25c9e0ff716605
+        imagePullPolicy: IfNotPresent
+        name: mariadb
+        ports:
+        - containerPort: 3306
+          protocol: TCP
+        resources: {}
+        terminationMessagePath: /dev/termination-log
+        terminationMessagePolicy: File
+        volumeMounts:
+        - mountPath: /var/lib/mysql
+          name: mariadb
+      dnsPolicy: ClusterFirst
+      restartPolicy: Always
+      schedulerName: default-scheduler
+      securityContext: {}
+      terminationGracePeriodSeconds: 30
+      volumes:
+      - name: mariadb
+        persistentVolumeClaim:
+          claimName: mariadb
@@ -0,0 +1,15 @@
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  annotations:
+  name: mariadb
+  namespace: gitea
+spec:
+  accessModes:
+  - ReadWriteMany
+  resources:
+    requests:
+      storage: 100Gi
+  storageClassName: nfs
+  volumeMode: Filesystem
@@ -0,0 +1,26 @@
+---
+apiVersion: v1
+kind: Service
+metadata:
+  annotations:
+  labels:
+    app: mariadb
+    app.kubernetes.io/component: mariadb
+    app.kubernetes.io/instance: mariadb
+  name: mariadb
+  namespace: gitea
+spec:
+  clusterIP:
+  internalTrafficPolicy: Cluster
+  ipFamilies:
+  - IPv4
+  ipFamilyPolicy: SingleStack
+  ports:
+  - name: 3306-tcp
+    port: 3306
+    protocol: TCP
+    targetPort: 3306
+  selector:
+    deployment: mariadb
+  sessionAffinity: None
+  type: ClusterIP
@@ -0,0 +1,8 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  annotations:
+  labels:
+    kubernetes.io/metadata.name: gitea
+  name: gitea
@@ -0,0 +1,3 @@
+no AI used in the deployment of headlamp, just a straightforward install from the instructions here https://headlamp.dev/docs/latest/installation/in-cluster/
+
+However, I did ask AI how to generate a non-expiring token since I didn't remember the exact command to do so.
@@ -0,0 +1,16 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: headlamp
+  namespace: headlamp
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`headlamp.k8s.lab.cudanet.org`)
+      kind: Rule
+      services:
+        - name: headlamp
+          port: 80
+  tls:
+    certResolver: letsencrypt
@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: headlamp-admin
+  namespace: headlamp
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: headlamp-admin-binding
+subjects:
+  - kind: ServiceAccount
+    name: headlamp-admin
+    namespace: headlamp
+roleRef:
+  kind: ClusterRole
+  name: cluster-admin
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: headlamp-admin-token
+  namespace: headlamp
+  annotations:
+    kubernetes.io/service-account.name: headlamp-admin
+type: kubernetes.io/service-account-token
@@ -0,0 +1 @@
+eyJhbGciOiJSUzI1NiIsImtpZCI6InJTeEp1ZTlwbzZnUkduNk5hN2Fhc0hMTHQtRTM1RlZlcGJZX2J4OS1GSFkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJoZWFkbGFtcCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJoZWFkbGFtcC1hZG1pbi10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJoZWFkbGFtcC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjgyMmU4ODUxLTE3OTQtNDhkZi1hYjA4LTJiY2FhMDQ0YzgyYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpoZWFkbGFtcDpoZWFkbGFtcC1hZG1pbiJ9.N7ybibaIl2_bljbsKcR-JiVCpdS15MgwurJqg9QTNchPoVYsncYFbTh4AbASLh97gbE5rYqsdXE7BnoOA70Zn_DZ9KcKI90NzFykDBUPZZaFv1rO5pIRGCdx5gGO5UXjFz3wCYhFTacEGNE9DVFrvBmeNlp8WI7xbHRM2SPG7BDFWwcxWeTXaVFoH6v3P9RRJWXec07JJEq51Un5jjmWV_gFh07pQAkUOwSYNX6ZQG6v2pRDEEAAu0cVmJI3kgvCeQBARXq7XKekGNIt0dQ6MR5B_4IronDDve3AbGLPB7cClKi4oTPD3GJT8cetAHhztIIQp0MnpW-2gNCGnTUmNg
@@ -0,0 +1 @@
+I mostly referred to my personal notes for the actual process of bootstrapping the cluster with kubeadm.  However, I did ask AI to generate an example kubeadm-config.yaml since I didn't seem to have an old copy to reference in my files.
@@ -0,0 +1,30 @@
+apiVersion: kubeadm.k8s.io/v1beta4
+kind: InitConfiguration
+localAPIEndpoint:
+  advertiseAddress: "10.4.0.10"
+  bindPort: 6443
+nodeRegistration:
+  name: "control-plane.k8s.lab.cudanet.org"
+  criSocket: "unix:///var/run/crio/crio.sock"
+  imagePullPolicy: "IfNotPresent"
+  taints:
+  - effect: "NoSchedule"
+    key: "node-role.kubernetes.io/control-plane"
+---
+apiVersion: kubeadm.k8s.io/v1beta4
+kind: ClusterConfiguration
+kubernetesVersion: "v1.37.0"
+clusterName: "kubernetes"
+controlPlaneEndpoint: "control-plane.k8s.lab.cudanet.org:6443"
+apiServer:
+  certSANs:
+  - "control-plane.k8s.lab.cudanet.org"
+  - "10.4.0.10"
+networking:
+  dnsDomain: "cluster.local"
+  serviceSubnet: "10.96.0.0/12"
+  podSubnet: "192.168.0.0/16"
+---
+apiVersion: kubeletconfiguration.k8s.io/v1beta1
+kind: KubeletConfiguration
+cgroupDriver: "systemd"
@@ -0,0 +1 @@
+no AI here, just a quick google "helm install metallb".  I've been spoiled with it just being an operator on Openshift for so long, I had to look it up.  Creating the ipaddresspool and l2advert was basically just a copy+paste from what I use on OCP.
@@ -0,0 +1,17 @@
+apiVersion: metallb.io/v1beta1
+kind: IPAddressPool
+metadata:
+  name: default-pool
+  namespace: metallb-system
+spec:
+  addresses:
+  - 10.4.0.200-10.4.0.220
+---
+apiVersion: metallb.io/v1beta1
+kind: L2Advertisement
+metadata:
+  name: l2-adv
+  namespace: metallb-system
+spec:
+  ipAddressPools:
+  - default-pool
@@ -0,0 +1,3 @@
+I did rely on AI to kludge my way through this process, because I've never had to do it before.  Thus far my experience managing RBAC on kuberneties has been leveraging OIDC with some form of SSO, like LDAP (Active Directory/FreeIPA).
+
+This was a learning experience for me.
@@ -0,0 +1,12 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: rcuda-cluster-admin-binding
+subjects:
+- kind: User
+  name: rcuda  # Must match the CN from the certificate
+  apiGroup: rbac.authorization.k8s.io
+roleRef:
+  kind: ClusterRole
+  name: cluster-admin
+  apiGroup: rbac.authorization.k8s.io
@@ -0,0 +1,18 @@
+-----BEGIN CERTIFICATE-----
+MIIC9TCCAd2gAwIBAgIQKfkUgQPfbuziunLbS0Y2BTANBgkqhkiG9w0BAQsFADAV
+MRMwEQYDVQQDEwprdWJlcm5ldGVzMB4XDTI2MTAzMDE2MTA1N1oXDTI2MTAzMTE2
+MTA1N1owEDEOMAwGA1UEAxMFcmN1ZGEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
+ggEKAoIBAQCTVTfCRu4v4LgHXsFem0KfkoSAxy498g2qCSsgDegDUUYUC7ptXNLG
+Y8McyZRlmiCGhfRKygq3zm/3da5w806/ldPZikiUuXIi2brc+2es6ey7tKQUXTtU
+/R8NmtyXVIErBwbQeHKXIH65+tJtmWhzvmvbzCIGisaldKnSQDoy3bXPUaawJm0W
+wnzsv9YWJ9JIr7RU15N9ngG3RAF+eYgxarJOZHHJK0qmPeSB6Ux9kIGlfSTNndAW
+dG0CB81kxV4FnlYQdw+/nBfoluhLAzL+VUzLUeE2ujin3o2Yy6v3Xiqyzg6Sa9y4
+1WcjtP78KuFiQOudiZlUS83bjmAcpBzbAgMBAAGjRjBEMBMGA1UdJQQMMAoGCCsG
+AQUFBwMCMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUkd2GsW22p+1bykQ+Pqb9
+JVefciIwDQYJKoZIhvcNAQELBQADggEBAIf+b5tyZYL9sx+eRVTPf0kMXGDDzG30
+eNPhiixChR+kD710OJgNrwfCSTF1/rtelOluVRYK4qs8ktp/ifczBF+AsMT1pW8/
+Ov8FVMT/cJC8P2GAR+6nK4pWYc0PyNpAyjkKq3xLnpc+NIucybPNXiFbXwGeAhFz
+ICHI529PVqxyMK+bMEs4wA3ankpkarOd3GkjX2tBhrWbAs8wFF52sRCzBHjTgj37
+sEUrYi7qEkmw4oJTr7LXw6Ajye/9GaFlhScyytr/WeBo3Mp/TcpkoxyIvGMs49wK
+5Q80UrCkZSk3zWApuMeqg/pUW7RcsZY9AQMYfe78ziK1wgLjNudH6Y4=
+-----END CERTIFICATE-----
@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE REQUEST-----
+MIICVTCCAT0CAQAwEDEOMAwGA1UEAwwFcmN1ZGEwggEiMA0GCSqGSIb3DQEBAQUA
+A4IBDwAwggEKAoIBAQCTVTfCRu4v4LgHXsFem0KfkoSAxy498g2qCSsgDegDUUYU
+C7ptXNLGY8McyZRlmiCGhfRKygq3zm/3da5w806/ldPZikiUuXIi2brc+2es6ey7
+tKQUXTtU/R8NmtyXVIErBwbQeHKXIH65+tJtmWhzvmvbzCIGisaldKnSQDoy3bXP
+UaawJm0Wwnzsv9YWJ9JIr7RU15N9ngG3RAF+eYgxarJOZHHJK0qmPeSB6Ux9kIGl
+fSTNndAWdG0CB81kxV4FnlYQdw+/nBfoluhLAzL+VUzLUeE2ujin3o2Yy6v3Xiqy
+zg6Sa9y41WcjtP78KuFiQOudiZlUS83bjmAcpBzbAgMBAAGgADANBgkqhkiG9w0B
+AQsFAAOCAQEAhPbV9ue8FWhX+rlW7qT+4qruzYKFSn5OVZ6qMOAJr4vQwJjQhQbT
+MWLxh2nBH1Q56iW1lUVU/FjvyGsHsoPEvNKWxYtBeGeICjllpkcDyCtB0UAHhj0X
+efFghXM5kzh6YhgjDiKIajx4m7C0GFZBYvyxo8gYfpp+PBgaWsSztp7dzxQdU74k
+sGQ69R/4yQY7uVYgHNCB7shN6jT/XUWGBypOtYejP8cdKf0ZecO1PL8DvsS4jVzL
+fVNVEvWf0DlaEJdIJEvFwZNxfqf7UdXARCReZBKs5rYPFBsyjiL4y3v43WzTzZKy
+ECLNLgDMicf00nTZ2+ob4q8vJ4kK1cqnXg==
+-----END CERTIFICATE REQUEST-----
@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCTVTfCRu4v4LgH
+XsFem0KfkoSAxy498g2qCSsgDegDUUYUC7ptXNLGY8McyZRlmiCGhfRKygq3zm/3
+da5w806/ldPZikiUuXIi2brc+2es6ey7tKQUXTtU/R8NmtyXVIErBwbQeHKXIH65
+tJtmWhzvmvbzCIGisaldKnSQDoy3bXPUaawJm0Wwnzsv9YWJ9JIr7RU15N9ngG3
+RAF+eYgxarJOZHHJK0qmPeSB6Ux9kIGlfSTNndAWdG0CB81kxV4FnlYQdw+/nBfo
+luhLAzL+VUzLUeE2ujin3o2Yy6v3Xiqyzg6Sa9y41WcjtP78KuFiQOudiZlUS83b
+jmAcpBzbAgMBAAECggEAGJQN0j4s/BW7/ZbEIXk7Ow2hRy5zO2hG+UfARv+q5erI
+Cw4N0rK6gtgfs4y5Ud3NcUOlhliAQzuQGDMsn7EnQPTIu7DC/PSW4tN5xeW23PTF
+RmcTPEZ1YG89n/JmvGQfjEMG0v5U93s3SOVb92axMyXgLnG2d2rtSqrUrbuDOfrb
+yxtpE0quBULpKuIdHwxOxntXo4aTAYX1lcS85U4bQXNQ1/7IqSgpHd2eqV0eatHE
+LEMhZ0Iicot//VXvWx/zDeAX7nwPSAhtzRlMR2LEh5eYj3lHNd2XxrhYBqhtbDqZ
+f2GYuxTkvz5QcGLirYkhOrGC7kZ4Lh9VxGBc6lAtmQKBgQDGLa75VAIBjMcUA7cq
+JIf26+IdmMIsbhW/kb4unXkMFLvHMDFJiNks6k9WQymbg2vxuCUeB82CRyahtSDv
+kUFgyT+LKRnakiYWnvdEtsC85z5CrpCvW6jQPnKvnecbD/2obxuoco8K6X4IaN/K
+EOFmrZbTRSC5oVXcV3ZhhTYzZQKBgQC+UcjLCL4WcvFmwso/CQedSxkrtMvjrMRi
+7zcJ+WaQBoTbihtCGmpeOCgNKBx90OdyxjutpREB55w5GUvROjVJOQS34IDr7qT0
+6XRVzP5xEiV7stPRPZq5TfPMlRh/15Iz3sxFL8U4jaLqVtF7b66RoiwhG5E1MrPq
+K5qlMwWrPwKBgQCvpbHdnk2g3gtce+aN3Y78lRj2Lnx7MJ7qCdzMKBovUPkVlGs6
+x6XlH09uH1zzTsoPlaygLz5AZ5uf24/zhYSyqew+BRG7HanQhtiV+HxfcX1HUL1A
+DzdSSucPLB63Z7bk4IVdCRwrCcQA5CLlT4zXmqeBXVWh8ZLOFl8AFDdz2QKBgQCI
+3NVxtKrGCAizZlb8t6ijI7NJK+4Hn2Fm6DIDWTe5UQbPveRWSBrAANBEjdjXXZhj
+gEzAVdkCLqW4pglQv/Q74OYUuTzc7bPwb8X6Qb2kYHFWIEaUzSFsKrOtJncE0GhS
+JuFsy6qfPbOvo5lnk+vMzPvBH0rwH7MkLhE6xZNJDwKBgBTMnQJxWCKYOGd0RKVv
+6MisPvd/nrVvaDaoozCXrWQ+SHa6qZDIHIEQOkpRi5x13A3UgjRfrNWTqinTeKAp
+Ue7G+mdfmd+HUVYS5/qJPnnoPsswJ9G9xbGUeo06U5ivs4qj4Rl5OU5nEgwKHpuq
+GBb9eonSCVDPpqhJL/50jZ+e
+-----END PRIVATE KEY-----
@@ -0,0 +1,247 @@
+#!/bin/bash
+set -e
+
+echo "# Running [control-plane.k8s.lab.cudanet.org] host initial configuration"
+
+
+exit_and_cancel_build() {
+  echo 'Host [control-plane.k8s.lab.cudanet.org] initial configuration failed'
+
+  SSL_CA_CERT=$(mktemp)
+  cat << EOF > $SSL_CA_CERT
+  -----BEGIN CERTIFICATE-----
+  MIIG6TCCBNGgAwIBAgIUIacbh/WYmORFpiddBg25g/W4jlQwDQYJKoZIhvcNAQEL
+  BQAwgYExCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEQMA4G
+  A1UEBwwHUmFsZWlnaDEQMA4GA1UECgwHS2F0ZWxsbzEUMBIGA1UECwwLU29tZU9y
+  Z1VuaXQxHzAdBgNVBAMMFnNhdGwwMS5sYWIuY3VkYW5ldC5vcmcwHhcNMjYwOTEx
+  MTc1NTA1WhcNMzgwMTE3MTc1NTA1WjCBgTELMAkGA1UEBhMCVVMxFzAVBgNVBAgM
+  Dk5vcnRoIENhcm9saW5hMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQKDAdLYXRl
+  bGxvMRQwEgYDVQQLDAtTb21lT3JnVW5pdDEfMB0GA1UEAwwWc2F0bDAxLmxhYi5j
+  dWRhbmV0Lm9yZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAN6uVBDW
+  Qp941uKRabwV2vtuGftmply2vA4cFbw0k1vAosRPE130ULNkhm8wK7qwu+bqNKi6
+  2okdfIBAm+u7Tw0NYufcMDyEjngqMCG54w3OGbLW4ZQIefIliKw/kdUhxvKvzxNT
+  DP7kK+uwnunW6l79G8azAtDddhNa0wafkMHpQYeU/Y650oto+c3qjETTIC3M2QVr
+  N7D5niWAmrLvHyR/2S/W2Oa2929PmENMuBUsEgcoYthagDjse7G2hp8Gll4cCVh4
+  LysBzTUFUBC5NRx7eqf0iCycexXcAZunHQfNCwHJNXdUFc61pBeFmTtslxTMdNmV
+  FZ9SV5++ESHK75RYlhnfHGSiByUtstDXGT1vCUkipmzI4p3zVRPj5whlg0i15hdz
+  R58q9uftSYOwr5od3UIXJQM2fNlip7rIy85av2le3TkwmjObQqP7nNmacWarwQfs
+  dTVcslv+CI7vXtdWGqcRMo0db+QQN8tSGOfSTGDqt4lmwk7ESg48UocOAhoOEzIY
+  pEQnBLtSxEDBsFEu4vwNlq5DanSD3vuxwyDC6T8AdxTQvVclM+x+CsANaVzrLM6W
+  e7KJOYfe7J/dM28OWOTdLSSJ+L9gYxtdXme9uYLmQptce7KnuwoHozRY/aP0hFN/
+  2hMNEYlPFkUGvvb+hKzziagh+Rky9IP8/dWjAgMBAAGjggFVMIIBUTAMBgNVHRME
+  BTADAQH/MAsGA1UdDwQEAwIBpjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH
+  AwIwEQYJYIZIAYb4QgEBBAQDAgJEMDUGCWCGSAGG+EIBDQQoFiZLYXRlbGxvIFNT
+  TCBUb29sIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUepliLqnv9s1n
+  TKkZeHtZA8eNPdgwgasGA1UdIwSBozCBoKGBh6SBhDCBgTELMAkGA1UEBhMCVVMx
+  FzAVBgNVBAgMDk5vcnRoIENhcm9saW5hMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYD
+  VQQKDAdLYXRlbGxvMRQwEgYDVQQLDAtTb21lT3JnVW5pdDEfMB0GA1UEAwwWc2F0
+  bDAxLmxhYi5jdWRhbmV0Lm9yZ4IUIacbh/WYmORFpiddBg25g/W4jlQwDQYJKoZI
+  hvcNAQELBQADggIBAGeB30OcUUdVO5gTWnNorYHhkhxuWri1ThCQqzBlaUKFQ35I
+  120CcZ2rc22Q/kQqLHfB3bvGEe379Bw/6kRKOZVf7cd34yqqORSQxWqkPSQIjKfw
+  K5kXRn2aCQSh/Ohfoowz0ow5HngTzShiL+KlPuix6Zuz2R+1uZwpDZifUiUVydnx
+  Ru0XiZBFX+u4zUAPJHYC94t14Omvpo8J1qbfC9Wc9Anlx4I7PXqJ8rygSrxU8him
+  QhExutff7TC4yA2nPVLvGJV96eAl9nQdiux8EKhvJ4y66/e4IqFNVYtiyCpH2epv
+  rLRgWCmpBDONHL2LM82HJZXKtU8rwRUCsKnt2pjRR2BTEkCH5GNQasLI76TEorV3
+  Z+d8LWBHZgmsbR2BcCv7eSd29NAMoRalCx1emEJm/yizzXm3ADlH9PWR9bwuwMxx
+  IRcB1EsKXpWlzaJY3TfJ8p1cUqcx3Dm8majSb9nkTkbH5XuVf2hipz1URpYlcDjk
+  DUcFWbs+2BqLU9Gg4mZJMHc6XDsduv+sjaA5D8w5bYZfgqLA39/LPKPQCQ2vcYIm
+  CviivD2WpDXdkp2BFNV3lX/H6Vnlc2Mc6utUxMem2Ip7bBISIGayZKHyjOsAU+hh
+  T/giRQe8jtPFNh+pnEWBo83XMalqn0V5C7m/Jwb28bc7IPtPPlRcfYsnhY5y
+  -----END CERTIFICATE-----
+  
+EOF
+  if [ -x /usr/bin/curl ]; then
+    /usr/bin/curl -o /dev/null -H 'Content-Type: text/plain' --fail --data 'Host initial configuration failed, please see the registration log for more details.' --noproxy \* --cacert $SSL_CA_CERT --silent 'https://satl01.lab.cudanet.org/unattended/failed?token=10fd076d-92f5-4f80-83ad-3cb330ac4069'
+  elif [ -x /usr/bin/wget ]; then
+    /usr/bin/wget -q -O /dev/null --method POST --header 'Content-Type: text/plain' --body-data='Host initial configuration failed, please see the registration log for more details.' --no-proxy --ca-certificate $SSL_CA_CERT 'https://satl01.lab.cudanet.org/unattended/failed?token=10fd076d-92f5-4f80-83ad-3cb330ac4069'
+  else
+    wget -q -O /dev/null --header 'Content-Type: text/plain' 'https://satl01.lab.cudanet.org/unattended/failed?token=10fd076d-92f5-4f80-83ad-3cb330ac4069'
+  fi
+  FINAL_STATUS=$?
+  
+  if [ -x "$(command -v subscription-manager)" ] ; then
+    subscription-manager facts --update
+    SUB_MAN_STATUS=$?
+  
+    if [ $FINAL_STATUS -eq 0 ]; then
+      FINAL_STATUS=$SUB_MAN_STATUS
+    fi
+  fi
+  
+  (exit $FINAL_STATUS);
+  exit 1
+}
+
+set +e
+trap 'exit_and_cancel_build' ERR
+
+
+
+
+# Select package manager for the OS (sets the $PKG_MANAGER* variables)
+if [ -z "$PKG_MANAGER" ]; then
+  if [ -f /etc/os-release ] ; then
+    . /etc/os-release
+  fi
+  
+  if [ "${NAME%.*}" = 'FreeBSD' ]; then
+    PKG_MANAGER='pkg'
+    PKG_MANAGER_INSTALL="${PKG_MANAGER} install -y"
+    PKG_MANAGER_REMOVE="${PKG_MANAGER} delete -y"
+    PKG_MANAGER_UPGRADE="${PKG_MANAGER} install -y"
+  elif [ -f /etc/fedora-release -o -f /etc/redhat-release -o -f /etc/amazon-linux-release -o -f /etc/system-release ]; then
+    PKG_MANAGER='dnf'
+    if [ -f /etc/redhat-release -a "${VERSION_ID%.*}" -le 7 ]; then
+      PKG_MANAGER='yum'
+    elif [ -f /etc/system-release ]; then
+      PKG_MANAGER='yum'
+    fi
+    PKG_MANAGER_INSTALL="${PKG_MANAGER} install -y"
+    PKG_MANAGER_REMOVE="${PKG_MANAGER} remove -y"
+    PKG_MANAGER_UPGRADE="${PKG_MANAGER} upgrade -y"
+  elif [ -f /etc/debian_version ]; then
+    PKG_MANAGER='apt-get'
+    PKG_MANAGER_INSTALL="${PKG_MANAGER} -o 'Dpkg::Options::=--force-confdef' -o 'Dpkg::Options::=--force-confold' install -y"
+    PKG_MANAGER_REMOVE="${PKG_MANAGER} -o 'Dpkg::Options::=--force-confdef' -o 'Dpkg::Options::=--force-confold' remove -y"
+    PKG_MANAGER_UPGRADE="${PKG_MANAGER} -o 'Dpkg::Options::=--force-confdef' -o 'Dpkg::Options::=--force-confold' -o APT::Get::Upgrade-Allow-New='true' upgrade -y"
+  elif [ -f /etc/arch-release ]; then
+    PKG_MANAGER='pacman'
+    PKG_MANAGER_INSTALL="${PKG_MANAGER} --noconfirm -S"
+    PKG_MANAGER_REMOVE="${PKG_MANAGER} --noconfirm -R"
+    PKG_MANAGER_UPGRADE="${PKG_MANAGER} --noconfirm -S"
+  elif [ x$ID = xopensuse-tumbleweed -o x$ID = xsles ]; then
+    PKG_MANAGER='zypper'
+    PKG_MANAGER_INSTALL="${PKG_MANAGER} --non-interactive install --auto-agree-with-licenses"
+    PKG_MANAGER_REMOVE="${PKG_MANAGER} --non-interactive remove"
+    PKG_MANAGER_UPGRADE="${PKG_MANAGER} --non-interactive update"
+  fi
+fi
+
+
+
+
+user_exists=false
+getent passwd root >/dev/null 2>&1 && user_exists=true
+
+
+if $user_exists; then
+
+
+  mkdir -p ~root/.ssh
+
+  cat << EOF >> ~root/.ssh/authorized_keys
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHN+jdbQeYICJHV6iVLv/dj+oqeAapadgAUM8y7ZRRvS foreman-proxy@satl01.lab.cudanet.org
+EOF
+
+  chmod 0700 ~root/.ssh
+  chmod 0600 ~root/.ssh/authorized_keys
+  chown -R root: ~root/.ssh
+
+  # Restore SELinux context with restorecon, if it's available:
+  command -v restorecon && restorecon -RvF ~root/.ssh || true
+
+
+
+else
+  echo 'The remote_execution_ssh_user does not exist and remote_execution_create_user is not set to true.  remote_execution_ssh_keys snippet will not install keys'
+fi
+
+
+
+
+echo '#'
+echo '# Installing Insights client'
+echo '#'
+
+yum install -y insights-client
+insights-client --register < /dev/null
+
+
+
+
+if command -v subscription-manager &>/dev/null; then
+  echo "Refreshing subscription data"
+  subscription-manager refresh
+fi
+
+
+
+
+
+# Call home to exit build mode
+
+trap - ERR
+SSL_CA_CERT=$(mktemp)
+cat << EOF > $SSL_CA_CERT
+-----BEGIN CERTIFICATE-----
+MIIG6TCCBNGgAwIBAgIUIacbh/WYmORFpiddBg25g/W4jlQwDQYJKoZIhvcNAQEL
+BQAwgYExCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEQMA4G
+A1UEBwwHUmFsZWlnaDEQMA4GA1UECgwHS2F0ZWxsbzEUMBIGA1UECwwLU29tZU9y
+Z1VuaXQxHzAdBgNVBAMMFnNhdGwwMS5sYWIuY3VkYW5ldC5vcmcwHhcNMjYwOTEx
+MTc1NTA1WhcNMzgwMTE3MTc1NTA1WjCBgTELMAkGA1UEBhMCVVMxFzAVBgNVBAgM
+Dk5vcnRoIENhcm9saW5hMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQKDAdLYXRl
+bGxvMRQwEgYDVQQLDAtTb21lT3JnVW5pdDEfMB0GA1UEAwwWc2F0bDAxLmxhYi5j
+dWRhbmV0Lm9yZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAN6uVBDW
+Qp941uKRabwV2vtuGftmply2vA4cFbw0k1vAosRPE130ULNkhm8wK7qwu+bqNKi6
+2okdfIBAm+u7Tw0NYufcMDyEjngqMCG54w3OGbLW4ZQIefIliKw/kdUhxvKvzxNT
+DP7kK+uwnunW6l79G8azAtDddhNa0wafkMHpQYeU/Y650oto+c3qjETTIC3M2QVr
+N7D5niWAmrLvHyR/2S/W2Oa2929PmENMuBUsEgcoYthagDjse7G2hp8Gll4cCVh4
+LysBzTUFUBC5NRx7eqf0iCycexXcAZunHQfNCwHJNXdUFc61pBeFmTtslxTMdNmV
+FZ9SV5++ESHK75RYlhnfHGSiByUtstDXGT1vCUkipmzI4p3zVRPj5whlg0i15hdz
+R58q9uftSYOwr5od3UIXJQM2fNlip7rIy85av2le3TkwmjObQqP7nNmacWarwQfs
+dTVcslv+CI7vXtdWGqcRMo0db+QQN8tSGOfSTGDqt4lmwk7ESg48UocOAhoOEzIY
+pEQnBLtSxEDBsFEu4vwNlq5DanSD3vuxwyDC6T8AdxTQvVclM+x+CsANaVzrLM6W
+e7KJOYfe7J/dM28OWOTdLSSJ+L9gYxtdXme9uYLmQptce7KnuwoHozRY/aP0hFN/
+2hMNEYlPFkUGvvb+hKzziagh+Rky9IP8/dWjAgMBAAGjggFVMIIBUTAMBgNVHRME
+BTADAQH/MAsGA1UdDwQEAwIBpjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH
+AwIwEQYJYIZIAYb4QgEBBAQDAgJEMDUGCWCGSAGG+EIBDQQoFiZLYXRlbGxvIFNT
+TCBUb29sIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUepliLqnv9s1n
+TKkZeHtZA8eNPdgwgasGA1UdIwSBozCBoKGBh6SBhDCBgTELMAkGA1UEBhMCVVMx
+FzAVBgNVBAgMDk5vcnRoIENhcm9saW5hMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYD
+VQQKDAdLYXRlbGxvMRQwEgYDVQQLDAtTb21lT3JnVW5pdDEfMB0GA1UEAwwWc2F0
+bDAxLmxhYi5jdWRhbmV0Lm9yZ4IUIacbh/WYmORFpiddBg25g/W4jlQwDQYJKoZI
+hvcNAQELBQADggIBAGeB30OcUUdVO5gTWnNorYHhkhxuWri1ThCQqzBlaUKFQ35I
+120CcZ2rc22Q/kQqLHfB3bvGEe379Bw/6kRKOZVf7cd34yqqORSQxWqkPSQIjKfw
+K5kXRn2aCQSh/Ohfoowz0ow5HngTzShiL+KlPuix6Zuz2R+1uZwpDZifUiUVydnx
+Ru0XiZBFX+u4zUAPJHYC94t14Omvpo8J1qbfC9Wc9Anlx4I7PXqJ8rygSrxU8him
+QhExutff7TC4yA2nPVLvGJV96eAl9nQdiux8EKhvJ4y66/e4IqFNVYtiyCpH2epv
+rLRgWCmpBDONHL2LM82HJZXKtU8rwRUCsKnt2pjRR2BTEkCH5GNQasLI76TEorV3
+Z+d8LWBHZgmsbR2BcCv7eSd29NAMoRalCx1emEJm/yizzXm3ADlH9PWR9bwuwMxx
+IRcB1EsKXpWlzaJY3TfJ8p1cUqcx3Dm8majSb9nkTkbH5XuVf2hipz1URpYlcDjk
+DUcFWbs+2BqLU9Gg4mZJMHc6XDsduv+sjaA5D8w5bYZfgqLA39/LPKPQCQ2vcYIm
+CviivD2WpDXdkp2BFNV3lX/H6Vnlc2Mc6utUxMem2Ip7bBISIGayZKHyjOsAU+hh
+T/giRQe8jtPFNh+pnEWBo83XMalqn0V5C7m/Jwb28bc7IPtPPlRcfYsnhY5y
+-----END CERTIFICATE-----
+
+EOF
+if [ -x /usr/bin/curl ]; then
+  /usr/bin/curl -o /dev/null -H 'Content-Type: text/plain' --fail --noproxy \* --cacert $SSL_CA_CERT --silent 'https://satl01.lab.cudanet.org/unattended/built?token=10fd076d-92f5-4f80-83ad-3cb330ac4069'
+elif [ -x /usr/bin/wget ]; then
+  /usr/bin/wget -q -O /dev/null --method POST --header 'Content-Type: text/plain' --no-proxy --ca-certificate $SSL_CA_CERT 'https://satl01.lab.cudanet.org/unattended/built?token=10fd076d-92f5-4f80-83ad-3cb330ac4069'
+else
+  wget -q -O /dev/null --header 'Content-Type: text/plain' 'https://satl01.lab.cudanet.org/unattended/built?token=10fd076d-92f5-4f80-83ad-3cb330ac4069'
+fi
+FINAL_STATUS=$?
+
+if [ -x "$(command -v subscription-manager)" ] ; then
+  subscription-manager facts --update
+  SUB_MAN_STATUS=$?
+
+  if [ $FINAL_STATUS -eq 0 ]; then
+    FINAL_STATUS=$SUB_MAN_STATUS
+  fi
+fi
+
+(exit $FINAL_STATUS);
+
+
+if [[ $? == 0 ]] ; then
+  echo "Host [control-plane.k8s.lab.cudanet.org] successfully configured."
+else
+  echo "ERROR: Host [control-plane.k8s.lab.cudanet.org] configured, but failed to report status to Foreman." >&2
+  exit 1
+fi
+
+subscription-manager repos
+
+exit 0
@@ -0,0 +1 @@
+no AI here.  literally just followed the instructions https://goteleport.com/docs/installation/self-hosted/helm-deployments/kubernetes-cluster/
@@ -0,0 +1,19 @@
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  annotations:
+  name: letsencrypt
+  namespace: teleport-cluster
+spec:
+  commonName: 'teleport.k8s.lab.cudanet.org'
+  dnsNames:
+  - 'teleport.k8s.lab.cudanet.org'
+  duration: 2160h0m0s
+  issuerRef:
+    kind: ClusterIssuer
+    name: letsencrypt
+  renewBefore: 360h0m0s
+  secretName: letsencrypt
+  usages:
+  - server auth
+  - client auth
@@ -0,0 +1,22 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: teleport-ingress
+  namespace: teleport-cluster
+  annotations:
+spec:
+  tls:
+    - hosts:
+        - teleport.k8s.lab.cudanet.org
+      secretName: letsencrypt
+  rules:
+    - host: teleport.k8s.lab.cudanet.org
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: teleport-cluster
+                port:
+                  number: 443
@@ -0,0 +1,26 @@
+---
+apiVersion: traefik.io/v1alpha1
+kind: ServersTransport
+metadata:
+  name: teleport-backend-transport
+  namespace: teleport-cluster
+spec:
+  insecureSkipVerify: true
+---
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: teleport-web
+  namespace: teleport-cluster
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`teleport.k8s.lab.cudanet.org`)
+      kind: Rule
+      services:
+        - name: teleport-cluster
+          port: 443
+          scheme: https # Crucial if Teleport expects HTTPS internally
+  tls:
+    secretName: letsencrypt
@@ -0,0 +1,10 @@
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: teleport-headers
+  namespace: teleport-cluster
+spec:
+  headers:
+    customRequestHeaders:
+      X-Forwarded-Proto: "https"
+      X-Forwarded-Ssl: "on"
@@ -0,0 +1,3 @@
+clusterName: clusterName
+proxyListenerMode: multiplex
+acme: false
@@ -0,0 +1,3 @@
+I did rely on AI to help deploy Traefik since it had been several years since the last time I'd done so.  Traefik has a lot of "knobs to fiddle with" when deploying and I wanted the simplest, most straightforward way to handle it.  Specifically, I wanted Openshift wildcard route style ingress eg; my-app.k8s.lab.cudanet.org, vs. the other way Traefik is often configured with apps as subpaths, eg; traefik.k8s.lab.cudanet.org/my-app.  Not what I wanted.
+
+Once you meet the requirements for Traefik - ie; need a LoadBalancer provider like MetalLB.  Need CSI set up to provision PVCs, Traefik is pretty straightforward.  You do also have to create your wildcard DNS entry once your LB is provisioned though.
@@ -0,0 +1,43 @@
+# Enable API & Dashboard
+api:
+  dashboard: true
+  insecure: true
+
+# Define ports
+ports:
+  web:
+    port: 8000
+    exposedPort: 80
+  websecure:
+    port: 8443
+    exposedPort: 443
+
+# Enable SafeNaming and allow cross-namespace routing
+providers:
+  kubernetesCRD:
+    allowCrossNamespace: true
+
+# If you want to enable the dashboard via IngressRoute
+ingressRoute:
+  dashboard:
+    enabled: true
+
+# Traefik static arguments for HTTP->HTTPS redirect & Let's Encrypt DNS-01 (Cloudflare)
+additionalArguments:
+  - "--providers.kubernetescrd.safenaming=true"
+  - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
+  - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
+  - "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
+  - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
+  - "--certificatesresolvers.letsencrypt.acme.email=admin@cudanet.org"
+  - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
+
+env:
+  - name: CF_DNS_API_TOKEN
+    value: "<changeme>"
+
+# Match existing 128Mi size to satisfy immutability checks
+persistence:
+  enabled: true
+  path: /data
+  size: 1Gi
@@ -0,0 +1,20 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: traefik-dashboard-wildcard
+  namespace: traefik
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`traefik.k8s.lab.cudanet.org`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
+      kind: Rule
+      services:
+        - name: api@internal
+          kind: TraefikService
+  tls:
+    certResolver: letsencrypt
+    domains:
+      - main: "*.k8s.lab.cudanet.org"
+        sans:
+          - "k8s.lab.cudanet.org"
				`@@ -0,0 +1 @@`
				`I used AI (Gemini) to take a long collection of commands required to prepare a node to run kubernetes and turned that into an Ansible playbook in order to make the process faster and easier to repeat in the future.`
				`@@ -0,0 +1 @@`
				`the initial deployment and configuration of ArgoCD on my cluster was a pretty straightforward helm chart deployment. However, I did run into some issues with how ArgoCD handles SSL that I leveraged AI (Gemini) to help quickly troubleshoot. Beyond that, creating a project, adding a repo, creating an application and deploying nginx was mostly handled referring to my personal notes.`
				`@@ -0,0 +1 @@`
				eyJhbGciOiJSUzI1NiIsImtpZCI6InJTeEp1ZTlwbzZnUkduNk5hN2Fhc0hMTHQtRTM1RlZlcGJZX2J4OS1GSFkifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzkzMzk2MzcyLCJpYXQiOjE3OTMzOTI3NzIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiMDlmZThkZWMtNmZlMy00NzMwLWEzNGYtNTlkNGNiNTE5Yzg2Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJoZWFkbGFtcCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJoZWFkbGFtcCIsInVpZCI6IjMxZjYyYzRlLWRjNjMtNDQwZi05ZWU4LTBhNDY0YTU2MTNmMyJ9fSwibmJmIjoxNzkzMzkyNzcyLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6aGVhZGxhbXA6aGVhZGxhbXAifQ.t_zHac1qGD2ea3VasvoFRUN1KpN8JR-YenjOcUdTPVIVAY4E2RAveveQKUPVQ_kI-p0NEUfs_cW_p6zZMcOMicib40BS-fVhgCTvtYo2qkmUXRR-sjvNLnk69z-QF_1d-QghCvFm0kSZosyJad_HNqiFwtwhJ5w4fYd_uiAu7zHQuBXLQsLDhvuHkkp4E5zrmZomFx-XQSdAf6YoROuVpz0aEAuAK9dMRc5tmb7u21m11OhemP1DJ43efHu0vV--xLCG738jF_yqonLm7Aj4DzS9-vyNBbaji5Ola57UQLKNeDFdnzEvtvGIenuXMLwpfE9WRfsXYUngFFJKZ0M2sA
				`@@ -0,0 +1 @@`
				`no AI here - just a quick google search. Once cert-manager was installed via helm, adding a cluster-issuer and requesting certs came from my personal notes having used cert-manager for years.`
				`@@ -0,0 +1 @@`
				`no AI used here. I've been using democratic-csi in my lab for years. Just a simple, straightforward helm deployment.`
				`@@ -0,0 +1 @@`
				`no AI used here. Pretty much just a copy+paste of my Gitea deployment on Openshift. Worth noting - I actually started out with a helm chart deployment, but ran into issues with pods not spinning up, so I went back to basics and did it fully manually and worked perfectly.`
				`@@ -0,0 +1 @@`
				`I mostly referred to my personal notes for the actual process of bootstrapping the cluster with kubeadm. However, I did ask AI to generate an example kubeadm-config.yaml since I didn't seem to have an old copy to reference in my files.`
				`@@ -0,0 +1 @@`
				`no AI here, just a quick google "helm install metallb". I've been spoiled with it just being an operator on Openshift for so long, I had to look it up. Creating the ipaddresspool and l2advert was basically just a copy+paste from what I use on OCP.`
				`@@ -0,0 +1 @@`
				`no AI here. literally just followed the instructions https://goteleport.com/docs/installation/self-hosted/helm-deployments/kubernetes-cluster/`