feat: Add RKE2 as alternative Kubernetes distribution for service cluster#4566
Open
j0hnL wants to merge 3 commits into
Open
feat: Add RKE2 as alternative Kubernetes distribution for service cluster#4566j0hnL wants to merge 3 commits into
j0hnL wants to merge 3 commits into
Conversation
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
3 times, most recently
from
f513dc2 to
11db64e
Compare
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
2 times, most recently
from
0759230 to
b4bff18
Compare
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
b4bff18 to
ea1db21
Compare
sujit-jadhav
approved these changes
May 27, 2026
sujit-jadhav
requested changes
May 29, 2026
sujit-jadhav
left a comment
sujit-jadhav
left a comment
Collaborator
There was a problem hiding this comment.
input/omnia_config.yml
Can we segrigate the blocks as per the k8s destribution?
service_k8s_cluster:
- cluster_name: service_cluster
deployment: true
etcd_on_local_disk: false
k8s_distro: "kubeadm"
k8s_cni: "calico"
pod_external_ip_range: "172.16.107.170-172.16.107.200"
k8s_service_addresses: "10.233.0.0/18"
k8s_pod_network_cidr: "10.233.64.0/18"
nfs_storage_name: "nfs_k8s"
k8s_crio_storage_size: "20G"
csi_powerscale_driver_secret_file_path: ""
csi_powerscale_driver_values_file_path: ""
We can add new block:
service_rke2_k8s_cluster
and have specific settings for it.
I will ask team to change the existing service_k8s_cluster to service_upstream_k8s_cluster
Same way we can add service_charmed_k8s_cluster for canonical.
service_upstream_k8s_cluster
service_rke2_k8s_cluster
service_charmed_k8s_cluster
| "cluster": [ | ||
| { "package": "docker.io/library/busybox", "type": "image", "tag": "1.36" }, | ||
| { "package": "firewalld", "type": "rpm", "repo_name": "x86_64_baseos" }, | ||
| { "package": "python3-firewall", "type": "rpm", "repo_name": "x86_64_baseos" }, |
Collaborator
There was a problem hiding this comment.
software_config.json changes required to add service_rke2 and need to add input validation for the samea
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
ea1db21 to
7cb6f0c
Compare
…ster Add support for deploying RKE2 (Rancher Kubernetes Engine 2) as an alternative to vanilla Kubernetes (kubeadm) for the service cluster. Users can choose the distribution via the new k8s_distro field in omnia_config.yml. Configuration changes: - Add k8s_distro field to service_k8s_cluster in omnia_config.yml (default: kubeadm) - Update omnia_config.json schema to validate k8s_distro enum (kubeadm/rke2) - Expand k8s_cni enum to include canal, cilium (RKE2-supported CNIs) Pipeline integration: - discovery.yml: Enable service_k8s tag when service_rke2 is in software_config.json - include_software_config.yml: Detect service_rke2 and set service_rke2_support fact - validate_software_config_json.yml: Handle service_rke2 arch and version detection - common_validation.py: Accept service_rke2 for cluster validation - image_package_collector.py: Dynamically select service_rke2.json for image builds - k8s_config/main.yml: Branch NFS setup based on k8s_distro - configure_cloud_init_group.yml: Select RKE2-specific templates when k8s_distro=rke2 New files: - 3 RKE2 cloud-init templates (first server, additional server, agent) - create_rke2_config_nfs.yml for RKE2-specific NFS directory setup - service_rke2.json package definitions for RHEL 10.0 x86_64 Key differences from kubeadm path: - RKE2 uses built-in containerd (no CRI-O) - RKE2 manages CNI lifecycle (calico/canal/cilium/flannel) - Token-based cluster join instead of kubeadm certificates - kube-vip deployed as RKE2 static pod manifest - RKE2 registries.yaml for Pulp mirror integration - Port 9345 for RKE2 supervisor API Existing kubeadm deployment is completely unaffected when k8s_distro=kubeadm (default). Signed-off-by: John Lockman <j.lockman@dell.com>
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
7cb6f0c to
f2ba1dc
Compare
- Read NFS share details from storage_config.yml 'mounts' (nfs_client_params no longer exists); use k8s_nfs_server_path in cloud-init fstab and nfs-client-provisioner values - Add rke2 (stable/1.35) and rke2-common RPM repos to local_repo_config.yml; source rke2-selinux from rke2-common - Set cluster-cidr/service-cidr on joining servers to avoid critical configuration value mismatch - Stage official RKE2 airgap image tarballs (core + CNI, v1.35.1+rke2r1) on the NFS share and import via /var/lib/rancher/rke2/agent/images so system images work offline - Deploy kube-vip as a kubelet static pod in agent/pod-manifests (not the server/manifests AddOn dir), mount kubeconfig at /etc/kubernetes/admin.conf, and place it after rke2.yaml exists to avoid hostPath dir race Signed-off-by: John Lockman <j.lockman@dell.com>
- Fix validation guard: use 'service_rke2' in tag_names (not 'service_k8s') when populating service_rke2_k8s_cluster for validation - Fix airgap image download: url used .tar.gz extension but files are .tar.zst - Add pod_external_ip_range and k8s_pod_network_cidr to required fields in omnia_config.json schema - Fix set -e placement: move to top of inline shell block in additional control plane cloud-init - Fix heredoc indentation: config.yaml content was over-indented by 2 spaces producing invalid YAML - Fix omnia_config.yml example: deployment should be true, not false - Move RKE2 repo comments inline with the two new repo entries in local_repo_config.yml - Add arch validation: error if service_rke2 is configured with non-x86_64 arch - Remove redundant inline comments across cloud-init templates and image_package_collector.py - Remove unused HELM_VERSION variable from install-helm.sh in both CP templates - Fix typo: 'nfs sever' → 'nfs server' in create_rke2_config_nfs.yml task name Signed-off-by: John Lockman <jlockman3@gmail.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
f606d62 to
174c1cb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add support for deploying RKE2 (Rancher Kubernetes Engine 2) as an alternative to vanilla Kubernetes (kubeadm) for the service cluster. The distribution is selected by listing
service_rke2(instead ofservice_k8s) insoftware_config.jsonand configuring the newservice_rke2_k8s_clusterblock inomnia_config.yml.How to use
input/software_config.json, replaceservice_k8swithservice_rke2in the softwares list and set its"version"(e.g.1.35.1)input/omnia_config.yml, uncomment and configure theservice_rke2_k8s_clusterblock withdeployment: true(only one ofservice_k8s_cluster/service_rke2_k8s_clustermay havedeployment: true)high_availability_config.yml— the kube-vip VIP is required for cluster joinlocal_repo.yml→build_image_x86_64.yml→discovery.yml)Configuration changes
service_rke2_k8s_clusterblock inomnia_config.ymlwith schema validation inomnia_config.json(supported CNIs: calico, canal, cilium, flannel)service_k8sandservice_rke2being present togetherrke2(stable/1.35 channel) andrke2-commonRPM repositories added tolocal_repo_config.ymlPipeline integration
provision.yml: enableservice_k8stag whenservice_rke2is insoftware_config.jsoninclude_software_config.yml/validate_software_config_json.yml: detectservice_rke2, set support facts, handle arch and versioncommon_validation.py: validateservice_rke2cluster configurationimage_package_collector.py: selectservice_rke2.jsonfor image buildsk8s_config/main.yml: branch NFS setup onk8s_distro;create_rke2_config_nfs.ymlreads NFS share details fromstorage_config.ymlmountsconfigure_cloud_init_group.yml: select RKE2-specific cloud-init templatesNew files
create_rke2_config_nfs.ymlfor RKE2-specific NFS setupservice_rke2.jsonpackage definitions for RHEL 10.0 x86_64Key differences from kubeadm path
/var/lib/rancher/rke2/agent/pod-manifests/) with the kubeconfig mounted at/etc/kubernetes/admin.confv1.35.1+rke2r1) are downloaded by local_repo, staged on the NFS share, and auto-imported from/var/lib/rancher/rke2/agent/images/on every noderegistries.yamlmirrors public registries to the Pulp mirror for all other imagesBackward compatibility
Existing kubeadm deployment is completely unaffected when
service_k8sis selected (the default). All branching logic only activates forservice_rke2.