In part 1 of the series, we explored some of the recent vCluster Helm chart changes and how we can create the simplest vCluster possible. We also explored how we can assign specific Kubernetes worker nodes to vClusters based on taints, tolerations, and labels. Today, we will walk through the process of setting up Cilium L2 Announcements to make the vCluster available via a LoadBalancer service and then deploy and use the vCluster Platform.

Source

Introduction​

While we can use a NodePort service to access the vCluster within our setup, this is a single point of failure in case the nodes, for whatever reason, go down. We want to achieve better scalability and ensure the vClusters are accessible via a stable IP address and not dependent on the Kubernetes nodes' availability. Thus, we can use the underlying Cilium functionality, enable IPAM to hand over LoadBalancer IP addresses and then use the L2 Announcements to make the endpoints reachable to the desired network subnet.

As the underlying control plane cluster is an RKE2 cluster, and Cilium is already installed, we will only need to update the Helm chart values to include the additional functionality. Once this is done, we will update the vCluster Helm chart values and expose the virtual clusters via a LoadBalancer IP address instead of a NodePort. Finally, by having the vCluster Platform deployed, we can have a single pane of glass when it comes to the management of a fleet of vClusters across different environments.

Lab Setup​

+-------------------------+--------------+------------------+
|        Resources        |     Type     |     Version      |
+-------------------------+--------------+------------------+
|  Control Plane Cluster  |     RKE2     | v1.34.3+rke2r1   |
|     vcluster-team-a     |     K8s      |     v1.36.0      |
|     vcluster-team-b     |     K8s      |     v1.36.0      |
+-------------------------+--------------+------------------+

Prerequisites​

1. A Kubernetes cluster available with at least two worker nodes
2. Helm installed
3. kubectl installed
4. Familiarity with vCluster

Cilium IPAM and L2 Announcements​

As mentioned in the beginning, we will create a pool that reflects the available IPv4 addresses to be assigned to LoadBalancer services. We will also define an interface where L2 Announcements should occur. The setup is simple. We will expand on the existing Cilium Helm chart values and enable the ones we need.

Extract Cilium Helm chart values.yaml​

$ helm get values rke2-cilium -n kube-system -o yaml > values_control_plane.yaml

Expand values_control_plane.yaml​

Based on the documentation, the values are required to meet our use case.

k8sClientRateLimit:
  burst: 40 # Important value when many services run on a Kubernetes cluster. Check out the documentation https://docs.cilium.io/en/v1.18/network/l2-announcements/#sizing-client-rate-limit
  qps: 20 # Important value when many services run on a Kubernetes cluster. Check out the documentation https://docs.cilium.io/en/v1.18/network/l2-announcements/#sizing-client-rate-limit
kubeProxyReplacement: true # Required
l2announcements:
  enabled: true

Update Cilium Helm Deployment​

$ helm upgrade rke2-cilium rke2-charts/rke2-cilium --version 1.18.300 --namespace kube-system -f values_control_plane.yaml

$ kubectl rollout restart deployment.apps/cilium-operator -n kube-system
$ kubectl rollout restart daemonset.apps/cilium  -n kube-system

Ensure the pods have been successfully restarted and that Cilium is using the updated values specified.

Create CiliumLoadBalancerIPPool​

The pool will allow us to assign IPv4 addresses to services of type LoadBalancer. The configuration needs to reflect your own setup. In my case, I have a dedicated VLAN I can use for handing over IPv4 addresses.

apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "vcluster-ipv4-pool"
spec:
  blocks:
  - start: "10.10.20.10"
    stop: "10.10.20.20"

Create CiliumL2AnnouncementPolicy​

The IPs will be announced from the network interface of a node with the interface name eth0. If the interface name in your setup is different, modify the file as needed.

apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: vcluster-l2-announcement-policy
  namespace: kube-system
spec:
  interfaces:
    - eth0
  loadBalancerIPs: true

Apply both manifests to the Control Plane Cluster.

$ kubectl apply -f vcluster_ipv4_pool.yaml,vcluster_l2_announcement.yaml

$ kubectl get CiliumL2AnnouncementPolicy,ippools
NAME                                                          AGE
ciliuml2announcementpolicy.cilium.io/vcluster-l2-announcement-policy   24h

NAME                                           DISABLED   CONFLICTING   IPS AVAILABLE   AGE
ciliumloadbalancerippool.cilium.io/vcluster-ipv4-pool   false      False         10              24h

Update vCluster Helm Values​

In the initial setup, we defined the Kubernetes API server of every vCluster to be exposed as a NodePort service. This can now change with the power that comes with Cilium as our Container Network Interface (CNI). With the new approach, we have a stable way to reach the virtual clusters. For both vClusters, update the controlPlane.service.spec.type to LoadBalancer.

controlPlane:
  # Service configuration for vCluster control plane access
  # The vcluster-dev will be accessible on a LoadBalancer IP Address and port 443
  service:
    enabled: true
    annotations: {}
    labels: {}
    spec:
      type: LoadBalancer

Update vCluster Helm Deployment​

$ helm upgrade --install vcluster-team-a vcluster \
  --repo https://charts.loft.sh \
  --namespace vcluster-team-a \
  --create-namespace \
  -f vcluster_team_a_lb.yaml

Validation​

$ export KUBECONFIG=control-plane-cluster.yaml 

$ kubectl get pods,svc -n vcluster-team-a
NAME                                                           READY   STATUS    RESTARTS      AGE
pod/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a   1/1     Running   1 (24h ago)   27h
pod/vcluster-team-a-0                                          1/1     Running   0             7m2s

NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns-x-kube-system-x-vcluster-team-a   ClusterIP      10.43.196.115   <none>        53/UDP,53/TCP,9153/TCP   27h
service/vcluster-team-a                            LoadBalancer   10.43.243.114   10.10.20.11   443:30445/TCP            27h
service/vcluster-team-a-headless                   ClusterIP      None            <none>        443/TCP                  27h
service/vcluster-team-a-node-el07                  ClusterIP      10.43.227.82    <none>        10250/TCP                27h

From the above output, the vcluster-team-a cluster will be available at https://10.10.20.11:443. This line should be included in the kubeconfig file on the virtual cluster. The kubeconfig of vcluster-team-a is saved as a secret named vcluster-team-a in the vcluster-team-a namespace.

$ export KUBECONFIG=vcluster-team-a.yaml 

$ kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE               KERNEL-VERSION             CONTAINER-RUNTIME
test-worker1   Ready    <none>   26h   v1.36.0   10.43.227.82  <none>        Fake Kubernetes Image  4.19.76-fakelinux (amd64)  docker://19.3.12

$ kubectl get pods -n kube-system
NAMESPACE     NAME                       READY   STATUS    RESTARTS      AGE
kube-system   coredns-754d567864-f9kgj   1/1     Running   1 (24h ago)   26h

Apply the same steps for the vcluster-team-b cluster.

vCluster Platform​

Imagine your team or your organisation handles a large number of vClusters. An easy way to manage all of them from a central location alongside providing tenant admins with an intuitive UI to perform operations, RBAC, resource quota, cluster lifecycle, etc., is the vCluster Platform.

What is the vCluster Platform?​

vCluster Platform is the management plane for your tenant cluster fleet. It provides a web UI, CLI, and API for deploying, configuring, and operating tenant clusters across one or more Control Plane Clusters. Access control, lifecycle automation, resource governance, and node management are all built in.

vCluster Platform Helm Deployment​

Helm Chart Values​

admin:
  create: true
  username: username
  password: "password"
# Loft service options
service:
  type: LoadBalancer
# Resources of the loft deployment
resources:
  requests:
    memory: 256Mi
    cpu: 200m
  limits:
    memory: 2Gi
    cpu: "2"
config:
  loftHost: https://<Accessible IP Address> # 127.0.0.1 can be set when vClusters are created on the same Control Plane Cluster where the vCluster Platform is installed 
  audit:
    enabled: true
insecureSkipVerify: true # Only for Development environments. Use your own valid TLS certificates for Production deployments.

Deployment​

$ helm upgrade --install vcluster-platform vcluster-platform \
  --repo https://charts.loft.sh/ \
  --namespace vcluster-platform \
  --create-namespace \
  --version 4.9.0 \
  --values vcluster_platform_values.yaml

Register Existing vClusters​

As long as the vCluster Platform is up and running, you should be able to reach the UI using the LoadBalancer IP address. As the vcluster-team-a and vcluster-team-b clusters have been created using a Helm chart, they are not associated with the vCluster Platform. However, we can add them under the vCluster Platform management using either the vcluster command-line utility or Helm. The recommended approach is to use the vcluster command-line utility.

1. Log in to the vCluster Platform from the UI
2. Navigate to the bottom left side, click the username of the logged-in user and click Access keys. Click the Create Access Key button to create a new key. Determine how long the key should be valid and the permissions are assigned to the key
3. On a machine with access to the vcluster CLI, perform the steps below
   1. vcluster platform login https://<vCluster Platform UI IP Address> --access-key <access-key-generated-from-UI> --insecure. In case a valid TLS certificate is used, there is no need to add the --insecure flag
   2. vcluster platform add vcluster vcluster-team-a -n vcluster-team-a --project default. The project can also be set to a different value. Default is the default project created by the deployment of the vCluster Platform
   3. vcluster platform add vcluster vcluster-team-b -n vcluster-team-b --project default

Refreshing the UI, the cluster should already be visible.

Conclusion​

In this post, we built on the foundation established in Part 1 by enabling Cilium IPAM and L2 Announcements to expose our vCluster API servers via stable LoadBalancer IP addresses, eliminating the single point of failure introduced by NodePort services. We also deployed the vCluster Platform, giving us a centralised management plane for operating our vCluster fleet across environments. In Part 3, we will take a deeper dive into the networking layer, examining how traffic flows between virtual clusters, how network policies can be enforced at both the host and virtual cluster level, and how Cilium's advanced features can further strengthen isolation and observability in a multi-tenant context. Stay tuned!

Resources​

• vCluster Documentation
• vCluster Platform
• Cilium Docs

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​

It has been a while since my last post on vCluster. After working more closely with the tool, I decided to update the series and show you what changes when it comes to a local setup.

Source

Introduction​

My first interaction with vCluster was about two years ago. I needed to create local development environments with minimal resources and low setup effort. This means easy maintenance and less troubleshooting. My goal was to use Sveltos to automate the deployment of these environments when a developer joined or left the team. To achieve my goal, and after looking around at the different open-source tools, I stumbled upon vCluster.

Fast forward a year since my last blog, and I will provide updates on what changes from a vCluster point of view. The first part will be an update of the Helm chart deployment and values. On the next one, we will integrate Cilium L2 Announcements and add the vCluster Platform to the mix for ease of management and operations. Next, we will briefly discuss the Enterprise version and some use cases I covered. Finally, we will explore Sveltos and how to use the Sveltos Event Framework. This will help us automatically set up local development environments using a GitOps approach.

Lab Setup​

+-------------------------+--------------+------------------+
|        Resources        |     Type     |     Version      |
+-------------------------+--------------+------------------+
|  Control Plane Cluster  |     RKE2     | v1.34.3+rke2r1   |
|      vcluster-dev       |     K8s      |     v1.36.0      |
|     vcluster-team-a     |     K8s      |     v1.36.0      |
|     vcluster-team-b     |     K8s      |     v1.36.0      |
+-------------------------+--------------+------------------+

Prerequisites​

1. A Kubernetes cluster available with at least two worker nodes
2. Helm installed
3. kubectl installed
4. Familiarity with vCluster

Scenario​

“Multitenancy (or multi-tenancy) refers to a single software installation that serves multiple tenants. A tenant is a user, application, or a group of users/applications that utilize the software to operate on their own data set.”

We will start with a basic vcluster-dev vCluster in the dev namespace. Then we will update the Helm chart values and create two additional vClusters, vcluster-team-a and vcluster-team-b, in their own namespaces, where they will be scheduled on different nodes based on Kubernetes taints and tolerations. As this is a local setup, the vCluster API Server is exposed as a node port. However, in a later post, we will demonstrate how to use Cilium L2 Announcements to reach the clusters with a LoadBalancer IP address.

vcluster-dev Setup​

values.yaml​

A few things have changed since the last blog post. The Helm values structure has been simplified, and configuration options are now more clearly organised. Find the vCluster Helm Chart values on GitHub. The main difference with the previous setup is that a NodePort is used to expose the API Server, while the distribution is defined as k8s and not as k3s.

CoreDNS is visible within the tenant cluster, as there might be cases for a custom DNS configuration or specific external DNS resolution domains.

For storage requirements, feel free to use your preferred open-source storage solution (Longhorn, Ceph with Rook). The simplest setup will be the local-path-provisioner, which can be installed as a Helm chart, and the setup will use the local storage on each node.

controlPlane:
  distro:
    k8s:
      enabled: true
      image:
        repository: "loft-sh/kubernetes"
        tag: "v1.36.0"
      resources:
        limits:
          cpu: 100m
          memory: 256Mi
        requests:
          cpu: 40m
          memory: 64Mi
  # Enable CoreDNS services per tenant cluster
  coredns:
    enabled: true
  statefulSet:
    resources:
      limits:
        ephemeral-storage: 2Gi
        memory: 2Gi
      requests:
        ephemeral-storage: 400Mi
        cpu: 200m
        memory: 256Mi
    highAvailability:
      replicas: 1
    security:
      podSecurityContext: {}
      containerSecurityContext:
        allowPrivilegeEscalation: false
        runAsUser: 0
        runAsGroup: 0
    persistence:
      volumeClaim:
        enabled: auto
        retentionPolicy: Retain
        size: 2Gi
        storageClass: "local-path"
        accessModes: [ "ReadWriteOnce" ]
  # Service configuration for vCluster control plane access
  # The vcluster-dev will be accessible on Node IP Address:30443
  service:
    enabled: true
    annotations: {}
    labels: {}
    httpsNodePort: 30443
    kubeletNodePort: 0
    spec:
      type: NodePort
exportKubeConfig:
  context: vcluster-dev
  insecure: false
  secret:
    name: vcluster-dev
    namespace: dev
experimental:
  deploy:
    vcluster:
      manifests: |-
        ---
        apiVersion: v1
        kind: Namespace
        metadata:
          name: nginx-app
        ---
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: nginx-dev
          namespace: nginx-app
        spec:
          selector:
            matchLabels:
              app: nginx
          replicas: 1
          template:
            metadata:
              labels:
                app: nginx
            spec:
              containers:
              - name: nginx
                image: nginx:latest
                ports:
                - containerPort: 80
        ---
        apiVersion: v1
        kind: Service
        metadata:
          name: nginx-dev
          namespace: nginx-app
        spec:
          selector:
            app: nginx
          ports:
            - protocol: TCP
              port: 80
              targetPort: 80
          type: ClusterIP

Helm Deployment​

$ helm repo add loft https://charts.loft.sh
$ helm repo update

$ export KUBECONFIG=/path/to/management/kubeconfig

$ helm upgrade --install vcluster-dev vcluster \
  --namespace dev \
  --create-namespace \
  --values /the/path/to/configuration/values.yaml \
  --repo https://charts.loft.sh \
  --repository-config=''

vCluster Validation​

$ helm list -n dev
NAME        	NAMESPACE	REVISION	UPDATED                                 	STATUS  	CHART          	APP VERSION
vcluster-dev	dev      	1       	2026-05-06 10:18:55.604901016 +0200 CEST	deployed	vcluster-0.34.0	0.34.0

$ kubectl get pods,svc,secret -n dev
NAME                                                        READY   STATUS    RESTARTS   AGE
pod/coredns-6dc6dfcb8f-zbrdx-x-kube-system-x-vcluster-dev   1/1     Running   0          2m
pod/nginx-dev-59c4c87bc6-4lrdc-x-nginx-app-x-vcluster-dev   1/1     Running   0          2m
pod/vcluster-dev-0                                          1/1     Running   0          2m21s

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/kube-dns-x-kube-system-x-vcluster-dev   ClusterIP   10.43.133.191   <none>        53/UDP,53/TCP,9153/TCP          2m
service/nginx-dev-x-nginx-app-x-vcluster-dev    ClusterIP   10.43.97.8      <none>        80/TCP                          2m
service/vcluster-dev                            NodePort    10.43.59.149    <none>        443:30443/TCP,10250:31662/TCP   2m21s
service/vcluster-dev-headless                   ClusterIP   None            <none>        443/TCP                         2m22s
service/vcluster-dev-node-el07-worker1          ClusterIP   10.43.207.196   <none>        10250/TCP                       2m

NAME                                        TYPE                 DATA   AGE
secret/sh.helm.release.v1.vcluster-dev.v1   helm.sh/release.v1   1      2m22s
secret/vc-config-vcluster-dev               Opaque               1      2m22s
secret/vc-vcluster-dev                      Opaque               5      2m
secret/vcluster-dev                         Opaque               5      2m
secret/vcluster-dev-certs                   Opaque               29     2m11s

Retrieve vcluster-dev Kubeconfig​

Once the virtual cluster is deployed, we can retrieve the kubeconfig by decoding the configuration of the secret with the name vcluster-dev.

$ kubectl get secret vcluster-dev -n dev --template={{.data.config}} | base64 -d > /the/path/to/configuration/vcluster-dev.yaml

Open the file and update the server: https://localhost:8443 section with server: https://<NODE IP>:30443. Ensure the Node IP address defined is the one the vcluster-dev-0 pod is scheduled on. Alternatively, we can define the proxy.extraSANs option to create a valid certificate for different DNS names and IPs.

Validation​

$ export KUBECONFIG=/the/path/to/configuration/vcluster-dev.yaml

$ kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
el07-worker1   Ready    <none>   134m   v1.36.0   10.43.207.196   <none>        Fake Kubernetes Image   4.19.76-fakelinux (amd64)   docker://19.3.12

$ kubectl get pods,svc -n nginx-app
NAME                             READY   STATUS    RESTARTS   AGE
pod/nginx-dev-59c4c87bc6-4lrdc   1/1     Running   0          8m13s

NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/nginx-dev   ClusterIP   10.43.97.8   <none>        80/TCP    8m12s

vcluster-team-a Setup​

values.yaml​

As mentioned, we will use the Kubernetes taints, tolerations, and Kubernetes labels to schedule vCluster to specific nodes based on the team we would like to onboard. The setup below is a simple example of how the vCluster configuration looks alongside the work done on the underlying control plane cluster and nodes.

controlPlane:
  distro:
    k8s:
      enabled: true
      image:
        repository: "loft-sh/kubernetes"
        tag: "v1.36.0"
      resources:
        limits:
          cpu: 100m
          memory: 256Mi
        requests:
          cpu: 40m
          memory: 64Mi
  # Define the node coreDNS should be scheduled on
  coredns:
    enabled: true
    deployment:
      nodeSelector:
        vcluster.loft.sh/team: "team-a"
      tolerations:
      - key: "vcluster.loft.sh/team"
        operator: "Equal"
        value: "team-a"
        effect: "NoSchedule"
  statefulSet:
    resources:
      limits:
        ephemeral-storage: 2Gi
        memory: 2Gi
      requests:
        ephemeral-storage: 400Mi
        cpu: 200m
        memory: 256Mi
    highAvailability:
      replicas: 1
    security:
      podSecurityContext: {}
      containerSecurityContext:
        allowPrivilegeEscalation: false
        runAsUser: 0
        runAsGroup: 0
    persistence:
      volumeClaim:
        enabled: auto
        retentionPolicy: Retain
        size: 2Gi
        storageClass: "local-path"
        accessModes: [ "ReadWriteOnce" ]
    # Scheduling vCluster Control plane pods run on nodes with the label set to team-a
    scheduling:
      nodeSelector:
        vcluster.loft.sh/team: "team-a"
      tolerations:
      - key: "vcluster.loft.sh/team"
        operator: "Equal"
        value: "team-a"
        effect: "NoSchedule"
  # Service configuration for vCluster control plane access
  # The vcluster-team-a will be accessible on Node IP Address with label team-a:30444
  service:
    enabled: true
    annotations: {}
    labels: {}
    httpsNodePort: 30444
    spec:
      type: NodePort
sync:
  # Tenants see fake nodes, pods still scheduled on correct nodes via taint and toleration
  fromHost:
    nodes:
      enabled: false
  toHost:
    pods:
      enabled: true
      enforceTolerations:
      - "vcluster.loft.sh/team=team-a:NoSchedule"
exportKubeConfig:
  context: vcluster-team-a
  secret:
    name: vcluster-team-a
    namespace: vcluster-team-a

Control Plane Cluster Configuration​

Connect to the control plane cluster, add the respective Kubernetes labels on the nodes we want the vcluster-team-a to use alongside the Kubernetes taints.

$ export KUBECONFIG=/path/to/Control Plane Cluster/kubeconfig

$ kubectl label node test-worker1 vcluster.loft.sh/team=team-a
$ kubectl get nodes --show-labels | grep "vcluster.loft.sh/team"

$ kubectl taint node test-worker1 vcluster.loft.sh/team=team-a:NoSchedule
$ kubectl taint node test-worker2 vcluster.loft.sh/team=team-b:NoSchedule

Helm Deployment​

$ helm repo add loft https://charts.loft.sh
$ helm repo update

$ export KUBECONFIG=/path/to/Control Plane Cluster/kubeconfig

$ helm upgrade --install vcluster-team-a vcluster \
  --namespace vcluster-team-a \
  --create-namespace \
  --values /the/path/to/configuration/values.yaml \
  --repo https://charts.loft.sh \
  --repository-config=''

vCluster Validation​

$ helm list -n vcluster-team-a
NAME           	NAMESPACE      	REVISION	UPDATED                                 	STATUS  	CHART          	APP VERSION
vcluster-team-a	vcluster-team-a	1       	2026-05-06 12:41:34.061614538 +0200 CEST	deployed	vcluster-0.34.0	0.34.0   

$ kubectl get pods -n vcluster-team-a -o wide
NAME                                                        READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
coredns-754d567864-5lfcg-x-kube-system-x-vcluster-team-a    1/1     Running   0          41s   10.42.0.66    test-worker1   <none>           <none>
vcluster-team-a-0                                           1/1     Running   0          70s   10.42.0.219   test-worker1   <none>           <none>

Retrieve vcluster-team-a Kubeconfig​

Once the virtual cluster is deployed, we can retrieve the kubeconfig by decoding the configuration of the secret with the name vcluster-team-a.

$ kubectl get secret vcluster-team-a -n vcluster-team-a --template={{.data.config}} | base64 -d > /the/path/to/configuration/vcluster-team-a.yaml

Open the file and update the server: https://localhost:8443 section with server: https://<DEDICATED NODE IP>:30444. Follow the same validation approach as in the vcluster-dev section.

Advantages Per-Team Dedicated Nodes​

• Resource Isolation: Teams cannot impact each other's resources
• Cost Tracking: Easier to track per-team infrastructure costs
• Compliance: Separate sensitive workloads onto specific nodes
• Performance: Predictable performance based on node allocation
• Failure Isolation: Node failure only affects one team assigned to the node/nodes

Conclusion​

This post concludes a basic vCluster setup with two separate configurations. Feel free to explore any other possible options provided by the Helm charts to cover different use cases. In the next post, we will enable the Cilium L2 Announcements feature and integrate the free version of the vCluster Platform! Stay tuned!

Resources​

• vCluster Documentation
• vCluster Schema Validation
• vCluster Helm Chart Values

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​

To make our lives easier when it comes to manual Sveltos progressive rollout approvals, we will use Botkube and Slack.

Scenario​

As we do not want the Platform teams or Kubernetes operators to manually approve or reject progressive rollout deployments by connecting to a Kubernetes cluster (s), we will allow them to use Slack, receive notifications about new or updated ClusterPromotion resources to approve or reject deployments through a Slack space. We will continue with the deployment demonstrated in part 2.

Tools​

Botkube​

Botkube is used as a ChatOps approach by platform engineers and is designed to integrate Kubernetes with messaging tools like Slack, Discord, and Mattermost. Built primarily in Go, it enables real-time monitoring of Kubernetes clusters by watching events such as pod failures, configuration changes, and deployments, then sending alerts directly to chat channels.

However, the project is considered stale, and there are no new commits. The demonstration here takes place to show the different options that could be used to perform an easy ChatOps approach. In a future post, we will use a Kubernetes MCP server to perform something similar.

Slack​

Slack is a cloud-based messaging platform designed for workplace collaboration, allowing teams to communicate through organised channels, direct messages, and file sharing.

Lab Setup​

+-----------------------------+------------------+----------------------+
|          Resources          |      Type        |       Version        |
+-----------------------------+------------------+----------------------+
|     Management Cluster      |    RKE2         |      v1.34.3+rke2r1   |
+-----------------------------+------------------+----------------------+

+-------------------+----------+
|      Tools        | Version  |
+-------------------+----------+
|     Sveltos       | v1.4.0   |
|     kubectl       | v1.34.1  |
|     BotKube       | v1.14.x  |
+-------------------+----------+

GitHub Resources​

The YAML outputs are not complete. Have a look at the GitHub repository.

Prerequisites​

To perform an integration with Slack, we need a free-tier account and the ability to create a Slack application with the correct permissions. Once we have that, we can continue with Sveltos, which will be used to install Botkube alongside the required configuration. Check out the official Botkube and Slack guide.

Slack Caveats​

• Must be installed manually into your Slack workspace using the provided configuration
• Slack channels must be managed manually, and you need to ensure the Botkube bot is invited to any channel you want to use with Botkube
• When using executor plugins (e.g. kubectl) in a multi-cluster environment, each cluster needs to have a dedicated Botkube bot for Slack in order to route commands to the correct cluster.

Slack does not support multi-cluster deployments. This could be addressed using Discord or Mattermost. However, having already a Slack account, it was an easier choice for my setup.

Slack free-tier account​

Follow the link to create a free-tier Slack account.

Create Slack Workspace​

If you do not already have a Slack workspace with admin rights, follow the Slack official documentation for more details.

Create Slack Application​

To allow Botkube to use the Slack socket, we need to create an Application for Botkube and define the right permissions for the users who are part of a channel. Perform the steps described below to create an application.

1. Go to the Slack App console
2. Create a new App and select "From an app manifest"
3. Choose the workspace created in a previous step
4. For Public and Private channels configuration

  display_information:
    name: Botkube
    description: Botkube
    background_color: "#a653a6"
  features:
    bot_user:
      display_name: Botkube
      always_online: false
  oauth_config:
    scopes:
      bot:
        - channels:read
        - channels:join
        - groups:read
        - app_mentions:read
        - reactions:write
        - chat:write
        - chat:write.public
        - users:read
        - im:read
        - mpim:read
        - commands
  settings:
    event_subscriptions:
      bot_events:
        - app_mention
    interactivity:
      is_enabled: true
    org_deploy_enabled: false
    socket_mode_enabled: true
    token_rotation_enabled: false # Enable for production workloads

1. Install the Botkube in the Slack workspace
2. Bot OAuth Token
   • Navigate to Features > OAuth & Permissions
   • Copy the "Bot Use OAuth Token"
3. App-Level Token
   • Navigate to "Settings > Basic Information > App-Level Tokens"
   • Click the "Generate Token and Scopes"
   • Define a name of your preference
   • Define the connections: write scope
   • Generate and copy the token

Add Botkube user to Slack Channel​

You can either create a new channel within a defined workspace or use one of the autogenerated channels. As we have an application ready, we can invite the @<your application name> user to a channel. For more information about inviting a user to a channel, take a look at the official documentation.

Install Botkube Helm Chart​

As Botkube offer the ability to use a Helm chart installation without needing to use the botkube cli utility, we will follow this approach. But, before we perform the deployment, let’s create a values.yaml file that contains all our settings first to enable the connection and communication with a specific Slack channel within a workspace, and then enable the Botkube capabilities and permissions working with the Kubernetes cluster.

Craft values.yaml file​

serviceAccountMountPath: /var/run/7e7fd2f5-b15d-4803-bc52-f54fba357e76/secrets/kubernetes.io/serviceaccount
groups:
  'botkube-plugins-default':
    create: true
    rules:
      - apiGroups: [""]
        resources: ["pods", "services", "configmaps", "events"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["customresourcedefinitions"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["config.projectsveltos.io"]
        resources: ["clusterpromotions", "clusterpromotions/status"]
        verbs: ["get", "list", "watch", "update", "patch"] 
actions:
'revoke-approval':
  enabled: true
  displayName: "ClusterPromotion Approval Handler"
  command: "kubectl describe {{ .Event.Kind | lower }}{{ if .Event.Namespace }} -n {{ .Event.Namespace }}{{ end }} {{ .Event.Name }}"
  bindings:
    sources:
      - k8s-clusterpromotions
    executors:
      - k8s-default-tools
sources:
'k8s-clusterpromotions':
  displayName: "ClusterPromotion Events"
  botkube/kubernetes:
    context: &default-plugin-context
      rbac:
        group:
          type: Static
          prefix: ""
          static:
            values: ["botkube-plugins-default"]
    enabled: true
    config:
      namespaces: &k8s-events-namespaces
        include:
          - ".*"
      event:
        types:
          - create
          - update
      filters:
        objectAnnotationChecker: true
      resources:
        - type: config.projectsveltos.io/v1beta1/clusterpromotions
          event:
            types:
              - create
              - update
          updateSetting:
            includeDiff: true
            fields:
              - spec.stages[*].trigger.manual.approved
executors:
k8s-default-tools:
  botkube/kubectl:
    displayName: "Kubectl"
    enabled: true
    config:
      defaultNamespace: "default"
    context: *default-plugin-context
aliases:
k:
  command: kubectl
  displayName: "Kubectl alias"
communications:
'default-group':
  socketSlack:
    enabled: true
    channels:
      'default':
        name: 'all-sveltos-promotions-demo'
        bindings:
          executors:
            - k8s-default-tools
          sources:
            - k8s-clusterpromotions
    botToken: 'xoxb-'
    appToken: 'xapp-'
settings:
clusterName: "dev-cluster"
log:
  level: info
plugins:
cacheDir: "/tmp"
repositories:
  botkube:
    url: https://github.com/kubeshop/botkube/releases/download/v1.14.0/plugins-index.yaml
  botkubeExtra:
    url: https://github.com/kubeshop/botkube-plugins/releases/download/v1.14.0/plugins-index.yaml

We instruct Botkube to have full permissions on the ClusterPromotion resource. Once a new or update ClusterPromotion resource is detected, we trigger a describe action. Then, we use kubectl to patch the resources from Slack.

Botkube Deployment​

As we have Sveltos already installed in the Kubernetes management cluster, we can utilise the ClusterProfile Custom Resource definition (CRD) and allow Sveltos to deploy Botkube to the cluster.

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-botkube-mgmt
spec:
  clusterSelector:
    matchLabels:
      type: mgmt
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://charts.botkube.io
    repositoryName:   botkube
    chartName:        botkube/botkube
    chartVersion:     v1.14.0
    releaseName:      botkube
    releaseNamespace: botkube
    helmChartAction:  Install
    valuesFrom:
    - kind: ConfigMap
      name: botkube-config
      namespace: default

The ConfigMap information is available at the blog post resources GitHub repository.

Alternatively, the following Helm chart commands will deploy Botkube to the exposed cluster.

$ export KUBECONFIG=/path/to/management/kubeconfig
$ helm repo add botkube https://charts.botkube.io/  
$ helm install botkube botkube/botkube -n botkube --create-namespace -f values.yaml # values.yaml contains the Helm chart values
$ kubectl get pods -n botkube

ClusterPromotion Example​

Every time we create or update a ClusterPromotion resource, Botkube sends a notification to a Slack channel. The members of the Slack channel can perform limited actions using the kubectl command from Slack based on the RBAC definition in the values.yaml file.

Workflow in Action​

┌─────────────────────────────────────────────────────────┐       ┌──────────────────────────────────────────────────────┐
│           KUBERNETES CLUSTER (dev-cluster)              │       │                  SLACK PLATFORM                      │
│                                                         │       │                                                      │
│  ┌───────────────────────────────────────────────────┐  │       │  Channel: #all-sveltos-promotions-demo               │
│  │                    BOTKUBE POD                    │  │       │                                                      │
│  │                                                   │  │       │  ┌──────────────────────────────────────────────┐    │
│  │  ┌─────────────────┐     ┌─────────────────────┐ │  │        │  │ 🤖 Botkube: ClusterPromotion Updated!         │   │
│  │  │  SOURCE PLUGIN  │     │  EXECUTOR PLUGIN    │ │  │        │  │    Diff: manual.approved: false ──► true      │   │
│  │  │  Watches:       │     │  kubectl describe   │ │  │        │  │  [ 🔍 Describe ][ ✅ Approve ][ ❌ Revoke ]    │   │
│  │  │  ClusterPromo   │     │  kubectl [k]        │ │  │        │  └──────────────────────────────────────────────┘   │
│  │  └────────┬────────┘     └──────────┬──────────┘ │  │        │                        │                            │
│  │           └──────────────┬──────────┘            │  │        │  ┌──────────────────────▼───────────────────────┐   │
│  │                    ┌─────▼──────────┐            │  │        │  │ 👤 User: @Botkube kubectl describe           │    │
│  │                    │  BOTKUBE CORE  │            │  │        │  │         my-promotion -n production           │   │
│  │                    │  RBAC + Action ├────────────────────────►  └──────────────────────┬───────────────────────┘   │
│  │                    │  Handler       │◄───────────────────────┐                         │                           │
│  │                    └────────┬───────┘  Outbound WSS          │  ┌──────────────────────▼───────────────────────┐   │
│  └─────────────────────────────┼─────────────────────┘  │       │  │ 🤖 Botkube: Name:   my-promotion             │    │
│                                │ Watch / Patch          │       │  │             Stages: manual.approved: true    │    │
│  ┌─────────────────────────────▼───────────────────┐    │       │  └──────────────────────────────────────────────┘    │
│  │              KUBERNETES API SERVER               │   │       │                                                      │
│  │              ClusterPromotion CRD                │   │       └──────────────────────────────────────────────────────┘
│  └──────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Conclusion​

In my opinion, ChatOps is a valid approach when it comes to the approval or rejection of ClusterPromotion actions. It provides teams with an easy and controlled way of checking what is going on in a cluster. However, nowadays, this could be easily done using an AI Agent instead of using different Slack plugins to achieve the same result.

In part 4 of the series, we will explore how to achieve a ChatOps approach using AI Agents instead. Stay tuned! 🚀

Resources​

• Sveltos Quick Start
• Sveltos Use Cases

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

👏 Support this project​

Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

Series Navigation​

In a previous post, we looked at using MkDocs and MkDocs Material with the Mike plugin for versioning. This helps in creating a static site on GitHub with GitHub workflows. As MkDocs Material entered maintenance mode on the 6th of November 2025, I decided to migrate to Zensical.

Introduction​

The team will continue to provide critical bug fixes and security updates for at least 12 months, with no new features to MkDocs Material, while users are encouraged to migrate to Zensical once dependent features are available.

What is Zensical?​

Zensical is a modern, free, and scalable open-source toolchain for static sites from the creators of Material for MkDocs. It is designed for speed and superior authoring, and the alpha version is already compatible with Material for MkDocs!

Effectively, if you already use MkDocs Material, you can switch to Zensical if the features you depend on are available.

GitHub Resources​

The showcase repository is available here.

Prerequisites​

1. Go through parts one and two of the series
2. Basic knowledge and understanding of MkDocs
3. Basic understanding of GitHub workflows

Migration - Plan Preparation​

The compatibility section shows that the team aimed for a smooth migration. This means users can expect easy compatibility with Material for MkDocs. If you already have an mkdocs.yml file, you can keep it and perform only the required changes for Zensical. To learn more about how Zensical understands an existing mkdocs.yml file, go through the configuration section. To learn more about the features, go through the features section.

In my case, I performed the steps below to migrate the Sveltos documentation site to Zensical. The approach might differ based on your project and structure. However, the core idea remains the same.

1. Read the Zensical documentation
2. Go through the features section
3. Understand how Mike's for versioning fits into the mix
4. Track down the features used for your deployment. Explore how the mkdocs.yml needs to be updated
5. Explore how the GitHub workflows can be modified for the migration

In Sveltos' case, the changes performed were minimal. In the sections below, we will outline how the configuration was modified to accommodate the migration.

Feature Used​

• Python Markdown Extensions
  • pymdownx.emoji
• palette.toggle.icon
• Mike Versioning

mkdocs.yml​

Before​

markdown_extensions:
- pymdownx.emoji:
    emoji_index: !!python/name:material.extensions.emoji.twemoji
    emoji_generator: !!python/name:material.extensions.emoji.to_svg
toggle:
    icon: material/brightness-7
toggle:
    icon: material/brightness-4
plugins:
 - search
 - mike

After​

markdown_extensions:
- pymdownx.emoji:
    emoji_index: !!python/name:zensical.extensions.emoji.twemoji ""
    emoji_generator: !!python/name:zensical.extensions.emoji.to_svg ""
toggle:
    icon: lucide/sun
toggle:
    icon: lucide/moon
plugins:
 - search

GitHub Workflow - Updates​

Only minimal changes were necessary here.

dev Workflow​

name: CI build dev docu
on:
  push:
    branches:
    - main
permissions:
  contents: write
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v6
      with:
        fetch-depth: 0
    - uses: actions/setup-python@v6
      with:
        python-version: 3.x
    - name: Install Dependencies
      run: |
        pip install zensical==0.0.32
        pip install git+https://github.com/squidfunk/mike.git
    - name: Setup Docs Deploy
      run: |
        git config --global user.name "Example Docu Deploy"
        git config --global user.email "docs.deploy@eleni.dev"
    - name: Build Docs Website
      run: |
        mike deploy --push main --update-aliases
        mike set-default main --push

Production Workflow​

name: CI Build Prod Docu
on:
  release:
    types: [published]
permissions:
  contents: write
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v6
        with:
          python-version: 3.x
      - name: Install Dependencies
        run: |
          pip install zensical==0.0.32
          pip install git+https://github.com/squidfunk/mike.git
      - name: Setup Docs Deploy
        run: |
          git config --global user.name "Example Docu Deploy"
          git config --global user.email "docs.deploy@eleni.dev"
      - name: Build Docs Website
        run: mike deploy --push --update-aliases ${{ github.event.release.tag_name }} latest
      - name: Hide Old Releases
        run: |
          VISIBLE_RELEASES=2
          # Include only versions starting with `v`.
          # Sort the versions from oldest to latest
          all_releases=$(mike list --json | \
                                 jq -r '.[] | select(.properties.hidden | not) | .version' | \
                                 grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+(\-.*)?$' | \
                                 sort -V)
          # Add the Mike versions into an array
          releases_array=($all_releases)
          num_releases=${#releases_array[@]}

          # Check if more than 2 releases are available
          if (( num_releases > VISIBLE_RELEASES )); then
            hide=$((num_releases - VISIBLE_RELEASES))
            echo "We have to hide $hide versions"
            
            # Take the oldest applicable releases and hide them
            for (( i=0; i<hide; i++ )); do
              hide_version=${releases_array[i]}
              echo "Hiding release version: $hide_version"
              mike props --set hidden=true "$hide_version" --push 
            done
          else
            echo "No release versions to hide. Exit"
          fi

A detailed PR for the Sveltos production documentation is available here.

Local Execution​

To run the code locally, only a few things are required. Follow the commands listed below.

Python Environment Setup​

$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install zensical==0.0.32
$ pip install git+https://github.com/squidfunk/mike.git

Static Site Execution​

$ zensical build --clean
$ zensical serve

The local static site is available at http://localhost:8000.

Conclusion​

Easy and headache-free approach to migrating from MkDocs Material to Zensical! Happy coding!

Resources​

• Zensical - Getting Started

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​

KubeCon Amsterdam 2026 highlights.

Introduction​

Well, first KubeCon ever for me! I was already hyped by the Cloud Native Rejekts and the energy of the community; I was looking forward to attending this one! Vlad's post got me thinking, "You will walk and talk a lot." He also shared useful tips: download the venue map, remember must-see sessions, avoid overscheduling, and pack light. And I would agree with most of the points mentioned! Indeed, it was going to be a long week, and I had to come up with a plan to keep the energy levels high. My plan was simple: I would run almost daily in the morning, shower, eat breakfast, pack, and head to the event. And to be honest, the morning runs were great! They were refreshing, especially with the unpredictable weather in Amsterdam. From sun to wind and rain!

From the moment I entered the venue, I noticed the vibrant community. It struck me that it was full of engineers who work with open-source and cloud-native technologies. They share my passions and interests, and we all face the same issues and frustrations. So, let the week begin. And in all truth, it was an amazing week, with amazing humans!

Briefly, my main focus was to discuss and exchange ideas with fellow platform engineers on scalability, security, and the different ways and approaches of running AI workloads. Apart from that, I wanted to visit a few booths and ask about roadmap items and what is coming next.

Day-0: Co-located Events​

I was lucky enough to attend the KubeCon co-located events. I was between the Platform Engineering, Flux, Cilium and MCP Con. My morning started with the session "Vibe Coding Meets Gitops" by Stefan Prodan, where the Flux MCP was introduced alongside its usability and its purpose when it comes to AI agents and GitOps. A bit later on, the session "Talking To Your Cluster: Conversational Gitops with Flux MCP by William Rizzo" gave a practical example of how to combine a GitOps approach with an AI agent specialised in Flux.

Max Koerbaecher provided a nice overview and practical insights on how to create an IDP for AI engineering. The session details are available here. To me, the "AIDP Reference Architecture" mentioned in the slides is something to take and start discussions with the platform team.

Continuing with Dominik Schmidle and the session "Overwhelmed by Scale: How Product Thinking Fixes Platform Teams", where he explained how platform teams started off well but ended up overwhelmed with different sorts of issues and concerns, the session explains that the fix is to treat the platform like a product!

At the MCP Con, I had the chance to watch a talk by Patrick Debois "The New AI Coding Fabric", on how AI-driven development is evolving and goes beyond traditional IDEs. I kept the phrase "Context is the fuel", and a way to remember that garbage in, garbage out. Context will be one of the most critical aspects when it comes to AI coding.

Last but not least, Cilium Con was a blast! My highlight sessions were one from Mikael Johansson Länsberg on how to replace legacy load balancers with Cilium, the lightning talk from Martynas Pumputis on what's happening in Cilium and the lessons learned when it comes to Tetragon policies from Alessio Biancalana.

KeyNotes​

From a keynote's point of view and more on the technical side, I felt we moved away from the AI general talks, and we headed into scalable, secure and extensible platforms for AI. Platforms are once again in the center of operations and serve developers the best way possible by the use of self-service portals or APIs. The Platform teams will have to provide Al infrastructure and capabilities using best practices and guardrails. Let the challenge begin!

Sovereignty and the Open Sovereign Cloud came up as a big topic and names like the NeoNephos, OVH and Upcloud popped up. Apart from that, llm-d joined the Cloud Native Computing Foundation (CNCF), and solo.io announced the launch of agentevals and the contribution of the open source project agentregistry to the CNCF.

Last but not least, day 1 keynotes concluded with an electric glider powered by renewable energy. A session out of the ordinary. Very interesting topic around sustainability.

Day-1​

Evangelista Tragni had a great session on KubeVirt. He shared many insights about the technology, how it works, and its current limitations. If your organisation is looking for ways to use Kubernetes for hosting virtual machines, I would recommend taking a look at the session notes.

I particularly enjoyed the session from Sunny Beatteay. They shared their journey of moving cloud infrastructure from regional to zonal clusters. Their team’s experiences and the lessons learned during the migration were especially insightful. Worth checking out.

Maciek Różacki and Artur Rodrigues discussed the architectural shifts required for Kubernetes to remain the backbone of modern computing when it comes to AI and LLMs.

Day-2​

So many great sessions at day 2; however, my personal highlight was a session from Stefan Bueringer and Fabrizio Pandini "In-place Updates with Cluster API: The Sweet Spot Between Immutable and Mutable Infrastructure", where they explored Cluster API's innovative in-place updates feature, which eliminates the need to choose between immutable and mutable infrastructure approaches.

Day 2 for me was mostly spent at the show expo floor and at the booth area, mainly talking to vendors I had an interest in. Apart from that, I met and had discussions with different people from different backgrounds and experiences! Loved it!

Booths I Always Visit​

I make it a point to visit the booths below at conferences and ask what is coming next:

• vCluster: I use vCluster to set up multitenant environments and automate workflows with Sveltos
• Sidero Labs: As a Talos fan, I always check in on what is next. I am particularly excited about SBOM capabilities for the open-source version.
• Isovalent: Great conversations with knowledgeable people about Cilium and the Isovalent Platform. Plus, nice stickers and hands-on labs to explore Cilium features
• SUSE: RKE2 clusters are heavily used in my lab setup alongside Talos. I am always curious about what is coming
• Dash0: Discovered a fresh approach to collecting and displaying metrics; it is worth checking out!

Day-3​

Day-3 was a short one for me as I had to make my way back home. If you are into Kubernetes networking and want to explore how Cilium could fit in your Platform and solve issues on the network layers using a modern and innovative approach, check out the session from Benoit Entzmann "An Immersive and Virsual Journey Into Kubernetes Networking".

Resources​

• KubeCon Europe Amsterdam Schedule 2026
• Youtube - KubeCon and CloudNativeCon Series

Conclusion​

I will take with me the chance to meet new people, build connections, and share ideas on various topics. AI was everywhere at the conference and will remain there. Still, we are humans after all!

Source

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!