Scenario​

As platform engineering teams, we always want control over what we deploy for our consumers such as cluster types, configuration, etc. In multitenant setups, this means allowing fundamental resources from the Control Plane cluster to be shared across tenant environments. The goal is to give teams the freedom to use their clusters as they see fit, while still be in full control over the deployments.

In today's demonstration, we will allow teams to share Gateway API resources backed by LoadBalancer services provided by Cilium. We introduce a straightforward way of sharing Gateway API resources between the Control Plane cluster and the underlying tenant environments. Cilium is configured to integrate with Gateway API, the required CRDs are installed only in the Control Plane cluster, and we keep the same versions and cluster names used in the previous posts.

Lab Setup​

+-------------------------+--------------+------------------+
|        Resources        |     Type     |     Version      |
+-------------------------+--------------+------------------+
|  Control Plane Cluster  |     RKE2     | v1.35.6+rke2r1   |
|     vcluster-team-d     |     K8s      |     v1.36.0      |
|         Cilium          |     CNI      |     v1.19.4      |
+-------------------------+--------------+------------------+

Introduction to Gateway API​

While the Ingress API supports basic routing based on path and host rules, it lacks support for advanced routing features. This is where Gateway API enters the discussion.

Gateway API is an official Kubernetes project focused on L4 and L7 routing in Kubernetes. This project represents the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs.

Why Gateway API Matters?​

Gateway API does not only overcome the constraints of supporting only HTTP and HTTPS traffic, it provides a more robust, extensible, and role-oriented approach to traffic engineering. The project represents the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs.

Source

Control Plane Cluster - Cilium Configuration​

As a starting point, we will expand the Cilium configuration and add the required Helm values to enable the integration of Cilium with the Gateway API. Cilium is the underlying layer for our networking, observability, and security, powered by eBPF, and since version 1.12.x, Cilium integrates nicely with the Gateway API.

Cilium Helm Chart Upgrade​

To enable the integration and going through the official documentation, a few prerequisites are required. Take a look at the updated Helm chart values file.

gatewayAPI:
  enabled: true
k8sClientRateLimit:
  burst: 40 # Important value when many services run on a Kubernetes cluster. Check out the documentation https://docs.cilium.io/en/v1.18/network/l2-announcements/#sizing-client-rate-limit
  qps: 20 # Important value when many services run on a Kubernetes cluster. Check out the documentation https://docs.cilium.io/en/v1.18/network/l2-announcements/#sizing-client-rate-limit
kubeProxyReplacement: true # Required for the Gateway API integration
l2announcements:
  enabled: true

$ helm upgrade rke2-cilium rke2-charts/rke2-cilium --version 1.19.400 --namespace kube-system -f values_control_plane.yaml

Once the helm upgrade command is complete, deploy the below Gateway API CRDs to the Control Plane cluster.

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml(Optional)

Validation​

$ kubectl get crds | grep gateway.networking.k8s.io

gatewayclasses.gateway.networking.k8s.io                   2026-07-28T06:52:01Z
gateways.gateway.networking.k8s.io                         2026-07-28T06:52:02Z
grpcroutes.gateway.networking.k8s.io                       2026-07-28T06:52:03Z
httproutes.gateway.networking.k8s.io                       2026-07-28T06:52:05Z
referencegrants.gateway.networking.k8s.io                  2026-07-28T06:52:06Z
tlsroutes.gateway.networking.k8s.io                        2026-07-28T06:52:07Z

$ kubectl get gatewayclass
NAME     CONTROLLER                     ACCEPTED   AGE
cilium   io.cilium/gateway-controller   True       47h

CiliumLoadBalancerIPPool​

We already described in a dedicated vCluster multi-pool setup post how to enable the Cilium Multi-Pool mode and CiliumLoadBalancerIPPool functionality leveraging L2 Announcements. For more details on the topic, follow the sections Cilium configuration and Cilium CRDs.

vCluster Helm Chart Update​

Documentation and Discovery​

Once we ensured Cilium is fully functional with Gateway API, it is time to take a look at the vCluster side and check how to share resources between the Control Plane and the multitenant cluster. According to the official documentation

syncing is the process by which vCluster replicates resources between the tenant cluster and the Control Plane Cluster. By default, vCluster only syncs low-level resources, such as pods, secrets, configmaps, or services, but a user can enable many other resources for all use cases.

This is promising, but not exactly what we look for. Reading more of the documentation, I found an example sharing a Gateway resource between the Control Plane cluster and the multitenant setups. This could have been something we could use, however, my main goal is to allow teams to use their own Gateway and Gateway related resources.

To conclude the discovery, I did not find a way to use the free version of vCluster to cover my use case. However, as I have a limited availability Enterprise license, I managed to achieve my goal. We will start with the vCluster Helm chart values, outline the additions and why they are required, while we demo a simple NGINX application at the underlying tenant cluster to prove our point.

vCluster Helm Chart Values​

We will update the existing Helm chart values used in parts 1 and 2. The section of interest is under the sync.toHost.customResources field. First, we will specify the CRDs we want to sync with the underlying tenant clusters and then add specific Patches to make the setup work as expected.

Synchronise CRDs​

sync:
  toHost:
    customResources: # Outline the CRDs dedicated to Gateway API
      gatewayclasses.gateway.networking.k8s.io:
        enabled: true
      gateways.gateway.networking.k8s.io:
        enabled: true
      httproutes.gateway.networking.k8s.io:
        enabled: true
      grpcroutes.gateway.networking.k8s.io:
        enabled: true
      referencegrants.gateway.networking.k8s.io:
        enabled: true
      tlsroutes.gateway.networking.k8s.io:
        enabled: true

When syncing resources from the tenant clusters to the Control Plane cluster, vCluster needs to rewrite references so they point to the correct resources in the Control Plane cluster. This can only happen using the Patches mechanism. Without that, we do see the below information from the Control Plane node when we create HTTPRoute resources. Effectively, the mapping between the tenant and the Control Plane node do not match.

$ kubectl get httproute -n vcluster-team-d -o yaml # Query the output on the Control Plane node
spec:
  parentRefs:
    - name: team-d-gateway        # The name does not exist on the Control plane node
      namespace: default          # The default namespaces does not reflect the namespace on the Control plane node
  rules:
    - backendRefs:
        - name: nginx             # The backendRefs is defined as nginx-x-default-x-vcluster-team-d in the Control plane node
          port: 80

To overcome the obstacle mentioned above, a patch needs to be added under sync.toHost.customResources.httproutes.gateway.networking.k8s.io.

httproutes.gateway.networking.k8s.io:
  enabled: true
  patches:
    - path: spec.parentRefs[*].name # Automatically rewrite Gateway references to synced names
      reference:
        apiVersion: gateway.networking.k8s.io/v1
        kind: Gateway
    - path: spec.rules[*].backendRefs[*].name # Automatically rewrite backend service references
      reference:
        apiVersion: v1
        kind: Service

Without these patches, unfortunately, the HTTPRoute resources would reference non-existent resources in the Control Plane cluster and fail to provide a working example.

Update vCluster Helm Deployment​

$ export KUBECONFIG=control-plane-cluster.yaml

$ helm upgrade --install vcluster-team-d vcluster \
  --repo https://charts.loft.sh \
  --namespace vcluster-team-d \
  --create-namespace \
  -f vcluster_team_d_syncer.yaml

Once the pods in the vcluster-team-d namespace of the Control Plane cluster are in a READY state, proceed with the NGINX application deployment.

Validation​

After the successful deployment, the Gateway API CRDs are available in the underlying tenant cluster.

$ export KUBECONFIG=vcluster-team-d.yaml

$ kubectl get crds | grep gateway.networking.k8s.io
gatewayclasses.gateway.networking.k8s.io    2026-07-28T07:16:49Z
gateways.gateway.networking.k8s.io          2026-07-28T07:16:51Z
grpcroutes.gateway.networking.k8s.io        2026-07-28T07:16:52Z
httproutes.gateway.networking.k8s.io        2026-07-28T07:16:53Z
referencegrants.gateway.networking.k8s.io   2026-07-28T07:16:55Z
tlsroutes.gateway.networking.k8s.io         2026-07-28T07:16:56Z

Deploy an NGINX Application - Multitenant Cluster​

Gateway Resource​

Let's start by creating a Gateway resource for an NGINX application deployment.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: team-d-gateway
spec:
  gatewayClassName: cilium
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

$ export KUBECONFIG=vcluster-team-d.yaml

$ kubectl apply -f team-d-gateway.yaml

$ export KUBECONFIG=control-plane-cluster.yaml

$ kubectl get gateway -n vcluster-team-d
NAME             CLASS    ADDRESS       PROGRAMMED   AGE
v48m682h72rpai   cilium   10.10.20.24   True         57s

$ kubectl get svc -n vcluster-team-d | grep -i cilium-gateway
NAME                                       TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
cilium-gateway-v48m682h72rpai              LoadBalancer   10.43.111.85    10.10.20.24   80:32583/TCP             67s

HTTPRoute Resource​

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: nginx-route
spec:
  parentRefs:
    - name: team-d-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: nginx
          port: 80

NGINX Resource​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Apply Resources​

$ export KUBECONFIG=vcluster-team-d.yaml

$ kubectl apply -f team-d-httproute.yaml,nginx-deployment.yaml

$ kubectl get pods,svc,httproute
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-59c4c87bc6-b5c56   1/1     Running   0          20s
NAME                                    TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/cilium-gateway-team-d-gateway   LoadBalancer   10.43.111.85   10.10.20.24   80:32583/TCP   3m11s
service/kubernetes                      ClusterIP      10.43.213.98   <none>        443/TCP        96m
service/nginx                           ClusterIP      10.43.42.143   <none>        80/TCP         20s
NAME                                              HOSTNAMES   AGE
httproute.gateway.networking.k8s.io/nginx-route               20s

Validation​

The application should be accessible from within the vcluster-team-d cluster and from any location within the defined subnet/network segment. We can simply perform a cURL test.

curl http://${GATEWAY_IP}

Common Issues and Troubleshooting Steps​

Cilium Gateway Class​

In some Cilium version between 1.17 and 1.18, when the Gateway API integration is enabled, the GatewayClass resource is not automatically created. To force the creation of the resource, update the Cilium Helm chart values and include the highlighted lines below.

gatewayAPI:
  enabled: true
  gatewayClass:
    create: "true"

Gateway Stuck in Pending State​

If the synchronisation of resources between the tenant and the Control Plane cluster is not done correctly, the Gateway resource on the tenant cluster will never get an IP address. Some common issues are around IP pool exhaustion, service selection does not match the tenant namespace, or some underlying issue with the Cilium Operator. Follow the commands described below to troubleshoot the issue further.

$ kubectl describe gateway team-d-gateway
$ kubectl get ciliumloadbalancerippool vcluster-team-d-pool -o yaml

HTTPRoute Not Working​

The HTTPRoute created does not route the traffic correctly. Some common issues relate to the backend service not being available or the port mismatch between the HTTPRoute and the service.

$ kubectl describe httproute nginx-route
$ kubectl get gateway team-d-gateway -o jsonpath='{.status.listeners}'
$ kubectl get svc nginx
$ kubectl get httproute nginx-route -o yaml

Advantages of Setup​

While we could allow teams to deploy their own CRDs on a tenant cluster, we give them the ability to utilise resources available in the Control Plane cluster without going through the process of installing CRDs or other resources required for their work. This matters as developers and teams can have their own Gateway and relevant resources without worrying about other tenants or conflicts!

Resources​

• vCluster Custom Resources Documentation
• Cilium Gateway API Documentation
• Cilium IPAM
• Gateway API Specification
• Kubernetes Gateway API Reference

Conclusion​

In today's post, we explored how to share Gateway API resources between the Control Plane cluster and underlying tenant clusters using vCluster's sync mechanism and Cilium. This approach gives platform teams full control over shared infrastructure while allowing tenant teams the flexibility to manage their own Gateway and HTTPRoute resources. In the next post, we will explore how to setup a Cilium multi-tenant pool for the clusters.

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​

When I started the vCluster series, I did not expect to keep writing and have already releases a few parts. Having said that, I strongly believe that combining an RKE2 Kubernetes cluster with vCluster and Cilium Multi-Pool IPAM allows us to create powerful multitenant setups for teams building development environments following a GitOps approach. In today's part, we extend the setup to include individual PodCIDRs and LoadBalancer IP ranges for each tenant. This is made possible by Cilium's Multi-Pool IPAM mode, which assigns pod IPs from different pools on a per-namespace basis. We also cover the RKE2 1.35.x cluster bootstrap, touching the cgroup v2 migration required by newer Kubelet versions.

Introduction​

While the first parts of series 1 and 2 covered the foundation and, with all honesty, a fast and easy way to create multitenant environments, this part of the series looks into extending this to enable teams to use different PodCIDRs for individual environments. This is made possible with the power of Cilium.

Cilium's IPAM works in three different modes: Kubernetes Host Scope, Cluster Scope, and Multi-Pool Scope. The first one is when Kubernetes assigns the PodCIDRs to each node. In the second mode, Cilium manages the PodCIDR allocation with the help of the Cilium Operator. The third and more flexible mode allows pods to receive an IP address from multiple IP pools, either from an annotated namespace or an annotation appended to an application manifest. Keep in mind, the default Cilium mode is set to Cluster Scope. For more information about the Cilium Helm chart values, take a look at the Cilium official GitHub repository.

What are the advantages of enabling Cilium Multi-Pool IPAM for multitenant environments? It enables us to allocate IP addresses to pods on the same node, but from different CiliumPodIPPool resources. This is ideal for teams who want to have granular control over per-namespace IP allocation while having the flexibility to add new CIDRs dynamically when required. Combined with CiliumLoadBalancerIPPool and L2 announcements or BGP, we can also scope the LoadBalancer IPs per tenant, ensuring no resource exhaustion between teams. This is a strong use case when teams provide services to different tenants and want to control the traffic exiting the cluster. They now have an easy way to identify the tenant behind the environment by source IP alone.

If you are new to RKE2 and Cilium, the Cilium Cluster Mesh on RKE2 post covers the foundational setup. For advanced Cilium networking on RKE2, take a look at the Dual-Stack RKE2 with Cilium on Proxmox series.

Lab Setup​

+-------------------------+--------------+------------------+
|        Resources        |     Type     |     Version      |
+-------------------------+--------------+------------------+
|  Control Plane Cluster  |     RKE2     | v1.35.6+rke2r1   |
|     vcluster-team-a     |     K8s      |     v1.36.0      |
|     vcluster-team-b     |     K8s      |     v1.36.0      |
|         Cilium          |     CNI      |     v1.19.4      |
+-------------------------+--------------+------------------+

Prerequisites​

Go through parts 1,2 and 3 of the series to gain a better understanding of the concept and what we aim to achieve.

RKE2 Initial Cluster Setup​

Before we dive into the Cilium configuration, it is worth taking a look at the control plane cluster configuration first. We will use the recommended way to build a cluster. This is usually via an Infrastructure as Code (IaC) approach, or, to keep things simple and easy, the automated installer. As the demo focuses on the underlying Cilium configuration, I decided to use the automated installer configuration for simplicity purposes. In case you want to explore what an IaC example looks like, take a look at the RKE2 on Azure blog.

$ journalctl -xu rke2-server.service -f
level=error msg="Kubelet exited: exit status 1"
msg="Failed to test etcd connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""

$ tail /var/lib/rancher/rke2/agent/logs/kubelet.log
E0731 07:50:00.631256    1983 run.go:72] "command failed" err="failed to validate kubelet configuration, error: kubelet is configured to not run on a host using cgroup v1. cgroup v1 support is unsupported and will be removed in a future release, path: &TypeMeta{Kind:,APIVersion:,}"

Configuration​

Once any issues with the cgroup version are resolved, it is time to set up the RKE2 cluster to support the Cilium Multi-Pool scope. Before going any further, ensure you have an understanding of the RKE2 official documentation server and agent settings.

write-kubeconfig-mode: 0644
tls-san:
  - <YOUR TLS SAN>
token: <YOUR token to connect nodes to the cluster>
cni: cilium
node-ip: "YOUR NODE IP Address"
advertise-address: "YOUR NODE IP Address"
disable-kube-proxy: true

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    image:
      tag: v1.19.4 # The Cilium Version supported by [RKE2 v1.35.6+rke2r1](https://docs.rke2.io/release-notes/v1.35.X)
    kubeProxyReplacement: true # We want to enable Cilium with kube-proxy replacement
    k8sServiceHost: 127.0.0.1
    k8sServicePort: 6443
    operator:
      replicas: 1
    bpf:
      masquerade: true # This is required by the Multi-Pool scope setup for Cilium.
    ipam:
      mode: "multi-pool"
      operator:
        autoCreateCiliumPodIPPools:
          default: # A default pool will be created by Cilium at start-up and assign prefixes to Nodes based on the `ipam.operator.autoCreateCiliumPodIPPools.default` value.
            ipv4:
              cidrs:
                - 10.10.0.0/16
              maskSize: 27
    l2announcements:
      enabled: true
    k8sClientRateLimit:
      qps: 50
      burst: 200
    hubble:
      enabled: true
      relay:
        enabled: true
      ui:
        enabled: true

Validation​

The validation needs to take place on the RKE2 side and the Cilium side. For RKE2, ensure the journal -xu rke2-server.service logs are clean, and the systemctl status rke2-server service is set to active (running). The next step is to validate the state of the cluster, Cilium, and assigned CIDRs.

$ kubectl get nodes
NAME            STATUS   ROLES                AGE     VERSION
mt08-master     Ready    control-plane,etcd   7d22h   v1.35.6+rke2r1
mt08-worker01   Ready    <none>               7d6h    v1.35.6+rke2r1
mt08-worker02   Ready    <none>               7d5h    v1.35.6+rke2r1

$ kubectl get pods -n kube-system -o wide -l k8s-app=cilium
NAME           READY   STATUS    RESTARTS       AGE
cilium-m9sp5   1/1     Running   0              7d6h
cilium-wwcbn   1/1     Running   0              7d22h
cilium-xm4qt   1/1     Running   0              7d5h

$ kubectl get ciliumpodippools.cilium.io default -o yaml
apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
  creationTimestamp: "2026-07-31T09:03:55Z"
  generation: 1
  name: default
  resourceVersion: "841"
  uid: 8987b28f-d362-4fa7-8b0d-4dd00f31dd7a
spec:
  ipv4:
    cidrs:
    - 10.10.0.0/16
    maskSize: 27

As expected, Cilium used the default pool set during the configuration. Now, we can continue with the configuration of the individual pools and LoadBalancer pools per namespace.

Cilium CRDs​

CiliumLoadBalancerIPPool​

As mentioned at the beginning of the post, we would like to assign each of the tenants its own unique CiliumLoadBalancerIPPool pool. For the vcluster-team-a tenant, we assign 11 IP addresses, while for the vcluster-team-b only 5. The flexibility of the setup is that we could extend the pool of LoadBalancer IPs based on the tenant's needs as we follow a declarative approach.

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
  name: rke2-pool-vcluster-team-a
spec:
  blocks:
  - start: 10.10.20.80
    stop: 10.10.20.90
  serviceSelector:
    matchLabels:
      io.kubernetes.service.namespace: vcluster-team-a

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
  name: rke2-pool-vcluster-team-b
spec:
  blocks:
  - start: 10.10.20.91
    stop: 10.10.20.95
  serviceSelector:
    matchLabels:
      io.kubernetes.service.namespace: vcluster-team-b

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ kubectl apply -f ciliumloadbalancerippool_team_a.yaml,ciliumloadbalancerippool_team_b.yaml

$ kubectl get CiliumLoadBalancerIPPool
NAME                          DISABLED   CONFLICTING   IPS AVAILABLE   AGE
rke2-pool-vcluster-team-a     false      false         11              3d3h
rke2-pool-vcluster-team-b     false      false         5               2d2h

CiliumPodIPPool​

Create dedicated Pools for the PodCIDRs within each tenant deployment.

apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
  name: team-a
spec:
  ipv4:
    cidrs:
      - 10.20.0.0/20
    maskSize: 24
---
apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
  name: team-b
spec:
  ipv4:
    cidrs:
      - 10.20.16.0/20
    maskSize: 24

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ kubectl apply -f ciliumpodippool_team_a.yaml,ciliumpodippool_team_b.yaml

$ kubectl get CiliumPodIPPool
NAME       AGE
default    3d20h
team-a     3d3h
team-b     3d3h

What did we achieve with the setup?​

The main goal was for us to have granural control of the CIDRs assigned to each tenant and control over the LoadBalancer IP addresses advertised by L2Announcement. For more details, take a look at the Part 2 of the series. Now, we cannot only control the IPs assigned, but also extend the setup to meet future needs of the individual tenants.

vCluster Creation​

We will not go through the vCluster configuration as this is covered extensively by parts 1 and 2 of the series. We will install the individual tenant environments and ensure they get the CIDRs based on our specification on the control plane cluster.

helm upgrade --install vcluster-tean-a vcluster \
--repo https://charts.loft.sh \
--namespace vcluster-team-a \
--create-namespace -f vcluster_team_a_lb.yaml

Once the Helm chart is applied to the control plane node, we can describe the pod and extract the logs of the vCluster pod in the vcluster-team-a namespace. Keep in mind the pod will go through different stages till it is running. The pod will follow the classic pod creation workflow: kubelet -> Container Runtime -> CNI Plugin Invoke -> Pod creation.

A snippet of the vCluster Pod might look like the below.

Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               112s               default-scheduler  Successfully assigned vcluster-team-a/vcluster-team-a-0 to mt08-worker01
  Warning  FailedCreatePodSandBox  112s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7177be8a6ed3f41734d8618b3c3124dc4f7a3800da4d97b3addc2664ecc92dda": plugin type="cilium-cni" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure "unable to allocate from pool \"team-a\" (family ipv4): pool not (yet) available"
  Normal   Pulled                  100s               kubelet            spec.initContainers{kubernetes}: Container image "ghcr.io/loft-sh/kubernetes:v1.36.0" already present on machine and can be accessed by the pod
  Normal   Created                 100s               kubelet            spec.initContainers{kubernetes}: Container created
  Normal   Started                 99s                kubelet            spec.initContainers{kubernetes}: Container started
  Normal   Pulled                  96s                kubelet            spec.containers{syncer}: Container image "ghcr.io/loft-sh/vcluster-pro:0.36.0" already present on machine and can be accessed by the pod
  Normal   Created                 96s                kubelet            spec.containers{syncer}: Container created

Validation​

$ export KUBECONFIG=/path/to/tenant/kubeconfig

$ kubectl get nodes -o wide
NAME            STATUS   ROLES    AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
mt08-worker01   Ready    <none>   2d3h   v1.36.0   10.43.114.26   <none>        Fake Kubernetes Image   4.19.76-fakelinux (amd64)   docker://19.3.12

$ kubectl get pods -n kube-system
NAME                      READY   STATUS    RESTARTS   AGE    IP             NODE            NOMINATED NODE   READINESS GATES
coredns-97ddf668d-7hdgr   1/1     Running   0          2d3h   10.20.18.184   mt08-worker01   <none>           <none>

Advantages of Setup​

• Predictable Egress traffic: Each tenant uses a distinct CIDR. Firewalls and monitoring tools can identify the source tenant by IP
• Dedicated LoadBalancer pool: With the CiliumLoadBalancerIPPool scoped per namespace, one tenant cannot exhaust the IP pool of another. Each team gets its own range, and we can extend it declaratively when required
• Dynamic Tenant Onboarding: Adding a new tenant means creating a new CiliumPodIPPool, a new CiliumLoadBalancerIPPool, and a vCluster Helm release.

Conclusion​

The proposed setup allows us to work on the next topic, which is Cilium and Gateway API. We will demonstrate how to share Gateway API resources between the control plane cluster and the underlying tenants, while in the final part of the series, we will advance the setup to include Grafana authentication using Keycloack OIDC and Cilium Gateway API. Stay tuned.

Resources​

• Cilium Up & Running
• Cilium Multi-Pool IPAM

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​

In part 2, we introduced the concept of using the Sveltos Event Framework and templating capabilities to dynamically instantiate a Flux HelmRelease based on cluster labels. This blog brings to life the concept of automated Flux HelmReleases using a hub-and-spoke case scenario. We will run a local Kubernetes fleet using KinD, bootstrap Flux, and watch Sveltos automatically generate and deploy Flux HelmReleases based on labels.

Motivation​

Platform teams already have a working Flux setup configured to their specifications and needs. Flux is used to deliver applications, manage infrastructure, and, most importantly, the teams know how to operate it. The challenge comes when adding, extending, and scaling workflows without rewriting everything or introducing yet another layer of complexity that makes operations harder.

The demo setup is based on the official Flux Hub and Spoke example repository. It aims to show teams how to integrate Sveltos into an existing Flux setup in a clean, non-disruptive way. We add Sveltos on top, and immediately we profit from label-driven automation, dynamic templating, and scalability that expands from 2 clusters to 50, 100, 10000 while keeping the repository size minimal.

The demo highlights two distinct events;

1. The cluster bootstrap approach, where Sveltos detects the label type: spoke on the managed clusters, Sveltos generates the Flux Kustomizations dynamically, and then Flux reconciles to the spoke clusters
2. The application deployment, where Sveltos detects the label cert-manager: required|kyverno: required, Sveltos generates a HelmRelease dynamically, and then the Flux helm-controller deploys to the specified spoke clusters

Lab Setup​

+-------------------------------+---------------------+
|          Deployment           |       Version       |
+-------------------------------+---------------------+
|           Sveltos             |       v1.12.0       |
|          Flux2 Helm           |       v2.18.3       |
|      Flux Operator Helm       |       v0.40.0       |
+-------------------------------+---------------------+

GitHub Resources​

The YAML outputs are not complete. Have a look at the GitHub repository.

Prerequisites​

1. Docker and KinD installed
2. Flux Command Line for the bootstrap process
3. make utility installed
4. Familiarity with Kubernetes manifest files
5. Familiarity with Flux and Sveltos

Flux Hub and Spoke Original Repository Structure​

Before we start talking about potential changes to the official Flux Hub and Spoke repository, let's take a look at the project structure and the logic behind it and its purpose.

Flux runs only on the hub cluster and plays the role of the "brain" around operations. Once Flux is deployed, the Flux controllers are responsible for reconciling workloads to different spoke clusters using their kubeconfig secrets. Everything related to clusters and their configuration is stored under the hub/ directory as a static YAML file.

Repository Outline​

Looking at the current project outline, it uses a static, explicit approach where each cluster's configuration is defined in the repository.

flux2-hub-spoke-example/
├── hub/                          # Hub cluster configuration
│   ├── flux-system/              # Flux bootstrap manifests
│   ├── staging.yaml              # Static Kustomizations for staging cluster
│   └── production.yaml           # Static Kustomizations for production cluster
│
├── clusters/                     # Per-cluster Kustomize overlays
│   ├── staging/
│   │   ├── tenants/
│   │   ├── infra-controllers/
│   │   ├── infra-configs/
│   │   └── apps/
│   └── production/
│   │   ├── tenants/
│   │   ├── infra-controllers/
│   │   ├── infra-configs/
│   │   └── apps/
│
└── deploy/                       # Base manifests
    ├── tenants/                  # Namespaces, ServiceAccounts, RBAC etc.
    ├── infra-controllers/        # Controller Flux HelmReleases
    ├── infra-configs/            # Infrastructure Components
    └── apps/                     # podinfo HelmRelease

To add a new cluster to the setup, a number of things need to happen before we can provision anything to the spoke cluster. First, create a new cluster under hub/<your cluster name>.yaml alongside four Kustomizations. Create a new overlay under clusters/<your cluster name>/. Provision the kubeconfig secret in the hub cluster and finally, commit and push the changes to Git so that Flux applies the defined configuration to this brand new cluster.

In today's demonstration, we would like to offer an alternative, fresh approach and propose a few changes to the existing repository structure to allow Sveltos flexibility to extend the setup and support automatic deployment of resources while keeping the manifest in the repository minimal.

Flux and Sveltos Repository Structure​

The whole idea is to give teams the flexibility to scale and expand their fleet with minimal effort, fewer changes, and fewer headaches. Once Sveltos is up and running, we deploy a few manifests to describe Sveltos logic and how it should work when a new spoke cluster is detected. More information about the files, outline, and logic is provided later in the post.

What Remains Unchanged​

The deploy/, clusters/<env>/, and hub/flux-system/ directories are untouched. Later on, we will see how Sveltos is used to dynamically bootstrap the managed clusters and deploy the information stored under the deploy/ directory.

What Changes​

The static hub/staging.yaml and hub/production.yaml files are replaced by dynamic generation taken over by the Sveltos Event Framework when a new managed cluster with the label type: spoke is detected.

The changes are introduced primarily in the hub/ directory as it reflects the brain of our operations. We kept the Flux configuration and added the Sveltos installation details and its logic to perform event-driven decisions based on spoke clusters with specific labels applied. The clusters/ directory contains the sveltos-clusters/ definition, which is used in this particular demo to apply the right labels to the managed clusters.

hub/
- hub/sveltos-install.yaml # Flux Kustomization to install Sveltos
- hub/sveltos-event-framework.yaml # Flux Kustomization to apply EventSources/EventTriggers
- hub/sveltos-clusters.yaml # Flux Kustomization to register SveltosClusters
- hub/sveltos/ # Sveltos main logic
  - install/ # Sveltos Flux HelmRelease
  - event-framework/event-sources/ # Sveltos EventSource resources
  - event-framework/event-triggers/ # EventTrigger resources and ConfigMap templates

clusters/
- clusters/sveltos-clusters/
  - hub.yaml # Hub cluster registration with `type: mgmt` label
  - staging.yaml # Namespace + SveltosCluster with labels (`type: spoke`, `environment: staging`)
  - production.yaml # Namespace + SveltosCluster with labels (`type: spoke`, `environment: production`)

Updated Repository Structure​

flux2-hub-spoke-example/
├── hub/
│   ├── flux-system/
│   ├── kustomization.yaml
│   ├── sveltos-install.yaml              # Installs Sveltos as a Flux HelmRelease
│   ├── sveltos-event-framework.yaml      # Applies the Sveltos Event Framework
│   ├── sveltos-clusters.yaml             # Automatically registers SveltosClusters (managed/spoke clusters)
│   └── sveltos/                          # Sveltos configuration and logic
│       ├── install/
│       │   ├── kustomization.yaml
│       │   └── sveltos-helmrelease.yaml
│       └── event-framework/
│           ├── kustomization.yaml
│           ├── event-sources/            # Sveltos Event Sources to detect when a cluster needs a particular deployment
│           │   ├── cert-manager.yaml
│           │   ├── kustomization.yaml
│           │   ├── kyverno.yaml
│           │   └── spoke-cluster.yaml
│           └── event-triggers/            # Sveltos Event Trigger to perform a specific deployment based on an Event
│               ├── bootstrap-cluster.yaml
│               ├── cert-manager.yaml
│               ├── kustomization.yaml
│               └── kyverno.yaml
├── clusters/
│   ├── production/                       # The production/ uses the old deployment approach
│   ├── staging/                          # The staging/ uses the old deployment approach
│   └── sveltos-clusters/                 # Includes the dynamic cluster registrations powered by Sveltos suitable for this demo
│       ├── hub.yaml
│       ├── kustomization.yaml
│       ├── production.yaml
│       └── staging.yaml
└── deploy/                               # Unchanged base manifests

Demo Flow Diagram​

Deployment Order Details​

The deployment of the demo follows a logical continuation. We start by creating three Kubernetes clusters. Once those are ready, we perform a Flux bootstrap on the management cluster and ensure Flux is healthy. Once this is done, Sveltos is installed in the management cluster using a Kustomization pointing to the ./hub/sveltos/install path. When Sveltos is healthy, we continue with the Sveltos Event Framework manifests and the registration of the managed clusters with Sveltos.

The Sveltos Event Framework follows as it detects managed clusters with specific labels. For example, once a managed cluster with the label set to type: spoke is detected, Sveltos will bootstrap the clusters using Kustomization pointing to the paths ./clusters/{{ .Resource.metadata.name }}/apps, ./clusters/{{ .Resource.metadata.name }}/infra-controllers and ./clusters/{{ .Resource.metadata.name }}/tenants. Traefik and podinfo will get deployed on the managed clusters.

Finally, cert-manager and kyverno are deployed on demand when a managed cluster with the labels set to cert-manager: required and kyverno: required is spotted. They are expressed as Sveltos templates and are dynamically instantiated using information from the management cluster. A HelmRepository and a HelmRelease are dynamically created for every cluster that needs them.

Running the Demo​

The repository has already been modified with the proposed changes to give readers the chance to execute it locally in any environment running Docker and KinD.

Step 1: Fork and Clone​

Fork the flux2-sveltos-hub-spoke-blog-demo repository and clone it in a local directory.

$ git clone https://github.com/<your-username>/flux2-sveltos-hub-spoke-blog-demo && cd flux2-sveltos-hub-spoke-blog-demo

Step 2: Fleet-up​

By executing the make fleet-up command, we create three KinD clusters. One reflects the management cluster where Flux and Sveltos are installed. The other two serve as the managed clusters, which are registered with Sveltos and have specific labels assigned.

$ make fleet-up

The script will perform a programmatic Sveltos cluster registration as described in the official guide. Additionally, a number of secrets required by Flux and Sveltos are created during this phase.

Step 3: Bootstrap Flux​

This is the most important part, as we need to have a working Flux deployment before we continue. To bootstrap Flux, we can either use the make utility and perform a make flux-up, or we can use the Flux CLI.

Automatic Flux Bootstrap​

Ensure the scripts/flux-up.sh is updated to reflect the repository of interest and authentication credentials. Once updated, perform make flux-up and wait until Flux is initialised in the management cluster.

Manual Flux Bootstrap​

To perform a manual Flux bootstrap, ensure the following points are satisfied. Start by exporting the required repository and authentication variables. Ensure the hub/flux-system/gotk-sync.yaml file in the new fork is updated accordingly.

$ export GITHUB_TOKEN=<your-token>
$ export GITHUB_USER=<your-username>
$ export GITHUB_REPO=<repo-name>

Then, bootstrap Flux to the management cluster.

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ flux bootstrap github \
    --token-auth \
    --context=kind-hub \
    --owner=${GITHUB_USER} \
    --repository=${GITHUB_REPO} \
    --branch=main \
    --personal \
    --path=hub

The flux bootstrap github command installs the Flux controllers in the management to reconcile everything under the hub/ directory. Once the phase is complete, Flux will automatically perform a number of tasks.

• Install Sveltos via the sveltos-install Kustomization
• Register the two managed clusters via the sveltos-clusters
• Apply the Sveltos Event Framework manifests to the management cluster via sveltos-event-framework

Step 4: Validation​

If the Flux CLI is already installed, we can perform the watch flux get kustomizations -A command to list the kustomization status in the management cluster. As every resource is controlled and managed by Flux, the output will give us a good understanding of what is happening within the cluster, alongside what works and what does not.

Initial Flux bootstrap​

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ flux get kustomizations -A
NAMESPACE       NAME            REVISION                SUSPENDED       READY   MESSAGE                              
flux-system     flux-system     main@sha1:eab79367      False           True    Applied revision: main@sha1:eab79367

Sveltos Deployment and Manifests​

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ flux get kustomizations -A   
NAMESPACE       NAME                    REVISION                SUSPENDED       READY   MESSAGE                              
flux-system     flux-system             main@sha1:eab79367      False           True    Applied revision: main@sha1:eab79367
flux-system     sveltos-clusters        main@sha1:eab79367      False           True    Applied revision: main@sha1:eab79367
flux-system     sveltos-event-framework main@sha1:eab79367      False           True    Applied revision: main@sha1:eab79367
flux-system     sveltos-install         main@sha1:eab79367      False           True    Applied revision: main@sha1:eab79367

All-in-One Output​

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ flux get kustomizations -A -w
NAMESPACE       NAME                    REVISION                SUSPENDED       READY   MESSAGE                              
flux-system     flux-system             main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
flux-system     sveltos-clusters        main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
flux-system     sveltos-event-framework main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
flux-system     sveltos-install         main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
production      apps                    main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
production      infra-controllers       main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
production      tenants                 main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
staging         apps                    main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
staging         infra-controllers       main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24
staging         tenants                 main@sha1:19c2ef24      False           True    Applied revision: main@sha1:19c2ef24

Notice how staging and production Kustomizations appeared automatically. Sveltos generated those dynamically using the Event Framework and by detecting managed clusters with the label set to type: spoke!

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ kubectl get helmrelease.helm.toolkit.fluxcd.io -A
NAMESPACE     NAME                      AGE   READY   STATUS
flux-system   projectsveltos            35h   True    Helm install succeeded for release projectsveltos/projectsveltos-projectsveltos.v1 with chart projectsveltos@1.11.4
production    cert-manager-production   35h   True    Helm install succeeded for release cert-manager/cert-manager-cert-manager-production.v1 with chart cert-manager@v1.18.6
production    kyverno-production        35h   True    Helm install succeeded for release kyverno/kyverno-kyverno-production.v1 with chart kyverno@3.6.3
production    podinfo                   35h   True    Helm install succeeded for release podinfo/podinfo.v1 with chart podinfo@6.14.0
production    traefik                   35h   True    Helm install succeeded for release traefik/traefik.v1 with chart traefik@40.3.0
staging       cert-manager-staging      35h   True    Helm install succeeded for release cert-manager/cert-manager-cert-manager-staging.v1 with chart cert-manager@v1.19.4
staging       kyverno-staging           35h   True    Helm install succeeded for release kyverno/kyverno-kyverno-staging.v1 with chart kyverno@3.7.1
staging       podinfo                   35h   True    Helm install succeeded for release podinfo/podinfo.v1 with chart podinfo@6.14.0
staging       traefik                   35h   True    Helm install succeeded for release traefik/traefik.v1 with chart traefik@40.3.0

From the output above, it is clear that Sveltos dynamically created the Flux HelmReleases based on an event. The event is the managed clusters with the right set of labels.

How does Sveltos work?​

The following sections and sub-sections describe how Sveltos logic is applied and how to automatically register and deploy resources across the managed clusters using a label approach. For this demo, the managed clusters are registered with Sveltos using the labels type: spoke, environment: staging|production, cert-manager: required, and kyverno: required.

Having different labels in place, we cannot only distinguish in an easy manner the different environments, but also add or remove add-ons and applications based on the identity of the cluster.

Pre-validation​

SveltosClusters Output​

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ kubectl get sveltosclusters -A --show-labels
NAMESPACE    NAME         READY   VERSION   AGE     SHARD   LABELS
mgmt         mgmt         true    v1.35.0   4h37m           kustomize.toolkit.fluxcd.io/name=sveltos-clusters,kustomize.toolkit.fluxcd.io/namespace=flux-system,projectsveltos.io/k8s-version=v1.35.0,sveltos-agent=present,type=mgmt
production   production   true    v1.35.0   4h37m           cert-manager=required,environment=production,kustomize.toolkit.fluxcd.io/name=sveltos-clusters,kustomize.toolkit.fluxcd.io/namespace=flux-system,kyverno=required,projectsveltos.io/k8s-version=v1.35.0,sveltos-agent=present,type=spoke
staging      staging      true    v1.35.0   4h37m           cert-manager=required,environment=staging,kustomize.toolkit.fluxcd.io/name=sveltos-clusters,kustomize.toolkit.fluxcd.io/namespace=flux-system,kyverno=required,projectsveltos.io/k8s-version=v1.35.0,sveltos-agent=present,type=spoke

Notice the custom labels added to each of the created clusters.

EventSource and EventTrigger Resources​

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ kubectl get eventsource,eventtrigger -A
NAME                                                                      AGE
eventsource.lib.projectsveltos.io/detect-cluster-requiring-cert-manager   4m7s
eventsource.lib.projectsveltos.io/detect-cluster-requiring-kyverno        4m7s
eventsource.lib.projectsveltos.io/detect-spoke-cluster                    4m7s

NAME                                                         AGE
eventtrigger.lib.projectsveltos.io/bootstrap-spoke-cluster   4m7s
eventtrigger.lib.projectsveltos.io/deploy-cert-manager       4m7s
eventtrigger.lib.projectsveltos.io/deploy-kyverno            4m7s

The different Sveltos resources are applied in the management cluster.

Sveltos ClusterProfile and ClusterSummary Resources​

$ export KUBECONFIG=/path/to/management cluster/kubeconfig

$ kubectl get clusterprofile,clustersummary -A

NAME                                                                   AGE
clusterprofile.config.projectsveltos.io/sveltos-2ned8r1c51d5m82fxyz7   4h39m
clusterprofile.config.projectsveltos.io/sveltos-3xtrl5sg1dgzqnspfup7   4h39m
clusterprofile.config.projectsveltos.io/sveltos-6bkznpyd5kbp3wujgftg   4h39m
clusterprofile.config.projectsveltos.io/sveltos-bs7nw7dhsper5pypyxbx   4h39m
clusterprofile.config.projectsveltos.io/sveltos-foajrwp2g7y3ablcstdr   4h39m
clusterprofile.config.projectsveltos.io/sveltos-sy2obzjictcys4y2w1je   4h39m

NAMESPACE   NAME                                                                                AGE
mgmt        clustersummary.config.projectsveltos.io/sveltos-2ned8r1c51d5m82fxyz7-sveltos-mgmt   4h39m
mgmt        clustersummary.config.projectsveltos.io/sveltos-3xtrl5sg1dgzqnspfup7-sveltos-mgmt   4h39m
mgmt        clustersummary.config.projectsveltos.io/sveltos-6bkznpyd5kbp3wujgftg-sveltos-mgmt   4h39m
mgmt        clustersummary.config.projectsveltos.io/sveltos-bs7nw7dhsper5pypyxbx-sveltos-mgmt   4h39m
mgmt        clustersummary.config.projectsveltos.io/sveltos-foajrwp2g7y3ablcstdr-sveltos-mgmt   4h39m
mgmt        clustersummary.config.projectsveltos.io/sveltos-sy2obzjictcys4y2w1je-sveltos-mgmt   4h39m

The available resources triggered by the Event Framework while Sveltos detected spoke clusters with specific labels set.

$ kubectl get clusterprofile,clustersummary -A
NAME                                                                   AGE
clusterprofile.config.projectsveltos.io/cert-manager-helmrepository    10m
clusterprofile.config.projectsveltos.io/kyverno-helmrepository         10m
clusterprofile.config.projectsveltos.io/mgmt-production-certmanager    115s
clusterprofile.config.projectsveltos.io/mgmt-production-kyverno        115s
clusterprofile.config.projectsveltos.io/mgmt-staging-certmanager       115s
clusterprofile.config.projectsveltos.io/mgmt-staging-kyverno           115s
clusterprofile.config.projectsveltos.io/sveltos-84646v00sc2ep9kutpif   10m
clusterprofile.config.projectsveltos.io/sveltos-9ne8pa9e04d3scfqomuo   10m

NAMESPACE   NAME                                                                                AGE
mgmt        clustersummary.config.projectsveltos.io/cert-manager-helmrepository-sveltos-mgmt    10m
mgmt        clustersummary.config.projectsveltos.io/kyverno-helmrepository-sveltos-mgmt         10m
mgmt        clustersummary.config.projectsveltos.io/mgmt-production-certmanager-sveltos-mgmt    115s
mgmt        clustersummary.config.projectsveltos.io/mgmt-production-kyverno-sveltos-mgmt        115s
mgmt        clustersummary.config.projectsveltos.io/mgmt-staging-certmanager-sveltos-mgmt       115s
mgmt        clustersummary.config.projectsveltos.io/mgmt-staging-kyverno-sveltos-mgmt           115s
mgmt        clustersummary.config.projectsveltos.io/sveltos-84646v00sc2ep9kutpif-sveltos-mgmt   10m
mgmt        clustersummary.config.projectsveltos.io/sveltos-9ne8pa9e04d3scfqomuo-sveltos-mgmt   10m

Sveltos Installation via Flux​

The Sveltos installation is declared as a Flux Kustomization defined in the hub/sveltos-install.yaml file. The reference directory defines a HelmRepository and a HelmRelease used by Flux to deploy Sveltos in the management cluster. Find the full demo code here.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: sveltos-install
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 3m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./hub/sveltos/install
  prune: true
  wait: true

Once Sveltos is deployed, it auto-registers the management cluster as a SveltosCluster instance under the mgmt namespace. The sveltos-clusters Kustomization patches the type: mgmt label onto it so Sveltos Event Framework can manage resources in the management cluster.

How does the Sveltos Event Framework Work?​

The Sveltos resources are applied to the management cluster using Kustomization. The Sveltos Event Framework is the magic behind the automation. Two important resources are used: the EventSource and the EventTrigger.

EventSource​

An EventSource tells Sveltos what to watch for. We want to watch for the labels below and deploy the required add-on to the managed cluster. Thus, three EventSource resources have been created.

1. detect-spoke-cluster: It watches for new clusters with the label set to type: spoke
2. detect-cluster-requiring-cert-manager: It watches for clusters with the label set to cert-manager: required
3. detect-cluster-requiring-kyverno: It watches for clusters with the label set to kyverno: required

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: detect-cluster-requiring-cert-manager
spec:
  collectResources: true
  resourceSelectors:
  - group: "lib.projectsveltos.io"
    version: "v1beta1"
    kind: "SveltosCluster"
    labelFilters:
    - key: cert-manager
      operation: Equal
      value: required

Sveltos watches the SveltosCluster resources. When it detects, for example, a managed cluster with the label set to cert-manager: required, it fires the associated EventTrigger.

EventTrigger​

An EventTrigger defines what happens when an event is detected. The sourceClusterSelector defines where to detect the event, the destinationClusterSelector defines where to apply the rendered resources, the eventSourceName defines the EventSource to watch, and finally, the policyRefs list any ConfigMaps or Secrets containing information to be applied.

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: deploy-cert-manager
spec:
  sourceClusterSelector:
    matchLabels:
      type: mgmt
  destinationClusterSelector:
    matchLabels:
      type: mgmt
  eventSourceName: detect-cluster-requiring-cert-manager
  oneForEvent: true
  policyRefs:
  - name: cert-manager-helmreleases
    namespace: default
    kind: ConfigMap

ConfigMap cert-manager-helmreleases​

The ConfigMap is what Sveltos deploys to the management cluster, and it is nothing more than a Sveltos template of a HelmRepository and a HelmRelease for cert-manager.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cert-manager-helmreleases
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data:
  helmreleases.yaml: |
    apiVersion: source.toolkit.fluxcd.io/v1
    kind: HelmRepository
    metadata:
      name: jetstack
      namespace: flux-system
    spec:
      interval: 1h
      url: https://charts.jetstack.io
    ---
    apiVersion: helm.toolkit.fluxcd.io/v2
    kind: HelmRelease
    metadata:
      name: cert-manager-{{ .Resource.metadata.name }}
      namespace: {{ .Resource.metadata.namespace }}
    spec:
      interval: 15m
      targetNamespace: cert-manager
      kubeConfig:
        secretRef:
          name: {{ .Resource.metadata.name }}-sveltos-kubeconfig
          key: kubeconfig
      chart:
        spec:
          chart: cert-manager
          version: "{{ if eq (index .Resource.metadata.labels "environment") "production" }}v1.18.6{{ else }}v1.19.4{{ end }}"
          sourceRef:
            kind: HelmRepository
            name: jetstack
            namespace: flux-system
          interval: 15m
      install:
        createNamespace: true
        timeout: 10m
        strategy:
          name: RetryOnFailure
      upgrade:
        timeout: 10m
        cleanupOnFail: true
        strategy:
          name: RetryOnFailure
      values:
        crds:
          enabled: true
          keep: true

version: "{{ if eq (index .Resource.metadata.labels "environment") "production" }}v1.18.6{{ else }}v1.19.4{{ end }}"

As we are working with templates, we have the ability to use simple logic and control what version of cert-manager should get deployed to which managed cluster. For example, if a managed cluster has the label set to environment: production, then install v1.18.6. Else, install v1.19.4. No overlay files, no duplication, just work with labels.

How to expand the Demo?​

To add a new managed cluster to the setup, it is pretty simple. Create a new managed cluster with the right set of labels. Register it with Sveltos. If the new cluster needs to get the podinfo and traefik Ingress Controller, add the required overlays.

$ mkdir -p clusters/cluster01/{tenants,infra-controllers,apps}

$ kubectl --context kind-hub create secret generic -n cluster01 cluster01-kubeconfig \
  --from-file=value=<path-to-kubeconfig>
$ kubectl --context kind-hub create secret generic -n cluster01 cluster01-sveltos-kubeconfig \
  --from-file=kubeconfig=<path-to-kubeconfig>

Clean Up​

To clean up the environment, simply execute the command below.

$ make fleet-down

Benefits​

The Flux and Sveltos combination solves the scalability problem of the traditional multi-cluster GitOps. A two-layer approach is used: Flux handles source reconciliation while Sveltos provides event-driven templating. The key benefit is effortless fleet scaling. Adding the 10th cluster requires the same single file as adding the 2nd cluster, with no growth in repository complexity. Simply label the cluster, and Sveltos will automatically generate all the necessary Flux Kustomizations and HelmReleases, eliminating manual per-cluster manifest creation.

Conclusion​

In this post, we walked through a demo of Flux and Sveltos working together. The hub-spoke model scales nicely. The repository does not grow regardless of the fleet size. Labels drive the cluster identity, workload deployment, and Helm chart version selection. Automation does not mean complexity. The Sveltos Event Framework simplifies multi-cluster deployments by making templates reusable and clusters self-describing. To learn more about how Sveltos completes the setup, take a look at the introductory blog post to Sveltos Progressive Rollouts alongside a more complex example.

Resources​

• Flux Documentation
• Sveltos GitHub Repo
• Sveltos Event Framework

✉️ Contact​

We are here to help! Whether you have questions, or issues or need assistance, our Slack channel is the perfect place for you. Click here to join us.

👏 Support this project​

Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

Series Navigation​

In part 1 of the series, we explored how Sveltos acts as the main brain for our deployments to a fleet of clusters. However, this is the case when we start with Continuous Deployments (CD) or when we are willing to perform a full migration to a new architecture. The next two posts are dedicated to the collaboration between Flux and Sveltos. Flux remains the core way of deploying applications using the Flux Customer Resource Definitions (CRDs), while Sveltos enters the play when we talk about scalability, automation, and dynamic instantiation of deployments.

Motivation​

Most engineers working in Platform Engineering already have a GitOps Controller like ArgoCD or Flux in place for Kubernetes deployments. It works, and it works well, until it does not. As teams grow and deployments spread across ten or more clusters, multiple environments, and different Hyperscalers, a once clean GitOps repository quickly ends up with duplicated Helm Releases, environment-specific overrides, and manual interventions nobody wants to own.

Having only a GitOps Controller in place is not enough when it comes to complex workloads, scalability, and management of deployments and add-ons across different environments. Teams end up juggling multiple tools just to add flexibility to an existing setup. Can we help teams work smaller, simpler, and not harder when it comes to continuous deployments? The answer is yes; follow along to explore the Sveltos magic! 🪄

Scenario​

In today's post, we will showcase a flexible way of using an existing Flux configuration, and how by adding Sveltos into the mix, we provide out-of-the-box capabilities like templating, event-driven workloads, and many more. We will work with an existing Flux deployment, and Sveltos will automate the dynamic creation of Flux HelmRelease based on a cluster type or identity.

Lab Setup​

+-------------------------------+---------------------+
|          Deployment           |       Version       |
+-------------------------------+---------------------+
|             RKE2              |   v1.35.3+rke2r3    |
|           Sveltos             |       v1.8.0        |
|          Flux2 Helm           |       v2.18.3       |
|      Flux Operator Helm       |       v0.40.0       |
+-------------------------------+---------------------+

GitHub Resources​

The YAML outputs are not complete. Have a look at the GitHub repository.

Prerequisites​

1. A Kubernetes cluster acting as the management cluster
2. At least one managed cluster
3. Familiarity with Kubernetes manifest files
4. Familiarity with Flux

Diagram​

The diagram looks very similar to the one we saw in part 1 of the series. The difference here is that we already have a Flux deployment and project outline. We will install Sveltos using the native deployment options and let Sveltos listen for Events or specific resources and automate the deployment of Flux HelmRelease. Let's dive into the details.

Sveltos Installation​

The installation of Sveltos using Flux could be similar to the below. Feel free to use your preferred way of installing Sveltos to the management cluster.

---
apiVersion: v1
kind: Namespace
metadata:
  name: projectsveltos
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: projectsveltos
  namespace: flux-system
spec:
  interval: 24h
  url: https://projectsveltos.github.io/helm-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: projectsveltos
  namespace: flux-system
spec:
  interval: 30m
  targetNamespace: projectsveltos
  storageNamespace: projectsveltos
  chart:
    spec:
      chart: projectsveltos
      version: ">=1.6.1"
      sourceRef:
        kind: HelmRepository
        name: projectsveltos
        namespace: flux-system
      interval: 12h
  install:
    crds: Create
    createNamespace: true
    timeout: 10m
    strategy:
      name: RetryOnFailure
  upgrade:
    crds: CreateReplace
    timeout: 10m
    cleanupOnFail: true
    strategy:
      name: RetryOnFailure

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - sveltos-helmrelease.yaml

The above yaml definition will install Sveltos to the management cluster in the projectsveltos nasmespace. The namespace is not negotiable. Sveltos has to be installed in this namespace.

Label Management Cluster​

To control resources in the management cluster with Sveltos, we will simply add the label type: mgmt to the sveltoscluster named mgmt in the mgmt namespace. The registration is done by Sveltos during installation. No manual intervention is needed. When we refer to the management cluster, is the cluster where Flux and Sveltos are installed.

$ kubectl label sveltoscluster mgmt -n mgmt type=mgmt

Automate Flux HelmRelease​

How does it work?​

Sveltos works very well with the concept of labeling. We can label clusters with a dedicated key: value pair and use this information to dynamically create Flux HelmRelease based on our needs. In this example, everytime a cluster with the label cert-manager: required appears, we trigger an action and dynamically pre-instantiate and deploy a Flux Helm Release using the information located in the management cluster. To achieve our goal, we use the Sveltos Event Framework.

Define an EventSource​

A Sveltos EventSource is a simple way of instructing Sveltos to look for an event or a specific resource. This can be an Event within a cluster or an Event outside a cluster like NATs. In our case, we want to detect clusters with the label set to cert-manager: required.

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: detect-cluster-requiring-cert-manager
spec:
  collectResources: true
  resourceSelectors:
  - group: "lib.projectsveltos.io"
    version: "v1beta1"
    kind: "SveltosCluster"
    labelFilters:
    - key: cert-manager
      operation: Equal
      value: required

Define an EventTrigger​

Once an EventSource is detected, an action or multiple actions can be triggered. In this example, once an event is detected, we will deploy a ConfigMap to the management cluster which includes the Flux HelmRelease details expressed as a Sveltos template.

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: deploy-cert-manager
spec:
  sourceClusterSelector:
    matchLabels:
      type: mgmt
  destinationClusterSelector:
    matchLabels:
      type: mgmt
  eventSourceName: detect-cluster-requiring-cert-manager
  oneForEvent: true
  policyRefs:
  - name: cert-manager-helmrelease
    namespace: default
    kind: ConfigMap

Automate Flux HelmRelease​

Now that we have our EventSource and EventTrigger in place, we need to define what actually gets deployed when an event is detected. We do this using a ConfigMap that holds the Flux Helm Release definition expressed as a Sveltos template.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cert-manager-helmrelease
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data:
  cert-manager.yaml: |
    apiVersion: helm.toolkit.fluxcd.io/v2
    kind: HelmRelease
    metadata:
      name: cert-manager-{{ .Resource.metadata.name  }}
      namespace: {{ .Resource.metadata.namespace }}
    spec:
      interval: 15m
      kubeConfig:
        secretRef:
          name: {{ .Resource.metadata.name }}-sveltos-kubeconfig
          key: kubeconfig
      chart:
        spec:
          chart: cert-manager
          version: "v1.16.x"
          sourceRef:
            kind: HelmRepository
            name: jetstack
            namespace: flux-system
          interval: 15m
      install:
        createNamespace: true
        timeout: 10m
        remediation:
          retries: 3
      upgrade:
        timeout: 10m
        cleanupOnFail: true
        remediation:
          retries: 3
          strategy: rollback
      values:
        crds:
          enabled: true
          keep: true

The annotation projectsveltos.io/instantiate: ok is what converts a plain ConfigMap into a Sveltos template. Sveltos will pull information directly from the management cluster and dynamically pre-instantiate and deploy the resource using the detected cluster's metadata. Notice how {{ .Resource.metadata.name }} and {{ .Resource.metadata.namespace }} are automatically resolved per cluster, one template, many clusters. The ConfigMap can be further templatised based on different use cases. Both Lua and CEL languages are supported for Sveltos templating.

Result​

Every time a cluster with the label cert-manager: required appears, Sveltos detects the event, resolves the template, and dynamically creates a dedicated Flux Helm Release using that cluster's information. No manual intervention, no duplicated YAML. One template, one label, done.

Conclusion​

In this post, we explored how Sveltos and Flux can work together without replacing one another. Flux remains the backbone of your GitOps workflow, while Sveltos adds the intelligence layer on top, handling scalability, dynamic templating, and event-driven automation with minimal overhead. By combining the EventSource, EventTrigger, and a templated ConfigMap, we reduced what would otherwise be repetitive, manually maintained Flux Helm Release definitions into a single, reusable template. The moment a cluster is labeled cert-manager: required, Sveltos takes over and does the heavy lifting automatically.

The key takeaway: you do not need to abandon your existing Flux setup to benefit from Sveltos. You layer it in, and your platform becomes smarter, not more complex.

What's Next?​

In the next post, we will work with a simple hub-spoke Flux deployment as our basis, and include Sveltos installation alongside the Event Framework approach to automate Flux HelmRelease. Stay tuned!

Resources​

• Flux Operator Documentation
• Sveltos Quick Start
• Sveltos Event Framework

✉️ Contact​

We are here to help! Whether you have questions, or issues or need assistance, our Slack channel is the perfect place for you. Click here to join us.

👏 Support this project​

Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

Series Navigation​

In parts 1 and 2 of the series, we explored the different node assignment strategies, Cilium L2 Announcements, and the vCluster Platform. Today, we take a step further. We will look at how Cilium CNI running on the underlying Control Plane cluster directly enhances networking for virtual clusters. We will cover traffic isolation, policy enforcement, DNS resolution, and traffic observability.

Source

Introduction​

The promise of vCluster is to create separate and isolated control planes for virtual clusters running on top of an underlying Control Plane cluster. The different components are placed in a separate namespace for logical isolation. However, this does not ensure that the running workloads are isolated. In a Kubernetes cluster, every pod can communicate with any other pod by default. What does that really mean? Yes, the end-user will not know they are on a shared cluster, but each of the workloads running could reach workloads on any other virtual cluster.

Lab Setup​

+-------------------------+--------------+------------------+
|        Resources        |     Type     |     Version      |
+-------------------------+--------------+------------------+
|  Control Plane Cluster  |     RKE2     | v1.34.3+rke2r1   |
|     vcluster-team-a     |     K8s      |     v1.36.0      |
|     vcluster-team-b     |     K8s      |     v1.36.0      |
+-------------------------+--------------+------------------+

GitHub Resources​

The YAML outputs are not complete. Have a look at the GitHub repository.

Prerequisites​

Go through parts 1 and 2 of the series to gain a better understanding of the concept and what we aim to achieve.

vCluster Network Model​

Before we jump into the examples and command outputs, it helps to understand the layered networking model. Two layers exist when we create virtual clusters.

+-----------------------------------------------+
|           Virtual Cluster (vcluster-team-a)    |
|  Pod A  <-->  Pod B   (virtual cluster DNS)    |
+------------------+----------------------------+
                   | Synced to Control Plane namespace
+------------------v----------------------------+
|     Control Plane Cluster (RKE2 + Cilium)     |
|  pod/app-x-ns-x-vcluster-team-a               |
|  pod/app-y-ns-x-vcluster-team-a               |
|  Cilium enforces eBPF-based policy.           |
+-----------------------------------------------+

When we deploy a pod inside vcluster-team-a, the vCluster syncs the pod down to the Control Plane cluster's namespace, vcluster-team-a. The pod gets an IP as it would normally do. The virtual cluster's internal DNS and service abstraction sit on top, but actual packet forwarding is handled by Cilium.

vCluster provides control plane isolation. Each virtual cluster has its own API server, scheduler, and controller manager. Tenants cannot see each other's Kubernetes resources through the API. However, vCluster does not provide network-level isolation between tenant namespaces by default.

To better understand this concept, we deployed an NGINX application in the default namespace. This was done for the vcluster-team-a virtual cluster during its creation. Below are the two separate views. One from the Control Plane cluster and one from the vcluster-team-a virtual cluster.

$ export KUBECONFIG=control-plane-cluster.yaml

$ kubectl get pods -n vcluster-team-a -o wide
NAME                                                       READY   STATUS    RESTARTS      AGE   IP            NODE   NOMINATED NODE   READINESS GATES
coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a   1/1     Running   1 (12d ago)   13d   10.42.0.48    el07   <none>           <none>
nginx-6797d5487-27gcf-x-default-x-vcluster-team-a          1/1     Running   0             11d   10.42.0.125   el07   <none>           <none>
vcluster-team-a-0                                          1/1     Running   0             11d   10.42.0.249   el07   <none>           <none>

$ export KUBECONFIG=vcluster-team-a.yaml

$ kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx-6797d5487-27gcf   1/1     Running   0          11d   10.42.0.125   el07   <none>           <none>

We already have a test pod in the vcluster-team-b virtual cluster, and we will ping the NGINX pod using the assigned IP address 10.42.0.125. As there is no network isolation, the ping would be successful. For network-related tests, feel free to use the nicolaka/netshoot Docker image.

$ export KUBECONFIG=vcluster-team-b.yaml

$ kubectl get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
test-pod   1/1     Running   0          11d   10.42.1.48   el07-worker1   <none>           <none>

$ kubectl exec -it test-pod -- ping -c3 10.42.0.125
PING 10.42.0.125 (10.42.0.125) 56(84) bytes of data.
64 bytes from 10.42.0.125: icmp_seq=1 ttl=63 time=0.165 ms
64 bytes from 10.42.0.125: icmp_seq=2 ttl=63 time=0.212 ms

From the test-pod, we can also cURL the NGINX service IP address. We will get a response as expected.

Observations​

• Pods from vcluster-team-a are synced into the vcluster-team-a namespace on the Control Plane cluster
• Pods from vcluster-team-b are synced into the vcluster-team-b namespace on the Control Plane cluster
• These namespaces sit on the same flat network managed by Cilium
• If someone knows the pod IP or service IP of a workload in vcluster-team-b, a pod in vcluster-team-a can reach it by default

For that reason, a modern CNI with capabilities and features made for complex and modern workloads is a must-have for every Kubernetes environment. Thus, Cilium is the preferred CNI for multi-tenant environments.

Cilium Hubble Observability​

To gain the most out of our Cilium installation, we can enable Hubble, Hubble UI, and Hubble Relay (optional) to have a visual representation of the traffic. We can expose the Hubble UI service as a LoadBalancer or NodePort, or simply port-forward it. Choose the option that suits your setup.

Helm Chart Values​

hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true

$ helm upgrade rke2-cilium rke2-charts/rke2-cilium --version 1.18.300 --namespace kube-system -f values_control_plane.yaml

Hubble UI View​

Once Hubble UI is accessible, we can choose the vcluster-team-a or the vcluster-team-b namespace view and explore the traffic flow. Keep in mind that both virtual clusters are registered with the vCluster Platform, thus we see traffic to TCP port 10443.

Traffic View vcluster-team-a​

Traffic View vcluster-team-b​

Network Isolation​

Observation at the Control Plane cluster is enabled, and we are ready to continue with the network isolation. As we already saw, there is no isolation between the different virtual clusters running on a Kubernetes cluster. Because virtual clusters are assigned to different teams, we want traffic to be allowed on the namespace they belong to, but blocked on other namespaces. We need to set up strong isolation and ensure different workloads are isolated. To achieve our goal, we have two options: either use the vCluster NetworkPolicy option provided during deployment and save time using the defaults, or use the Cilium advanced networking capabilities.

Network Isolation: vCluster Helm Values​

When we create virtual clusters, we can define network isolation by using the Helm values and working with the policies.networkPolicy configuration. By default, this will allow traffic between pods within a tenant cluster, block traffic from other namespaces, and permit DNS and API server communication. However, with this approach, we use the NetworkPolicy resource and not Cilium's full capabilities. For more details, take a look at the official documentation on network policies.

Network Isolation: Cilium​

CiliumNetworkPolicy enhances Kubernetes security by using eBPF to filter traffic without the performance degradation of traditional iptables. It goes beyond simple IP-based rules to enforce identity-based security, while enabling deep Layer 7 filtering for specific HTTP paths, methods, and FQDNs.

Enforce Tenant Network Isolation​

Crafting a CiliumNetworkPolicy for the underlying virtual clusters was more complex than anticipated. It is not only about the communication between the pods within the same namespace, but also the communication between the virtual cluster and the API server, kubectl execution from machines on the local network, DNS resolution, etc. I started with the simplest possible version.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: isolate-vcluster-team-a
  namespace: vcluster-team-a
spec:
  endpointSelector: {}  #Applies this policy to all pods in the namespace
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a

Applied the manifest to the Control Plane cluster, and guess what, kubectl commands did not work from another machine in the same network. The next step was to think about Ingress traffic. The CiliumNetworkPolicy was modified. This time, we scope external management access using the fromCIDR and the fromEntities host for node local traffic.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: isolate-vcluster-team-a
  namespace: vcluster-team-a
spec:
  endpointSelector: {}  #Applies this policy to all pods in the namespace
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a
    - fromCIDR:
        - <IP Subnet> # e.g. 10.x.x.x/24 Replace with your management network CIDR
      toPorts:
        - ports:
            - port: "8443"
              protocol: TCP
    - fromEntities:
        - host # Matches traffic routed through the local node's network interface
      toPorts:
        - ports:
            - port: "8443" # vCluster API server port; Cilium L2 LB fronts this and forwards to the Control Plane cluster API on 443
              protocol: TCP
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a

With the second version of the policy, the kubectl command worked. I was able to list resources and check everything in the virtual cluster. But the next issue arose, I was unable to create any resources. I tried to create a busybox pod, and it remained in a Pending state. The issue?

Events:
  Type     Reason     Age   From        Message
  ----     ------     ----  ----        -------
  Warning  SyncError  19s   pod-syncer  Error syncing to host cluster: create object: Post "https://10.43.0.1:443/api/v1/namespaces/vcluster-team-a/pods": http2: client connection lost

The policy was updated once again to include an additional Egress rule. The following is required for the virtual cluster control plane pod to sync with the Control Plane Cluster API.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: isolate-vcluster-team-a
  namespace: vcluster-team-a
spec:
  endpointSelector: {}  #Applies this policy to all pods in the namespace
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a
    - fromCIDR:
        - <IP Subnet> # e.g. 10.10.x.x/24 Replace with your management network CIDR
      toPorts:
        - ports:
            - port: "8443"
              protocol: TCP
    - fromEntities:
        - host # Matches traffic routed through the local node's network interface
      toPorts:
        - ports:
            - port: "8443" # vCluster API server port; Cilium L2 LB fronts this and forwards to the Control Plane cluster API on 443
              protocol: TCP
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a
    - toEntities:
        - kube-apiserver
        - host

With the latest update, we can create workloads on the virtual clusters and isolate them. DNS egress to the virtual cluster's CoreDNS pod is covered by the same-namespace egress rule toEndpoints.matchLabels.io.kubernetes.pod.namespace, as CoreDNS is synced into the vcluster-team-a namespace. If your setup uses an external DNS resolver outside this namespace, add an explicit egress rule targeting that resolver.

ingress:
  - fromEndpoints:
      - matchLabels:
          io.kubernetes.pod.namespace: vcluster-platform # vCluster Platform namespace
    toPorts:
      - ports:
          - port: "8443" #vCluster API server
            protocol: TCP
egress:
  - toEndpoints:
      - matchLabels:
          io.kubernetes.pod.namespace: vcluster-platform # vCluster Platform namespace
    toPorts:
      - ports:
          - port: "10443" # vCluster Platform API port
            protocol: TCP

Validation​

$ export KUBECONFIG=vcluster-team-b.yaml

$ kubectl get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
test-pod   1/1     Running   0          11d   10.42.1.48   el07-worker1   <none>           <none>

$ kubectl exec -it test-pod -- ping -c3 10.42.0.125
PING 10.42.0.125 (10.42.0.125): 56 data bytes

--- 10.42.0.125 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1

DNS in Action​

Each virtual cluster deploys a dedicated CoreDNS instance that handles DNS queries independently from other tenants.

$ export KUBECONFIG=vcluster-team-a.yaml

$ kubectl get pods,svc -n kube-system
NAME                           READY   STATUS    RESTARTS      AGE
pod/coredns-754d567864-f9kgj   1/1     Running   1 (14d ago)   14d

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.43.196.115   <none>        53/UDP,53/TCP,9153/TCP   14d

Let's test and see how the requests are resolved from within the virtual clusters.

$ export KUBECONFIG=vcluster-team-a.yaml

$ kubectl exec -it dns-test -- bash
# nslookup kubernetes.default
;; Got recursion not available from 10.43.196.115
;; Got recursion not available from 10.43.196.115
Server:		10.43.196.115
Address:	10.43.196.115#53

Name:	kubernetes.default.svc.cluster.local
Address: 10.43.243.114
;; Got recursion not available from 10.43.196.115

# nslookup nginx.default.svc.cluster.local
;; Got recursion not available from 10.43.196.115
;; Got recursion not available from 10.43.196.115
;; Got recursion not available from 10.43.196.115
;; Got recursion not available from 10.43.196.115
Server:		10.43.196.115
Address:	10.43.196.115#53

Name:	nginx.default.svc.cluster.local
Address: 10.43.141.183
;; Got recursion not available from 10.43.196.115

Packet Capture​

$ export KUBECONFIG=vcluster-team-a.yaml

$ kubectl exec -it dns-test -- bash
dns-test:~# tcpdump -i any -n port 1053
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
06:03:24.653097 eth0  Out IP 10.42.0.72.37346 > 10.42.0.48.1053: UDP, length 57
06:03:24.653292 eth0  In  IP 10.42.0.48.1053 > 10.42.0.72.37346: UDP, length 150
06:03:24.654800 eth0  Out IP 10.42.0.72.53187 > 10.42.0.48.1053: UDP, length 49
06:03:24.654933 eth0  In  IP 10.42.0.48.1053 > 10.42.0.72.53187: UDP, length 96
06:03:24.656420 eth0  Out IP 10.42.0.72.36073 > 10.42.0.48.1053: UDP, length 49
06:03:24.656545 eth0  In  IP 10.42.0.48.1053 > 10.42.0.72.36073: UDP, length 142

$ kubectl exec -it ds/cilium -n kube-system -- bash
/home/cilium# hubble observe --to-port 1053 --namespace vcluster-team-a -f
May 26 06:09:26.930: vcluster-team-a/dns-test-x-default-x-vcluster-team-a (ID:10933) <> vcluster-team-a/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a:1053 (ID:1616) post-xlate-fwd TRANSLATED (UDP)
May 26 06:09:26.930: vcluster-team-a/dns-test-x-default-x-vcluster-team-a:34123 (ID:10933) -> vcluster-team-a/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a:1053 (ID:1616) policy-verdict:L3-Only EGRESS ALLOWED (UDP)
May 26 06:09:26.930: vcluster-team-a/dns-test-x-default-x-vcluster-team-a:34123 (ID:10933) -> vcluster-team-a/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a:1053 (ID:1616) policy-verdict:L3-Only INGRESS ALLOWED (UDP)
May 26 06:09:26.930: vcluster-team-a/dns-test-x-default-x-vcluster-team-a:34123 (ID:10933) -> vcluster-team-a/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a:1053 (ID:1616) to-endpoint FORWARDED (UDP)

Why Dedicated CoreDNS Matters​

Each tenant cluster runs its own CoreDNS instance. DNS resolution works entirely within the boundaries of the tenant cluster. Pods in the tenant cluster can only resolve service names in their virtual cluster. They use standard Kubernetes DNS names. For example, my-service.default.svc.cluster.local. Different teams can deploy services with identical names without conflicts. When Team A creates redis.default and Team B creates redis.default, each vCluster's CoreDNS resolves the name within its own isolated environment. No naming collisions, no prefixes, no Helm chart updates.

Conclusion​

Using a CNI capable of handling modern workloads matters! In today's post, we explored how the Cilium CNI and Cilium Hubble help teams to control, isolate, and observe multitenant environments while giving them the flexibility to work on what matters the most, coding. In the next post, we will demonstrate how to use Cilium and Gateway API as a shared resource so that each tenant cluster can create their own Gateway API resources for its workloads. Stay tuned!

Resources​

• vCluster Networking
• Cilium Network Policies

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​