In part 1 of the series, we explored how Sveltos acts as the main brain for our deployments to a fleet of clusters. However, this is the case when we start with Continuous Deployments (CD) or when we are willing to perform a full migration to a new architecture. The next two posts are dedicated to the collaboration between Flux and Sveltos. Flux remains the core way of deploying applications using the Flux Customer Resource Definitions (CRDs), while Sveltos enters the play when we talk about scalability, automation, and dynamic instantiation of deployments.

Motivation​

Most engineers working in Platform Engineering already have a GitOps Controller like ArgoCD or Flux in place for Kubernetes deployments. It works, and it works well, until it does not. As teams grow and deployments spread across ten or more clusters, multiple environments, and different Hyperscalers, a once clean GitOps repository quickly ends up with duplicated Helm Releases, environment-specific overrides, and manual interventions nobody wants to own.

Having only a GitOps Controller in place is not enough when it comes to complex workloads, scalability, and management of deployments and add-ons across different environments. Teams end up juggling multiple tools just to add flexibility to an existing setup. Can we help teams work smaller, simpler, and not harder when it comes to continuous deployments? The answer is yes; follow along to explore the Sveltos magic! 🪄

Scenario​

In today's post, we will showcase a flexible way of using an existing Flux configuration, and how by adding Sveltos into the mix, we provide out-of-the-box capabilities like templating, event-driven workloads, and many more. We will work with an existing Flux deployment, and Sveltos will automate the dynamic creation of Flux Helm Releases based on a cluster type or identity.

Lab Setup​

+-------------------------------+---------------------+
|          Deployment           |       Version       |
+-------------------------------+---------------------+
|             RKE2              |   v1.35.3+rke2r3    |
|           Sveltos             |       v1.8.0        |
|          Flux2 Helm           |       v2.18.3       |
|      Flux Operator Helm       |       v0.40.0       |
+-------------------------------+---------------------+

GitHub Resources​

The YAML outputs are not complete. Have a look at the GitHub repository.

Prerequisites​

1. A Kubernetes cluster acting as the management cluster
2. At least one managed cluster
3. Familiarity with Kubernetes manifest files
4. Familiarity with Flux

Diagram​

The diagram looks very similar to the one we saw in part 1 of the series. The difference here is that we already have a Flux deployment and project outline. We will install Sveltos using the native deployment options and let Sveltos listen for Events or specific resources and automate the deployment of Flux Helm Releases. Let's dive into the details.

Sveltos Installation​

The installation of Sveltos using Flux could be similar to the below. Feel free to use your preferred way of installing Sveltos to the management cluster.

---
apiVersion: v1
kind: Namespace
metadata:
  name: projectsveltos
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: projectsveltos
  namespace: flux-system
spec:
  interval: 24h
  url: https://projectsveltos.github.io/helm-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: projectsveltos
  namespace: flux-system
spec:
  interval: 30m
  targetNamespace: projectsveltos
  storageNamespace: projectsveltos
  chart:
    spec:
      chart: projectsveltos
      version: ">=1.6.1"
      sourceRef:
        kind: HelmRepository
        name: projectsveltos
        namespace: flux-system
      interval: 12h
  install:
    crds: Create
    createNamespace: true
    timeout: 10m
    strategy:
      name: RetryOnFailure
  upgrade:
    crds: CreateReplace
    timeout: 10m
    cleanupOnFail: true
    strategy:
      name: RetryOnFailure

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - sveltos-helmrelease.yaml

The above yaml definition will install Sveltos to the management cluster in the projectsveltos nasmespace. The namespace is not negotiable. Sveltos has to be installed in this namespace.

Label Management Cluster​

To control resources in the management cluster with Sveltos, we will simply add the label type: mgmt to the sveltoscluster named mgmt in the mgmt namespace. The registration is done by Sveltos during installation. No manual intervention is needed. When we refer to the management cluster, is the cluster where Flux and Sveltos are installed.

$ kubectl label sveltoscluster mgmt -n mgmt type=mgmt

Automate Flux Helm Releases​

How does it work?​

Sveltos works very well with the concept of labeling. We can label clusters with a dedicated key: value pair and use this information to dynamically create Flux Helm Releases based on our needs. In this example, everytime a cluster with the label cert-manager: required appears, we trigger an action and dynamically pre-instantiate and deploy a Flux Helm Release using the information located in the management cluster. To achieve our goal, we use the Sveltos Event Framework.

Define an EventSource​

A Sveltos EventSource is a simple way of instructing Sveltos to look for an event or a specific resource. This can be an Event within a cluster or an Event outside a cluster like NATs. In our case, we want to detect clusters with the label set to cert-manager: required.

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventSource
metadata:
  name: detect-cluster-requiring-cert-manager
spec:
  collectResources: true
  resourceSelectors:
  - group: "lib.projectsveltos.io"
    version: "v1beta1"
    kind: "SveltosCluster"
    labelFilters:
    - key: cert-manager
      operation: Equal
      value: required

Define an EventTrigger​

Once an EventSource is detected, an action or multiple actions can be triggered. In this example, once an event is detected, we will deploy a ConfigMap to the management cluster which includes the Flux Helm Releases details expressed as a Sveltos template.

apiVersion: lib.projectsveltos.io/v1beta1
kind: EventTrigger
metadata:
  name: deploy-cert-manager
spec:
  sourceClusterSelector:
    matchLabels:
      type: mgmt
  destinationClusterSelector:
    matchLabels:
      type: mgmt
  eventSourceName: detect-cluster-requiring-cert-manager
  oneForEvent: true
  policyRefs:
  - name: cert-manager-helmrelease
    namespace: default
    kind: ConfigMap

Automate Flux Helm Releases​

Now that we have our EventSource and EventTrigger in place, we need to define what actually gets deployed when an event is detected. We do this using a ConfigMap that holds the Flux Helm Release definition expressed as a Sveltos template.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cert-manager-helmrelease
  namespace: default
  annotations:
    projectsveltos.io/instantiate: ok
data:
  cert-manager.yaml: |
    apiVersion: helm.toolkit.fluxcd.io/v2
    kind: HelmRelease
    metadata:
      name: cert-manager-{{ .Resource.metadata.name  }}
      namespace: {{ .Resource.metadata.namespace }}
    spec:
      interval: 15m
      kubeConfig:
        secretRef:
          name: {{ .Resource.metadata.name }}-sveltos-kubeconfig
          key: kubeconfig
      chart:
        spec:
          chart: cert-manager
          version: "v1.16.x"
          sourceRef:
            kind: HelmRepository
            name: jetstack
            namespace: flux-system
          interval: 15m
      install:
        createNamespace: true
        timeout: 10m
        remediation:
          retries: 3
      upgrade:
        timeout: 10m
        cleanupOnFail: true
        remediation:
          retries: 3
          strategy: rollback
      values:
        crds:
          enabled: true
          keep: true

The annotation projectsveltos.io/instantiate: ok is what converts a plain ConfigMap into a Sveltos template. Sveltos will pull information directly from the management cluster and dynamically pre-instantiate and deploy the resource using the detected cluster's metadata. Notice how {{ .Resource.metadata.name }} and {{ .Resource.metadata.namespace }} are automatically resolved per cluster, one template, many clusters. The ConfigMap can be further templatised based on different use cases. Both Lua and CEL languages are supported for Sveltos templating.

Result​

Every time a cluster with the label cert-manager: required appears, Sveltos detects the event, resolves the template, and dynamically creates a dedicated Flux Helm Release using that cluster's information. No manual intervention, no duplicated YAML. One template, one label, done.

Conclusion​

In this post, we explored how Sveltos and Flux can work together without replacing one another. Flux remains the backbone of your GitOps workflow, while Sveltos adds the intelligence layer on top, handling scalability, dynamic templating, and event-driven automation with minimal overhead. By combining the EventSource, EventTrigger, and a templated ConfigMap, we reduced what would otherwise be repetitive, manually maintained Flux Helm Release definitions into a single, reusable template. The moment a cluster is labeled cert-manager: required, Sveltos takes over and does the heavy lifting automatically.

The key takeaway: you do not need to abandon your existing Flux setup to benefit from Sveltos. You layer it in, and your platform becomes smarter, not more complex.

What's Next?​

In the next post, we will work with a simple hub-spoke Flux deployment as our basis, and include Sveltos installation alongside the Event Framework approach to automate Flux Helm Releases. Stay tuned!

Resources​

• Flux Operator Documentation
• Sveltos Quick Start
• Sveltos Event Framework

✉️ Contact​

We are here to help! Whether you have questions, or issues or need assistance, our Slack channel is the perfect place for you. Click here to join us.

👏 Support this project​

Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

Series Navigation​

In parts 1 and 2 of the series, we explored the different node assignment strategies, Cilium L2 Announcements, and the vCluster Platform. Today, we take a step further. We will look at how Cilium CNI running on the underlying Control Plane cluster directly enhances networking for virtual clusters. We will cover traffic isolation, policy enforcement, DNS resolution, and traffic observability.

Source

Introduction​

The promise of vCluster is to create separate and isolated control planes for virtual clusters running on top of an underlying Control Plane cluster. The different components are placed in a separate namespace for logical isolation. However, this does not ensure that the running workloads are isolated. In a Kubernetes cluster, every pod can communicate with any other pod by default. What does that really mean? Yes, the end-user will not know they are on a shared cluster, but each of the workloads running could reach workloads on any other virtual cluster.

Lab Setup​

+-------------------------+--------------+------------------+
|        Resources        |     Type     |     Version      |
+-------------------------+--------------+------------------+
|  Control Plane Cluster  |     RKE2     | v1.34.3+rke2r1   |
|     vcluster-team-a     |     K8s      |     v1.36.0      |
|     vcluster-team-b     |     K8s      |     v1.36.0      |
+-------------------------+--------------+------------------+

GitHub Resources​

The YAML outputs are not complete. Have a look at the GitHub repository.

Prerequisites​

Go through parts 1 and 2 of the series to gain a better understanding of the concept and what we aim to achieve.

vCluster Network Model​

Before we jump into the examples and command outputs, it helps to understand the layered networking model. Two layers exist when we create virtual clusters.

+-----------------------------------------------+
|           Virtual Cluster (vcluster-team-a)    |
|  Pod A  <-->  Pod B   (virtual cluster DNS)    |
+------------------+----------------------------+
                   | Synced to Control Plane namespace
+------------------v----------------------------+
|     Control Plane Cluster (RKE2 + Cilium)     |
|  pod/app-x-ns-x-vcluster-team-a               |
|  pod/app-y-ns-x-vcluster-team-a               |
|  Cilium enforces eBPF-based policy.           |
+-----------------------------------------------+

When we deploy a pod inside vcluster-team-a, the vCluster syncs the pod down to the Control Plane cluster's namespace, vcluster-team-a. The pod gets an IP as it would normally do. The virtual cluster's internal DNS and service abstraction sit on top, but actual packet forwarding is handled by Cilium.

vCluster provides control plane isolation. Each virtual cluster has its own API server, scheduler, and controller manager. Tenants cannot see each other's Kubernetes resources through the API. However, vCluster does not provide network-level isolation between tenant namespaces by default.

To better understand this concept, we deployed an NGINX application in the default namespace. This was done for the vcluster-team-a virtual cluster during its creation. Below are the two separate views. One from the Control Plane cluster and one from the vcluster-team-a virtual cluster.

$ export KUBECONFIG=control-plane-cluster.yaml

$ kubectl get pods -n vcluster-team-a -o wide
NAME                                                       READY   STATUS    RESTARTS      AGE   IP            NODE   NOMINATED NODE   READINESS GATES
coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a   1/1     Running   1 (12d ago)   13d   10.42.0.48    el07   <none>           <none>
nginx-6797d5487-27gcf-x-default-x-vcluster-team-a          1/1     Running   0             11d   10.42.0.125   el07   <none>           <none>
vcluster-team-a-0                                          1/1     Running   0             11d   10.42.0.249   el07   <none>           <none>

$ export KUBECONFIG=vcluster-team-a.yaml

$ kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
nginx-6797d5487-27gcf   1/1     Running   0          11d   10.42.0.125   el07   <none>           <none>

We already have a test pod in the vcluster-team-b virtual cluster, and we will ping the NGINX pod using the assigned IP address 10.42.0.125. As there is no network isolation, the ping would be successful. For network-related tests, feel free to use the nicolaka/netshoot Docker image.

$ export KUBECONFIG=vcluster-team-b.yaml

$ kubectl get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
test-pod   1/1     Running   0          11d   10.42.1.48   el07-worker1   <none>           <none>

$ kubectl exec -it test-pod -- ping -c3 10.42.0.125
PING 10.42.0.125 (10.42.0.125) 56(84) bytes of data.
64 bytes from 10.42.0.125: icmp_seq=1 ttl=63 time=0.165 ms
64 bytes from 10.42.0.125: icmp_seq=2 ttl=63 time=0.212 ms

From the test-pod, we can also cURL the NGINX service IP address. We will get a response as expected.

Observations​

• Pods from vcluster-team-a are synced into the vcluster-team-a namespace on the Control Plane cluster
• Pods from vcluster-team-b are synced into the vcluster-team-b namespace on the Control Plane cluster
• These namespaces sit on the same flat network managed by Cilium
• If someone knows the pod IP or service IP of a workload in vcluster-team-b, a pod in vcluster-team-a can reach it by default

For that reason, a modern CNI with capabilities and features made for complex and modern workloads is a must-have for every Kubernetes environment. Thus, Cilium is the preferred CNI for multi-tenant environments.

Cilium Hubble Observability​

To gain the most out of our Cilium installation, we can enable Hubble, Hubble UI, and Hubble Relay (optional) to have a visual representation of the traffic. We can expose the Hubble UI service as a LoadBalancer or NodePort, or simply port-forward it. Choose the option that suits your setup.

Helm Chart Values​

hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true

$ helm upgrade rke2-cilium rke2-charts/rke2-cilium --version 1.18.300 --namespace kube-system -f values_control_plane.yaml

Hubble UI View​

Once Hubble UI is accessible, we can choose the vcluster-team-a or the vcluster-team-b namespace view and explore the traffic flow. Keep in mind that both virtual clusters are registered with the vCluster Platform, thus we see traffic to TCP port 10443.

Traffic View vcluster-team-a​

Traffic View vcluster-team-b​

Network Isolation​

Observation at the Control Plane cluster is enabled, and we are ready to continue with the network isolation. As we already saw, there is no isolation between the different virtual clusters running on a Kubernetes cluster. Because virtual clusters are assigned to different teams, we want traffic to be allowed on the namespace they belong to, but blocked on other namespaces. We need to set up strong isolation and ensure different workloads are isolated. To achieve our goal, we have two options: either use the vCluster NetworkPolicy option provided during deployment and save time using the defaults, or use the Cilium advanced networking capabilities.

Network Isolation: vCluster Helm Values​

When we create virtual clusters, we can define network isolation by using the Helm values and working with the policies.networkPolicy configuration. By default, this will allow traffic between pods within a tenant cluster, block traffic from other namespaces, and permit DNS and API server communication. However, with this approach, we use the NetworkPolicy resource and not Cilium's full capabilities. For more details, take a look at the official documentation on network policies.

Network Isolation: Cilium​

CiliumNetworkPolicy enhances Kubernetes security by using eBPF to filter traffic without the performance degradation of traditional iptables. It goes beyond simple IP-based rules to enforce identity-based security, while enabling deep Layer 7 filtering for specific HTTP paths, methods, and FQDNs.

Enforce Tenant Network Isolation​

Crafting a CiliumNetworkPolicy for the underlying virtual clusters was more complex than anticipated. It is not only about the communication between the pods within the same namespace, but also the communication between the virtual cluster and the API server, kubectl execution from machines on the local network, DNS resolution, etc. I started with the simplest possible version.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: isolate-vcluster-team-a
  namespace: vcluster-team-a
spec:
  endpointSelector: {}  #Applies this policy to all pods in the namespace
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a

Applied the manifest to the Control Plane cluster, and guess what, kubectl commands did not work from another machine in the same network. The next step was to think about Ingress traffic. The CiliumNetworkPolicy was modified. This time, we scope external management access using the fromCIDR and the fromEntities host for node local traffic.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: isolate-vcluster-team-a
  namespace: vcluster-team-a
spec:
  endpointSelector: {}  #Applies this policy to all pods in the namespace
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a
    - fromCIDR:
        - <IP Subnet> # e.g. 10.x.x.x/24 Replace with your management network CIDR
      toPorts:
        - ports:
            - port: "8443"
              protocol: TCP
    - fromEntities:
        - host # Matches traffic routed through the local node's network interface
      toPorts:
        - ports:
            - port: "8443" # vCluster API server port; Cilium L2 LB fronts this and forwards to the Control Plane cluster API on 443
              protocol: TCP
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a

With the second version of the policy, the kubectl command worked. I was able to list resources and check everything in the virtual cluster. But the next issue arose, I was unable to create any resources. I tried to create a busybox pod, and it remained in a Pending state. The issue?

Events:
  Type     Reason     Age   From        Message
  ----     ------     ----  ----        -------
  Warning  SyncError  19s   pod-syncer  Error syncing to host cluster: create object: Post "https://10.43.0.1:443/api/v1/namespaces/vcluster-team-a/pods": http2: client connection lost

The policy was updated once again to include an additional Egress rule. The following is required for the virtual cluster control plane pod to sync with the Control Plane Cluster API.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: isolate-vcluster-team-a
  namespace: vcluster-team-a
spec:
  endpointSelector: {}  #Applies this policy to all pods in the namespace
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a
    - fromCIDR:
        - <IP Subnet> # e.g. 10.10.x.x/24 Replace with your management network CIDR
      toPorts:
        - ports:
            - port: "8443"
              protocol: TCP
    - fromEntities:
        - host # Matches traffic routed through the local node's network interface
      toPorts:
        - ports:
            - port: "8443" # vCluster API server port; Cilium L2 LB fronts this and forwards to the Control Plane cluster API on 443
              protocol: TCP
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: vcluster-team-a
    - toEntities:
        - kube-apiserver
        - host

With the latest update, we can create workloads on the virtual clusters and isolate them. DNS egress to the virtual cluster's CoreDNS pod is covered by the same-namespace egress rule toEndpoints.matchLabels.io.kubernetes.pod.namespace, as CoreDNS is synced into the vcluster-team-a namespace. If your setup uses an external DNS resolver outside this namespace, add an explicit egress rule targeting that resolver.

ingress:
  - fromEndpoints:
      - matchLabels:
          io.kubernetes.pod.namespace: vcluster-platform # vCluster Platform namespace
    toPorts:
      - ports:
          - port: "8443" #vCluster API server
            protocol: TCP
egress:
  - toEndpoints:
      - matchLabels:
          io.kubernetes.pod.namespace: vcluster-platform # vCluster Platform namespace
    toPorts:
      - ports:
          - port: "10443" # vCluster Platform API port
            protocol: TCP

Validation​

$ export KUBECONFIG=vcluster-team-b.yaml

$ kubectl get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
test-pod   1/1     Running   0          11d   10.42.1.48   el07-worker1   <none>           <none>

$ kubectl exec -it test-pod -- ping -c3 10.42.0.125
PING 10.42.0.125 (10.42.0.125): 56 data bytes

--- 10.42.0.125 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1

DNS in Action​

Each virtual cluster deploys a dedicated CoreDNS instance that handles DNS queries independently from other tenants.

$ export KUBECONFIG=vcluster-team-a.yaml

$ kubectl get pods,svc -n kube-system
NAME                           READY   STATUS    RESTARTS      AGE
pod/coredns-754d567864-f9kgj   1/1     Running   1 (14d ago)   14d

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.43.196.115   <none>        53/UDP,53/TCP,9153/TCP   14d

Let's test and see how the requests are resolved from within the virtual clusters.

$ export KUBECONFIG=vcluster-team-a.yaml

$ kubectl exec -it dns-test -- bash
# nslookup kubernetes.default
;; Got recursion not available from 10.43.196.115
;; Got recursion not available from 10.43.196.115
Server:		10.43.196.115
Address:	10.43.196.115#53

Name:	kubernetes.default.svc.cluster.local
Address: 10.43.243.114
;; Got recursion not available from 10.43.196.115

# nslookup nginx.default.svc.cluster.local
;; Got recursion not available from 10.43.196.115
;; Got recursion not available from 10.43.196.115
;; Got recursion not available from 10.43.196.115
;; Got recursion not available from 10.43.196.115
Server:		10.43.196.115
Address:	10.43.196.115#53

Name:	nginx.default.svc.cluster.local
Address: 10.43.141.183
;; Got recursion not available from 10.43.196.115

Packet Capture​

$ export KUBECONFIG=vcluster-team-a.yaml

$ kubectl exec -it dns-test -- bash
dns-test:~# tcpdump -i any -n port 1053
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
06:03:24.653097 eth0  Out IP 10.42.0.72.37346 > 10.42.0.48.1053: UDP, length 57
06:03:24.653292 eth0  In  IP 10.42.0.48.1053 > 10.42.0.72.37346: UDP, length 150
06:03:24.654800 eth0  Out IP 10.42.0.72.53187 > 10.42.0.48.1053: UDP, length 49
06:03:24.654933 eth0  In  IP 10.42.0.48.1053 > 10.42.0.72.53187: UDP, length 96
06:03:24.656420 eth0  Out IP 10.42.0.72.36073 > 10.42.0.48.1053: UDP, length 49
06:03:24.656545 eth0  In  IP 10.42.0.48.1053 > 10.42.0.72.36073: UDP, length 142

$ kubectl exec -it ds/cilium -n kube-system -- bash
/home/cilium# hubble observe --to-port 1053 --namespace vcluster-team-a -f
May 26 06:09:26.930: vcluster-team-a/dns-test-x-default-x-vcluster-team-a (ID:10933) <> vcluster-team-a/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a:1053 (ID:1616) post-xlate-fwd TRANSLATED (UDP)
May 26 06:09:26.930: vcluster-team-a/dns-test-x-default-x-vcluster-team-a:34123 (ID:10933) -> vcluster-team-a/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a:1053 (ID:1616) policy-verdict:L3-Only EGRESS ALLOWED (UDP)
May 26 06:09:26.930: vcluster-team-a/dns-test-x-default-x-vcluster-team-a:34123 (ID:10933) -> vcluster-team-a/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a:1053 (ID:1616) policy-verdict:L3-Only INGRESS ALLOWED (UDP)
May 26 06:09:26.930: vcluster-team-a/dns-test-x-default-x-vcluster-team-a:34123 (ID:10933) -> vcluster-team-a/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a:1053 (ID:1616) to-endpoint FORWARDED (UDP)

Why Dedicated CoreDNS Matters​

Each tenant cluster runs its own CoreDNS instance. DNS resolution works entirely within the boundaries of the tenant cluster. Pods in the tenant cluster can only resolve service names in their virtual cluster. They use standard Kubernetes DNS names. For example, my-service.default.svc.cluster.local. Different teams can deploy services with identical names without conflicts. When Team A creates redis.default and Team B creates redis.default, each vCluster's CoreDNS resolves the name within its own isolated environment. No naming collisions, no prefixes, no Helm chart updates.

Conclusion​

Using a CNI capable of handling modern workloads matters! In today's post, we explored how the Cilium CNI and Cilium Hubble help teams to control, isolate, and observe multitenant environments while giving them the flexibility to work on what matters the most, coding. In the next post, we will demonstrate how to use Cilium and Gateway API as a shared resource so that each tenant cluster can create their own Gateway API resources for its workloads. Stay tuned!

Resources​

• vCluster Networking
• Cilium Network Policies

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​

After many discussions at the KubeCon Europe in Amsterdam, I decided to start a new series covering the most commonly seen scenarios and approaches on how Sveltos and different GitOps Controllers can work together. Sveltos is not a replacement for your GitOps Controller. It is a tool to enhance and extend existing capabilities. When we talk about GitOps Controllers, we primarily refer to either ArgoCD or Flux. In the first part of the series, we will demonstrate how Sveltos fits into the Platform engineering space and, more specifically, in the Continuous Deployment (CD) part. We will provide a commonly seen scenario and explore how Sveltos can control all the deployments.

Motivation​

I had the chance to meet Gianluca Mardenete (creator of Sveltos and K8Scleaner) during an internal call while working on a project. He introduced me to Sveltos almost two and a half years ago. I was impressed by the functionality and features that came out of the box, one tool to rule them all, and decided to give it a spin. Since then, I can say it not only gave me back time to work on other issues, but also made deployments way easier and scalable for us! Follow along to explore Sveltos together!

Scenario​

Let’s start with a quick introduction to Sveltos. It is a Kubernetes add-on controller. It makes deploying and managing Kubernetes add-ons and applications easier using a label approach. We can use it across multiple clusters, whether on-premises, in the cloud, or in multitenant setups. Instead of configuring each cluster individually, Sveltos inverts the model: we define intent through ClusterProfiles,Profiles and labels, and clusters become consumers of that intent. Sveltos integrates very well with existing GitOps Controllers and extends its capabilities using out of the box features like advanced templating, Event Framework, and native integration with Cluster API (CAPI). To learn more about the Sveltos features, take a look at the official documentation. As mentioned in the beginning, the goal of the new series is to demonstrate the different scenarios when it comes to CD. How can someone use Sveltos capabilities when starting with deployments, and how can Sveltos fit and collaborate with other Controllers to help teams scale their operations?

In this post, we will focus on the scenario where Sveltos is the “brain” of our deployments. We will showcase how Sveltos can install ArgoCD and Flux to a central Kubernetes management cluster and how the same approach can be used to deploy a GitOps controller to a managed cluster that requires a GitOps Controller with a custom definition. What that means is that we will bring any Sveltos manifests stored in a git repo using an available GitOps Controller. From there, Sveltos takes over and works with the deployments based on the labelling concept.

By the end of this post, you will have a working setup where Sveltos installs and orchestrates both ArgoCD and Flux on a management cluster, and deploys the appropriate GitOps controller to managed clusters based on labels.

Lab Setup​

+---------------------------+------------------+
|        Deployment         |     Version      |
+---------------------------+------------------+
|           RKE2            | v1.35.3+rke2r3   |
|         Sveltos           |     v1.8.0       |
|    ArgoCD Helm Chart      |     v9.4.17      |
|    Flux2 Helm Chart       |     v2.18.3      |
|    Flux Operator Helm     |     v0.40.0      |
+---------------------------+------------------+

GitHub Resources​

The YAML outputs are not complete. Have a look at the GitHub repository.

Prerequisites​

1. A Kubernetes cluster acting as the management cluster
2. At least two managed clusters
3. Familiarity with Kubernetes manifest files
4. Familiarity with ArgoCD, Flux and GitOps practices

Diagram​

Taking a look at the diagram, we follow a classic GitOps approach. The DevOps or Platform engineering team will store the source of truth in a git or multiple git repositories. In this scenario, we instruct ArgoCD or Flux to synchronise with any repositories that hold the Sveltos resources like ClusterProfiles, Profiles, EventSource, and EventTrigger to a central management cluster. Then, Sveltos takes over the deployment of applications and add-ons using a label approach. Once a change is performed following a standard Merge or PR request, the GitOps controller will bring the updated code to the management cluster, and Sveltos will ensure the changes are performed to the affected managed clusters.

We start by installing the different GitOps Controllers to a Kubernetes management cluster (the same cluster where Sveltos is installed) using a ClusterProfile. Then, synchronise the required repositories, and Sveltos performs different deployments across different managed clusters following the Kubernetes labelling approach. For example, if a managed cluster has the label git-controller: argo, Sveltos will deploy ArgoCD to that cluster. If there is another managed cluster with the label set to git-controller: flux, Sveltos will deploy Flux to that cluster. The labelling approach provides teams with extra flexibility while keeping deployments as simple as possible. The DRY and KISS frameworks are embraced by Sveltos architecture.

My Project Structure​

I like to keep things organised. When I use Sveltos as the brains of operations, I usually have the following file structure in a repository.

sveltos-resources/
├── mgmt/               # Resources for the management cluster
├── base/               # Global configurations (applied to all clusters)
│   ├── cni/
│   └── security-baseline/
├── providers/          # Provider-specific configurations (Labels: provider=aks/eks/on-prem)
│   ├── aks/
│   ├── eks/
│   └── on-prem/
├── environments/       # Env-specific configurations (Labels: env=dev/staging/prod)
│   ├── dev/
│   ├── staging/
│   └── prod/

The mgmt/ holds the Sveltos resources to be applied in the Kubernetes management cluster. The base/ directory holds the Sveltos manifests that apply to every cluster, such as the Container Network Interface (CNI) and network policies. For specialised needs, we use providers/ and environments/ folders. Instead of nesting these, we use labels to target the right Sveltos resources, allowing each cluster to automatically pick up the configurations it needs based on its specific role.

As our demonstration is way simpler, we will use the labels env: dev, env: staging, gitops: argocd, and gitops: flux to deploy specific resources to these clusters. Feel free to experiment with your own file structure based on your use cases.

Sveltos Installation​

To start with the demo, we need to install Sveltos on the management cluster. I will use the Helm installation using Mode 1. Choose your preferred installation mode by reading the installation documentation.

Helm Chart Installation​

$ export KUBECONFIG=<directory of the management kubeconfig>

$ helm repo add projectsveltos https://projectsveltos.github.io/helm-charts
$ helm repo update

$ helm install projectsveltos projectsveltos/projectsveltos -n projectsveltos --create-namespace --version=1.8.0

Label Management Cluster​

To control resources in the management cluster using Sveltos, we will simply add the label type: mgmt to the sveltoscluster named mgmt in the mgmt namespace. The registration is done by Sveltos during installation.

$ kubectl label sveltoscluster mgmt -n mgmt type=mgmt

GitOps Controller - Management Cluster​

Flux Deployment​

We will use a Sveltos ClusterProfile to install the Flux-Operator as a Helm chart and include all the required resources for the desired setup. The official OCI registry is used to pull the Flux Operator Helm chart. With Sveltos, we can deploy ConfigMap and Secret resources that contain information about the cluster. That means we can add the Flux Instance, the GitLab secret, and the GitRepository resources into a ConfigMap and instruct Sveltos to deploy it to the management cluster.

Flux Operator ClusterProfile​

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: flux
spec:
  clusterSelector:
    matchLabels:
      type: mgmt
  helmCharts:
  - repositoryURL: oci://ghcr.io/controlplaneio-fluxcd/charts
    repositoryName: flux-operator
    chartName: flux-operator
    chartVersion: 0.40.0
    releaseName: flux-operator
    releaseNamespace: flux-system
    helmChartAction: Install
  policyRefs:
  - name: flux-resources
    namespace: default
    kind: ConfigMap

  - kind: GitRepository
    name: sveltos-repo-sync # Define the Flux GitRepository resource name defined in the flux-resources ConfigMap
    namespace: flux-system
    path: ./resources/sveltos-manifests/

flux-resources ConfigMap​

apiVersion: v1
kind: ConfigMap
metadata:
  name: flux-resources
  namespace: default
data:
  flux_resources.yaml: |
    ---
    apiVersion: fluxcd.controlplane.io/v1
    kind: FluxInstance
    metadata:
      name: flux
      namespace: flux-system
      annotations:
        fluxcd.controlplane.io/reconcile: "enabled"
        fluxcd.controlplane.io/reconcileEvery: "1h"
        fluxcd.controlplane.io/reconcileTimeout: "5m"
    spec:
      distribution:
        version: "2.x"
        registry: "ghcr.io/fluxcd"
      components:
        - source-controller
        - kustomize-controller
        - helm-controller
        - notification-controller
        - image-reflector-controller
        - image-automation-controller
      cluster:
        type: kubernetes
        size: medium
        multitenant: false
        networkPolicy: false
        domain: "cluster.local"
      commonMetadata:
        labels:
          app.kubernetes.io/name: flux
      kustomize:
        patches:
          - target:
              kind: Deployment
            patch: |
              - op: replace
                path: /spec/template/spec/nodeSelector
                value:
                  kubernetes.io/os: linux
              - op: add
                path: /spec/template/spec/tolerations
                value:
                  - key: "CriticalAddonsOnly"
                    operator: "Exists"
    ---
    apiVersion: source.toolkit.fluxcd.io/v1
    kind: GitRepository
    metadata:
      name: sveltos-repo-sync
      namespace: flux-system
    spec:
      interval: 30s
      ref:
        branch: main
      timeout: 60s
      url: https://<your domain>/<group name>/<repository name>.git

apiVersion: v1
kind: Secret
data:
  password: <BASE64 encoded string>
  username: <BASE64 encoded string>
metadata:
  name: git-creds
  namespace: flux-system
type: Opaque
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: a-repo-sync
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: git-creds
  timeout: 60s
  url: https://<your domain>/<group name>/<repository name>.git

ArgoCD Installation - Management Cluster​

ArgoCD ClusterProfile​

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: argocd
spec:
  clusterSelector:
    matchLabels:
      type: mgmt
  helmCharts:
  - repositoryURL: https://argoproj.github.io/argo-helm
    repositoryName: argo
    chartName: argo-cd
    chartVersion: 9.4.17
    releaseName: argocd
    releaseNamespace: argocd
    helmChartAction: Install
  policyRefs:
  - name: argo-resources
    namespace: default
    kind: ConfigMap

argo-resources ConfigMap​

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-resources
  namespace: default
data:
  argo_resources.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: sveltos-manifests
      namespace: argocd
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: default
      source:
        repoURL: "https://<your domain>/<group name>/<repository name>.git"
        targetRevision: HEAD
        path: resources/sveltos-manifests/
      destination:
        server: https://kubernetes.default.svc
        namespace: default
      syncPolicy:
        automated:
          selfHeal: true
          prune: true
        retry:
          limit: 5
          backoff:
            duration: 5s
            maxDuration: 3m0s
            factor: 2
      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: sveltos-repo-sync
        namespace: argocd
        labels:
          argocd.argoproj.io/secret-type: repository
      stringData:
        url: "https://<your domain>/<group name>/<repository name>.git"

Once ArgoCD is running, we create an Application resource pointing at the Sveltos resources directory in a GitHub repo. We can do this through the ArgoCD UI, via kubectl apply, or by including the Application manifest in a ConfigMap referenced by the same ClusterProfile using policyRefs. As we use Sveltos to handle the installation of ArgoCD, we take care of the deployment using a Sveltos resource.

Register Managed Clusters​

The Sveltos "magic" 🪄✨ happens when we want to deploy add-ons and applications to a fleet of clusters. First, we need to register the clusters with Sveltos using either the sveltosctl or the programmatic approach. Make your choice. During the registration process, we assign specific labels to the managed clusters. For example, define the environment, whether they need a special configuration, etc.

Flux vs ArgoCD ClusterProfile​

Base ClusterProfile​

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: common-staging
spec:
  clusterSelector:
    matchExpressions:
    - {key: env, operator: In, values: [dev, staging]}
  helmCharts:
  - chartName: cilium/cilium
    chartVersion: 1.18.5
    helmChartAction: Install
    releaseName: cilium
    releaseNamespace: kube-system
    repositoryName: cilium
    repositoryURL: https://helm.cilium.io/
    values: |
      hubble:
        enabled: true
        peerService:
          clusterDomain: cluster.local
        relay:
          enabled: true
        tls:
          auto:
            certValidityDuration: 1095
            enabled: true
            method: helm
        ui:
          enabled: true
      nodePort:
        enabled: true
      debug:
        enabled: true
  - repositoryURL: https://charts.jetstack.io
    repositoryName: jetstack
    chartName: jetstack/cert-manager
    chartVersion: v1.16.3
    releaseName: cert-manager
    releaseNamespace: cert-manager
    helmChartAction: Install
    values: |
      crds:
        enabled: true

The base configuration will be applied to every managed cluster that has the label env: dev or env: staging defined. In this case, we would like to install a specific version of Cilium and cert-manager for certificate management.

Now, depending on whether the developers like to work with ArgoCD or Flux, we can create two different ClusterProfile resources and deploy the preferred GitOps Controller based on the git-controller label defined. Check out the configuration below for more details.

ArgoCD Controller​

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: argocd
spec:
  clusterSelector:
    matchLabels:
      git-controller: argo
  helmCharts:
  - repositoryURL: https://argoproj.github.io/argo-helm
    repositoryName: argo
    chartName: argo-cd
    chartVersion: 9.4.17
    releaseName: argocd
    releaseNamespace: argocd
    helmChartAction: Install
  policyRefs:
  - name: argo-resources
    namespace: default
    kind: ConfigMap

Flux GitOps Controller​

apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: flux
spec:
  clusterSelector:
    matchLabels: 
      git-controller: flux
  helmCharts:
  - repositoryURL: oci://ghcr.io/controlplaneio-fluxcd/charts
    repositoryName: flux-operator
    chartName: flux-operator
    chartVersion: 0.40.0
    releaseName: flux-operator
    releaseNamespace: flux-system
    helmChartAction: Install
  policyRefs:
  - name: flux-resources
    namespace: default
    kind: ConfigMap

Updating Deployments​

Once the initial deployment is complete, updating applications across the fleet follows the same GitOps workflow. When a ClusterProfile is modified, whether it is a Helm chart version, a change in the values field, or an update to the policyRefs, Sveltos detects the change and performs a Helm upgrade on every Sveltos-managed cluster matching the label selector.

For example, upgrading Cilium from 1.18.5 to 1.18.6 in the base ClusterProfile will trigger a Rolling Update across all clusters with the label env: dev or env: staging. Sveltos ensures the desired state is applied, and Kubernetes handles the pod rollout mechanics.

Because we use a GitOps Controller to sync ClusterProfile manifests to the management cluster, the update workflow looks like the following.

1. Update a ClusterProfile manifest in Git (e.g., change chart version or values)
2. Merge the changes via a PR or Merge Request
3. The GitOps Controller syncs the updated ClusterProfile to the management cluster
4. Sveltos detects the change and performs a Helm upgrade on all matching clusters
5. Kubernetes rolls out the updated workloads

No manual kubectl apply or cluster-by-cluster intervention is required. The entire update lifecycle is driven by a single Git commit.

Conclusion​

In today's blog, we demonstrated a simple yet powerful and extensible way of using Sveltos as our main brain for the application and add-on deployments across a fleet of clusters. Using a labelling approach and expanding to complex deployments becomes easy using features like Sveltos templating and Event Framework, which we will cover in Parts 2 and 3.

In the next two blog posts, we will work on how Sveltos integrates with an existing Flux deployment and how we can extend Flux capabilities using Sveltos out of the box advanced features. Stay tuned.

Resources​

• Flux Operator Documentation
• Sveltos Quick Start
• Sveltos Event Framework

✉️ Contact​

We are here to help! Whether you have questions, or issues or need assistance, our Slack channel is the perfect place for you. Click here to join us.

👏 Support this project​

Every contribution counts! If you enjoyed this article, check out the Projectsveltos GitHub repo. You can star 🌟 the project if you find it helpful.

The GitHub repo is a great resource for getting started with the project. It contains the code, documentation, and many more examples.

Thanks for reading!

Series Navigation​

In part 1 of the series, we explored some of the recent vCluster Helm chart changes and how we can create the simplest vCluster possible. We also explored how we can assign specific Kubernetes worker nodes to vClusters based on taints, tolerations, and labels. Today, we will walk through the process of setting up Cilium L2 Announcements to make the vCluster available via a LoadBalancer service and then deploy and use the vCluster Platform.

Source

Introduction​

While we can use a NodePort service to access the vCluster within our setup, this is a single point of failure in case the nodes, for whatever reason, go down. We want to achieve better scalability and ensure the vClusters are accessible via a stable IP address and not dependent on the Kubernetes nodes' availability. Thus, we can use the underlying Cilium functionality, enable IPAM to hand over LoadBalancer IP addresses and then use the L2 Announcements to make the endpoints reachable to the desired network subnet.

As the underlying control plane cluster is an RKE2 cluster, and Cilium is already installed, we will only need to update the Helm chart values to include the additional functionality. Once this is done, we will update the vCluster Helm chart values and expose the virtual clusters via a LoadBalancer IP address instead of a NodePort. Finally, by having the vCluster Platform deployed, we can have a single pane of glass when it comes to the management of a fleet of vClusters across different environments.

Lab Setup​

+-------------------------+--------------+------------------+
|        Resources        |     Type     |     Version      |
+-------------------------+--------------+------------------+
|  Control Plane Cluster  |     RKE2     | v1.34.3+rke2r1   |
|     vcluster-team-a     |     K8s      |     v1.36.0      |
|     vcluster-team-b     |     K8s      |     v1.36.0      |
+-------------------------+--------------+------------------+

Prerequisites​

1. A Kubernetes cluster available with at least two worker nodes
2. Helm installed
3. kubectl installed
4. Familiarity with vCluster

Cilium IPAM and L2 Announcements​

As mentioned in the beginning, we will create a pool that reflects the available IPv4 addresses to be assigned to LoadBalancer services. We will also define an interface where L2 Announcements should occur. The setup is simple. We will expand on the existing Cilium Helm chart values and enable the ones we need.

Extract Cilium Helm chart values.yaml​

$ helm get values rke2-cilium -n kube-system -o yaml > values_control_plane.yaml

Expand values_control_plane.yaml​

Based on the documentation, the values are required to meet our use case.

k8sClientRateLimit:
  burst: 40 # Important value when many services run on a Kubernetes cluster. Check out the documentation https://docs.cilium.io/en/v1.18/network/l2-announcements/#sizing-client-rate-limit
  qps: 20 # Important value when many services run on a Kubernetes cluster. Check out the documentation https://docs.cilium.io/en/v1.18/network/l2-announcements/#sizing-client-rate-limit
kubeProxyReplacement: true # Required
l2announcements:
  enabled: true

Update Cilium Helm Deployment​

$ helm upgrade rke2-cilium rke2-charts/rke2-cilium --version 1.18.300 --namespace kube-system -f values_control_plane.yaml

$ kubectl rollout restart deployment.apps/cilium-operator -n kube-system
$ kubectl rollout restart daemonset.apps/cilium  -n kube-system

Ensure the pods have been successfully restarted and that Cilium is using the updated values specified.

Create CiliumLoadBalancerIPPool​

The pool will allow us to assign IPv4 addresses to services of type LoadBalancer. The configuration needs to reflect your own setup. In my case, I have a dedicated VLAN I can use for handing over IPv4 addresses.

apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "vcluster-ipv4-pool"
spec:
  blocks:
  - start: "10.10.20.10"
    stop: "10.10.20.20"

Create CiliumL2AnnouncementPolicy​

The IPs will be announced from the network interface of a node with the interface name eth0. If the interface name in your setup is different, modify the file as needed.

apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: vcluster-l2-announcement-policy
  namespace: kube-system
spec:
  interfaces:
    - eth0
  loadBalancerIPs: true

Apply both manifests to the Control Plane Cluster.

$ kubectl apply -f vcluster_ipv4_pool.yaml,vcluster_l2_announcement.yaml

$ kubectl get CiliumL2AnnouncementPolicy,ippools
NAME                                                          AGE
ciliuml2announcementpolicy.cilium.io/vcluster-l2-announcement-policy   24h

NAME                                           DISABLED   CONFLICTING   IPS AVAILABLE   AGE
ciliumloadbalancerippool.cilium.io/vcluster-ipv4-pool   false      False         10              24h

Update vCluster Helm Values​

In the initial setup, we defined the Kubernetes API server of every vCluster to be exposed as a NodePort service. This can now change with the power that comes with Cilium as our Container Network Interface (CNI). With the new approach, we have a stable way to reach the virtual clusters. For both vClusters, update the controlPlane.service.spec.type to LoadBalancer.

controlPlane:
  # Service configuration for vCluster control plane access
  # The vcluster-dev will be accessible on a LoadBalancer IP Address and port 443
  service:
    enabled: true
    annotations: {}
    labels: {}
    spec:
      type: LoadBalancer

Update vCluster Helm Deployment​

$ helm upgrade --install vcluster-team-a vcluster \
  --repo https://charts.loft.sh \
  --namespace vcluster-team-a \
  --create-namespace \
  -f vcluster_team_a_lb.yaml

Validation​

$ export KUBECONFIG=control-plane-cluster.yaml 

$ kubectl get pods,svc -n vcluster-team-a
NAME                                                           READY   STATUS    RESTARTS      AGE
pod/coredns-754d567864-f9kgj-x-kube-system-x-vcluster-team-a   1/1     Running   1 (24h ago)   27h
pod/vcluster-team-a-0                                          1/1     Running   0             7m2s

NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns-x-kube-system-x-vcluster-team-a   ClusterIP      10.43.196.115   <none>        53/UDP,53/TCP,9153/TCP   27h
service/vcluster-team-a                            LoadBalancer   10.43.243.114   10.10.20.11   443:30445/TCP            27h
service/vcluster-team-a-headless                   ClusterIP      None            <none>        443/TCP                  27h
service/vcluster-team-a-node-el07                  ClusterIP      10.43.227.82    <none>        10250/TCP                27h

From the above output, the vcluster-team-a cluster will be available at https://10.10.20.11:443. This line should be included in the kubeconfig file on the virtual cluster. The kubeconfig of vcluster-team-a is saved as a secret named vcluster-team-a in the vcluster-team-a namespace.

$ export KUBECONFIG=vcluster-team-a.yaml 

$ kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE               KERNEL-VERSION             CONTAINER-RUNTIME
test-worker1   Ready    <none>   26h   v1.36.0   10.43.227.82  <none>        Fake Kubernetes Image  4.19.76-fakelinux (amd64)  docker://19.3.12

$ kubectl get pods -n kube-system
NAMESPACE     NAME                       READY   STATUS    RESTARTS      AGE
kube-system   coredns-754d567864-f9kgj   1/1     Running   1 (24h ago)   26h

Apply the same steps for the vcluster-team-b cluster.

vCluster Platform​

Imagine your team or your organisation handles a large number of vClusters. An easy way to manage all of them from a central location alongside providing tenant admins with an intuitive UI to perform operations, RBAC, resource quota, cluster lifecycle, etc., is the vCluster Platform.

What is the vCluster Platform?​

vCluster Platform is the management plane for your tenant cluster fleet. It provides a web UI, CLI, and API for deploying, configuring, and operating tenant clusters across one or more Control Plane Clusters. Access control, lifecycle automation, resource governance, and node management are all built in.

vCluster Platform Helm Deployment​

Helm Chart Values​

admin:
  create: true
  username: username
  password: "password"
# Loft service options
service:
  type: LoadBalancer
# Resources of the loft deployment
resources:
  requests:
    memory: 256Mi
    cpu: 200m
  limits:
    memory: 2Gi
    cpu: "2"
config:
  loftHost: https://<Accessible IP Address> # 127.0.0.1 can be set when vClusters are created on the same Control Plane Cluster where the vCluster Platform is installed 
  audit:
    enabled: true
insecureSkipVerify: true # Only for Development environments. Use your own valid TLS certificates for Production deployments.

Deployment​

$ helm upgrade --install vcluster-platform vcluster-platform \
  --repo https://charts.loft.sh/ \
  --namespace vcluster-platform \
  --create-namespace \
  --version 4.9.0 \
  --values vcluster_platform_values.yaml

Register Existing vClusters​

As long as the vCluster Platform is up and running, you should be able to reach the UI using the LoadBalancer IP address. As the vcluster-team-a and vcluster-team-b clusters have been created using a Helm chart, they are not associated with the vCluster Platform. However, we can add them under the vCluster Platform management using either the vcluster command-line utility or Helm. The recommended approach is to use the vcluster command-line utility.

1. Log in to the vCluster Platform from the UI
2. Navigate to the bottom left side, click the username of the logged-in user and click Access keys. Click the Create Access Key button to create a new key. Determine how long the key should be valid and the permissions are assigned to the key
3. On a machine with access to the vcluster CLI, perform the steps below
   1. vcluster platform login https://<vCluster Platform UI IP Address> --access-key <access-key-generated-from-UI> --insecure. In case a valid TLS certificate is used, there is no need to add the --insecure flag
   2. vcluster platform add vcluster vcluster-team-a -n vcluster-team-a --project default. The project can also be set to a different value. Default is the default project created by the deployment of the vCluster Platform
   3. vcluster platform add vcluster vcluster-team-b -n vcluster-team-b --project default

Refreshing the UI, the cluster should already be visible.

Conclusion​

In this post, we built on the foundation established in Part 1 by enabling Cilium IPAM and L2 Announcements to expose our vCluster API servers via stable LoadBalancer IP addresses, eliminating the single point of failure introduced by NodePort services. We also deployed the vCluster Platform, giving us a centralised management plane for operating our vCluster fleet across environments. In Part 3, we will take a deeper dive into the networking layer, examining how traffic flows between virtual clusters, how network policies can be enforced at both the host and virtual cluster level, and how Cilium's advanced features can further strengthen isolation and observability in a multi-tenant context. Stay tuned!

Resources​

• vCluster Documentation
• vCluster Platform
• Cilium Docs

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​

It has been a while since my last post on vCluster. After working more closely with the tool, I decided to update the series and show you what changes when it comes to a local setup.

Source

Introduction​

My first interaction with vCluster was about two years ago. I needed to create local development environments with minimal resources and low setup effort. This means easy maintenance and less troubleshooting. My goal was to use Sveltos to automate the deployment of these environments when a developer joined or left the team. To achieve my goal, and after looking around at the different open-source tools, I stumbled upon vCluster.

Fast forward a year since my last blog, and I will provide updates on what changes from a vCluster point of view. The first part will be an update of the Helm chart deployment and values. On the next one, we will integrate Cilium L2 Announcements and add the vCluster Platform to the mix for ease of management and operations. Next, we will briefly discuss the Enterprise version and some use cases I covered. Finally, we will explore Sveltos and how to use the Sveltos Event Framework. This will help us automatically set up local development environments using a GitOps approach.

Lab Setup​

+-------------------------+--------------+------------------+
|        Resources        |     Type     |     Version      |
+-------------------------+--------------+------------------+
|  Control Plane Cluster  |     RKE2     | v1.34.3+rke2r1   |
|      vcluster-dev       |     K8s      |     v1.36.0      |
|     vcluster-team-a     |     K8s      |     v1.36.0      |
|     vcluster-team-b     |     K8s      |     v1.36.0      |
+-------------------------+--------------+------------------+

Prerequisites​

1. A Kubernetes cluster available with at least two worker nodes
2. Helm installed
3. kubectl installed
4. Familiarity with vCluster

Scenario​

“Multitenancy (or multi-tenancy) refers to a single software installation that serves multiple tenants. A tenant is a user, application, or a group of users/applications that utilize the software to operate on their own data set.”

We will start with a basic vcluster-dev vCluster in the dev namespace. Then we will update the Helm chart values and create two additional vClusters, vcluster-team-a and vcluster-team-b, in their own namespaces, where they will be scheduled on different nodes based on Kubernetes taints and tolerations. As this is a local setup, the vCluster API Server is exposed as a node port. However, in a later post, we will demonstrate how to use Cilium L2 Announcements to reach the clusters with a LoadBalancer IP address.

vcluster-dev Setup​

values.yaml​

A few things have changed since the last blog post. The Helm values structure has been simplified, and configuration options are now more clearly organised. Find the vCluster Helm Chart values on GitHub. The main difference with the previous setup is that a NodePort is used to expose the API Server, while the distribution is defined as k8s and not as k3s.

CoreDNS is visible within the tenant cluster, as there might be cases for a custom DNS configuration or specific external DNS resolution domains.

For storage requirements, feel free to use your preferred open-source storage solution (Longhorn, Ceph with Rook). The simplest setup will be the local-path-provisioner, which can be installed as a Helm chart, and the setup will use the local storage on each node.

controlPlane:
  distro:
    k8s:
      enabled: true
      image:
        repository: "loft-sh/kubernetes"
        tag: "v1.36.0"
      resources:
        limits:
          cpu: 100m
          memory: 256Mi
        requests:
          cpu: 40m
          memory: 64Mi
  # Enable CoreDNS services per tenant cluster
  coredns:
    enabled: true
  statefulSet:
    resources:
      limits:
        ephemeral-storage: 2Gi
        memory: 2Gi
      requests:
        ephemeral-storage: 400Mi
        cpu: 200m
        memory: 256Mi
    highAvailability:
      replicas: 1
    security:
      podSecurityContext: {}
      containerSecurityContext:
        allowPrivilegeEscalation: false
        runAsUser: 0
        runAsGroup: 0
    persistence:
      volumeClaim:
        enabled: auto
        retentionPolicy: Retain
        size: 2Gi
        storageClass: "local-path"
        accessModes: [ "ReadWriteOnce" ]
  # Service configuration for vCluster control plane access
  # The vcluster-dev will be accessible on Node IP Address:30443
  service:
    enabled: true
    annotations: {}
    labels: {}
    httpsNodePort: 30443
    kubeletNodePort: 0
    spec:
      type: NodePort
exportKubeConfig:
  context: vcluster-dev
  insecure: false
  secret:
    name: vcluster-dev
    namespace: dev
experimental:
  deploy:
    vcluster:
      manifests: |-
        ---
        apiVersion: v1
        kind: Namespace
        metadata:
          name: nginx-app
        ---
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: nginx-dev
          namespace: nginx-app
        spec:
          selector:
            matchLabels:
              app: nginx
          replicas: 1
          template:
            metadata:
              labels:
                app: nginx
            spec:
              containers:
              - name: nginx
                image: nginx:latest
                ports:
                - containerPort: 80
        ---
        apiVersion: v1
        kind: Service
        metadata:
          name: nginx-dev
          namespace: nginx-app
        spec:
          selector:
            app: nginx
          ports:
            - protocol: TCP
              port: 80
              targetPort: 80
          type: ClusterIP

Helm Deployment​

$ helm repo add loft https://charts.loft.sh
$ helm repo update

$ export KUBECONFIG=/path/to/management/kubeconfig

$ helm upgrade --install vcluster-dev vcluster \
  --namespace dev \
  --create-namespace \
  --values /the/path/to/configuration/values.yaml \
  --repo https://charts.loft.sh \
  --repository-config=''

vCluster Validation​

$ helm list -n dev
NAME        	NAMESPACE	REVISION	UPDATED                                 	STATUS  	CHART          	APP VERSION
vcluster-dev	dev      	1       	2026-05-06 10:18:55.604901016 +0200 CEST	deployed	vcluster-0.34.0	0.34.0

$ kubectl get pods,svc,secret -n dev
NAME                                                        READY   STATUS    RESTARTS   AGE
pod/coredns-6dc6dfcb8f-zbrdx-x-kube-system-x-vcluster-dev   1/1     Running   0          2m
pod/nginx-dev-59c4c87bc6-4lrdc-x-nginx-app-x-vcluster-dev   1/1     Running   0          2m
pod/vcluster-dev-0                                          1/1     Running   0          2m21s

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/kube-dns-x-kube-system-x-vcluster-dev   ClusterIP   10.43.133.191   <none>        53/UDP,53/TCP,9153/TCP          2m
service/nginx-dev-x-nginx-app-x-vcluster-dev    ClusterIP   10.43.97.8      <none>        80/TCP                          2m
service/vcluster-dev                            NodePort    10.43.59.149    <none>        443:30443/TCP,10250:31662/TCP   2m21s
service/vcluster-dev-headless                   ClusterIP   None            <none>        443/TCP                         2m22s
service/vcluster-dev-node-el07-worker1          ClusterIP   10.43.207.196   <none>        10250/TCP                       2m

NAME                                        TYPE                 DATA   AGE
secret/sh.helm.release.v1.vcluster-dev.v1   helm.sh/release.v1   1      2m22s
secret/vc-config-vcluster-dev               Opaque               1      2m22s
secret/vc-vcluster-dev                      Opaque               5      2m
secret/vcluster-dev                         Opaque               5      2m
secret/vcluster-dev-certs                   Opaque               29     2m11s

Retrieve vcluster-dev Kubeconfig​

Once the virtual cluster is deployed, we can retrieve the kubeconfig by decoding the configuration of the secret with the name vcluster-dev.

$ kubectl get secret vcluster-dev -n dev --template={{.data.config}} | base64 -d > /the/path/to/configuration/vcluster-dev.yaml

Open the file and update the server: https://localhost:8443 section with server: https://<NODE IP>:30443. Ensure the Node IP address defined is the one the vcluster-dev-0 pod is scheduled on. Alternatively, we can define the proxy.extraSANs option to create a valid certificate for different DNS names and IPs.

Validation​

$ export KUBECONFIG=/the/path/to/configuration/vcluster-dev.yaml

$ kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
el07-worker1   Ready    <none>   134m   v1.36.0   10.43.207.196   <none>        Fake Kubernetes Image   4.19.76-fakelinux (amd64)   docker://19.3.12

$ kubectl get pods,svc -n nginx-app
NAME                             READY   STATUS    RESTARTS   AGE
pod/nginx-dev-59c4c87bc6-4lrdc   1/1     Running   0          8m13s

NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/nginx-dev   ClusterIP   10.43.97.8   <none>        80/TCP    8m12s

vcluster-team-a Setup​

values.yaml​

As mentioned, we will use the Kubernetes taints, tolerations, and Kubernetes labels to schedule vCluster to specific nodes based on the team we would like to onboard. The setup below is a simple example of how the vCluster configuration looks alongside the work done on the underlying control plane cluster and nodes.

controlPlane:
  distro:
    k8s:
      enabled: true
      image:
        repository: "loft-sh/kubernetes"
        tag: "v1.36.0"
      resources:
        limits:
          cpu: 100m
          memory: 256Mi
        requests:
          cpu: 40m
          memory: 64Mi
  # Define the node coreDNS should be scheduled on
  coredns:
    enabled: true
    deployment:
      nodeSelector:
        vcluster.loft.sh/team: "team-a"
      tolerations:
      - key: "vcluster.loft.sh/team"
        operator: "Equal"
        value: "team-a"
        effect: "NoSchedule"
  statefulSet:
    resources:
      limits:
        ephemeral-storage: 2Gi
        memory: 2Gi
      requests:
        ephemeral-storage: 400Mi
        cpu: 200m
        memory: 256Mi
    highAvailability:
      replicas: 1
    security:
      podSecurityContext: {}
      containerSecurityContext:
        allowPrivilegeEscalation: false
        runAsUser: 0
        runAsGroup: 0
    persistence:
      volumeClaim:
        enabled: auto
        retentionPolicy: Retain
        size: 2Gi
        storageClass: "local-path"
        accessModes: [ "ReadWriteOnce" ]
    # Scheduling vCluster Control plane pods run on nodes with the label set to team-a
    scheduling:
      nodeSelector:
        vcluster.loft.sh/team: "team-a"
      tolerations:
      - key: "vcluster.loft.sh/team"
        operator: "Equal"
        value: "team-a"
        effect: "NoSchedule"
  # Service configuration for vCluster control plane access
  # The vcluster-team-a will be accessible on Node IP Address with label team-a:30444
  service:
    enabled: true
    annotations: {}
    labels: {}
    httpsNodePort: 30444
    spec:
      type: NodePort
sync:
  # Tenants see fake nodes, pods still scheduled on correct nodes via taint and toleration
  fromHost:
    nodes:
      enabled: false
  toHost:
    pods:
      enabled: true
      enforceTolerations:
      - "vcluster.loft.sh/team=team-a:NoSchedule"
exportKubeConfig:
  context: vcluster-team-a
  secret:
    name: vcluster-team-a
    namespace: vcluster-team-a

Control Plane Cluster Configuration​

Connect to the control plane cluster, add the respective Kubernetes labels on the nodes we want the vcluster-team-a to use alongside the Kubernetes taints.

$ export KUBECONFIG=/path/to/Control Plane Cluster/kubeconfig

$ kubectl label node test-worker1 vcluster.loft.sh/team=team-a
$ kubectl get nodes --show-labels | grep "vcluster.loft.sh/team"

$ kubectl taint node test-worker1 vcluster.loft.sh/team=team-a:NoSchedule
$ kubectl taint node test-worker2 vcluster.loft.sh/team=team-b:NoSchedule

Helm Deployment​

$ helm repo add loft https://charts.loft.sh
$ helm repo update

$ export KUBECONFIG=/path/to/Control Plane Cluster/kubeconfig

$ helm upgrade --install vcluster-team-a vcluster \
  --namespace vcluster-team-a \
  --create-namespace \
  --values /the/path/to/configuration/values.yaml \
  --repo https://charts.loft.sh \
  --repository-config=''

vCluster Validation​

$ helm list -n vcluster-team-a
NAME           	NAMESPACE      	REVISION	UPDATED                                 	STATUS  	CHART          	APP VERSION
vcluster-team-a	vcluster-team-a	1       	2026-05-06 12:41:34.061614538 +0200 CEST	deployed	vcluster-0.34.0	0.34.0   

$ kubectl get pods -n vcluster-team-a -o wide
NAME                                                        READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
coredns-754d567864-5lfcg-x-kube-system-x-vcluster-team-a    1/1     Running   0          41s   10.42.0.66    test-worker1   <none>           <none>
vcluster-team-a-0                                           1/1     Running   0          70s   10.42.0.219   test-worker1   <none>           <none>

Retrieve vcluster-team-a Kubeconfig​

Once the virtual cluster is deployed, we can retrieve the kubeconfig by decoding the configuration of the secret with the name vcluster-team-a.

$ kubectl get secret vcluster-team-a -n vcluster-team-a --template={{.data.config}} | base64 -d > /the/path/to/configuration/vcluster-team-a.yaml

Open the file and update the server: https://localhost:8443 section with server: https://<DEDICATED NODE IP>:30444. Follow the same validation approach as in the vcluster-dev section.

Advantages Per-Team Dedicated Nodes​

• Resource Isolation: Teams cannot impact each other's resources
• Cost Tracking: Easier to track per-team infrastructure costs
• Compliance: Separate sensitive workloads onto specific nodes
• Performance: Predictable performance based on node allocation
• Failure Isolation: Node failure only affects one team assigned to the node/nodes

Conclusion​

This post concludes a basic vCluster setup with two separate configurations. Feel free to explore any other possible options provided by the Helm charts to cover different use cases. In the next post, we will enable the Cilium L2 Announcements feature and integrate the free version of the vCluster Platform! Stay tuned!

Resources​

• vCluster Documentation
• vCluster Schema Validation
• vCluster Helm Chart Values

✉️ Contact​

If you have any questions, feel free to get in touch! You can use the Discussions option found here or reach out to me on any of the social media platforms provided. 😊 We look forward to hearing from you!

Series Navigation​