Cilium is a widely used open-source solution that integrates into the Container Network Interface (CNI) hooks of Kubernetes to provide software-based load balancing and SSL processing. This allows generic computing horsepower to be used in lieu of expensive and proprietary hardware more commonly used for these functions in years prior. While the capabilities and lower financial costs of Cilium are attractive, use of Cilium does impose a significant learning curve and exhibits a few technical difficulties that make it more difficult to confirm configurations are correct for desired solutions.

The differences between hardware-based and software-based load balancing and SSL processing, installation of Cilium and use of ARP versus BGP access patterns are provided in these related posts:

Cilium and Kubernetes - Caveats / Concepts
Cilium and Kubernetes - Installing Cilium Within Kubernetes
Cilium and Kubernetes - Configuring SSL and Load Balancing
Cilium and Kubernetes - Externally Accessing Services via ARP
Cilium and Kubernetes - Externally Accessing Services via BGP

Important Cilium Caveats

After eventually resolving specific quirks with the implementation of Cilium, use of Cilium for load balancing and SSL / TLS processing within Kubernetes is relatively straightforwrard in hindsight. However, the Cilium capabilities come with two CRUCIAL caveats. One is related to a documented bug within Cilium which has not been fixed as of version 1.19.3 (April 30, 2026) and has no specific timeline for development / resolution. The second involves a likely flaw with FRR (Free Range Router) used in illustrating BGP based access to services deployed using Cilium. The SHORT descriptions of these two flaws are provided below.

Missing tags on auto-generated services for Gateways -- When a Gateway is defined for SSL processing, Cilium generates a second virtual load balancer Service object. However, this second Service is not created with the tags used for the parent Gateway that trigger advertisement of the assigned virtual IP when using ARP or BGP. This makes the Gateway VIP unreachable from outside the cluster, defeating the very purpose of using Cilium.

WORKAROUND -- This problem can be solved by manually applying the required key=value label to the auto-generated Service resource after Cilium creates it.

Use of Free Range Router (FRR) in virtualized environments can encounter IP forwarding glitches -- Cilium provides its own code that implements a BGP router process within Kubernetes. External to the cluster, either a real router with BGP capabilities or a virtual router must speak BGP to learn routes from the Kubernetes cluster for advertised IPs. Free Range Router (FRR) is a common open-source BGP router implementatio that runs atop Linux. Use of FRR on a virtual machine that is also talking to a Cilium BGP router also running on a virtualized machine can encounter situations where a Linux server running FRR will not properly forward incoming traffic towards the Kubernetes cluster. At the same time, test traffic originated WITHIN that Linux FRR host will function correctly.

WORKAROUND -- This problem can be mitigated by running the FRR instance on a hardware Linux instance rather than a VM. The problem is likely due to too many layers of nested virtualized Ethernet packet handling. Running FRR on a physical Linux host exhibits no issues accepting incoming traffic and forwarding it correctly into the Kubernetes BGP router instances.

Overall, Cilium is relatively easy to deploy, undeploy, re-deploy, ad nauseum. However, partly due to its nature and that of Kubernetes in general, the process of getting a service deployed on a Kubernetes + Cilium platform can be INCREDIBLY error prone unless one works from a very formal checklist to make sure all reqirements have been addressed. This document will provide a visual image illustrating all of these capabilities:

defining distinct IP ranges for pods deployed to dev and prod environments
defining distinct IP ranges for virtual IPs deployed to dev and prod environments
using unique ConfigMap entries for dev and prod environments
using unqiue SSL cert/key files for dev and prod environments
making hostname distinctions in URLs for HTTPRoute objects
configuring distinct dev / prod Gateway objects with SSL termination

If Kubernetes is ONLY being used in a hobbyist scenario purely for hobbies, virtually none of this complexity is required or warranted. However, if you are learning Kubernetes as part of work requirements, it is likely ANY use of Kubernetes will require these types of operational distinctions to be understood and incorporated into local practices.

Traditional Hardware Based Load Balancing and SSL

Understanding the configuration and use of Cilium is easier after first comparing a traditional hardware based service topology to a Cilium-based footprint within Kubernetes. The approach with Cilium and Kubernetes is replicating each task required via hardware but doing so using dynamically assigned resources. A single administrator can drive all of the assignments but that requires a single administrator to UNDERSTAND all of the steps that were often handled by multiple people in a hardware driven environment.

In this example, a Java Spring Boot web service that accesses financial CD information is being deployed to run in three instances behind a load balancer which not only balances the traffic but handles all SSL processing that encrypts client traffic prior to executing the core service.

In a hardware based deployment, the topology might look like this:

In this physical topology,

The web service runs as a JVM on three physical hosts on the arbitrary subnet 192.168.101.0/24
One or more routers connects this host segment to another segment housing a load balancer appliance - a physical system commonly using ASICs to accelerate balancing decisions and handle de-encryption and encryption of traffic using SSL / TLS keys.
The load balancer's configuration for this service includes the public and private key pair associated with the hostname api.mdhlabs.com
Actual clients accessing the web service behind api.mdhlabs.com exist on another subnet 192.168.99.0/24 which might be one or more router hops away from the load balancer.

In order to turn up this configuration,

A load balancer appliance had to be purchased and installed, at a cost which might easily surpass $100,000 per appliance for high-capacity units.
A security administrator had to generate or purchase the public / private keys for api.mdhlabs.com
Server administrators had to assign IPs for the three physical hosts, turn up those hosts and deploy the JAR file and JVM to run the three instances of the service.
The load balancer administrator had to assign an IP for the virtual IP of api.mdhlabs.com
The load balancer administrator and network administrator had to ensure the IP range used for the virtual IPs was routed within all of these networks and reachable from the 192.168.99.0/24 subnet.
The load balancer administrator had to coordinate with the application administrators to identify what health checks to perform on the service to eliminate dead members from the pool to avoid outages, etc.

Software Based Load Balancing and SSL

Use of Cilium within Kubernetes to perform the "layer 7" functions of load balancing and SSL termination eliminates the need for expensive appliance hardware by spreading the processing work among all of the compute nodes within the Kubernetes cluster. However, virtually all of the same administrative tasks are still required, they are just performed in a different way.

The topology below reflects the typical Deployment, Service, HTTPRoute and Gateway objects configured within Kubernetes to implement load balancing and SSL termination.

In this virtual software-based topology,

A deployment defines a ReplicaSet that operates three instances of the web service as pods on the three Kubernetes nodes.
A service defines a single load balancer IP in front of the three service pods that evenly distributes traffic to the unencrypted endpoints. This virtual IP is automatically assigned from a previously configured pool of IP addresses.
An HTTPRoute object identifies hostname references and URI paths that should be steered to the Service load balancer.
A Gateway object identifies SSL keys and expected hostnames that should be used to decode incoming external traffic that can then be routed by the HTTPRoute into the Service into the Pods. A second virtual IP is automatically assigned to this endpoint from a predefined range.

In order to turn up this configuration,

No hardware load balancer was required -- all processing is performed by the hosts within the Kubernetes cluster.
The Kubernetes administrator had to coordinate with the network administrator to pick a pool of IP addresses to automatically allocate within Cilium.
IP addresses for the pods are automatically assigned.
The Kubernetes administrator had to obtain the same public / private SSL keys for api.mdhlabs.com
The Kubernetes administrator had to coordinate with the application administrators to identify what health checks to perform on the service to eliminate dead members from the pool to avoid outages, etc.
The Kubernetes administrator and application administrator had to define the Service, HTTPRoute and Gateway resources within additional meta tags to drive advertisement of the assigned virtual IPs into the larger network to allow client access.

Use of Cilium or similar "layer 7" tools within Kubernetes involves addressing one final networking problem -- ensuring access to the resulting service from outside the cluster. Cilium provides two approaches for doing this:

Advertising via ARP -- This approach requires the virtual IPs to be assigned from the same subnet used for the physical Kubernetes hosts. The nodes then answer any ARP request attempting to find a MAC for the active virtual IP then process any traffic sent to that VIP.
Advertising via BGP -- This approach uses the BGP routing protocol to advertise host routes (e.g. 192.168.77.128/32) for each virtual IP to routers outside the Kubernetes cluster which use them to route traffic to the VIPs into the cluster for processing.

The ARP approach is somewhat less complicated but poses more potential for administrative mistakes if IPs meant for virtual IPs are accidently assigned to new physical hosts on the network segment serving the physical Kubernetes hosts. The BGP approach requires more expertise at network design and routing protocols but makes it easier to avoid assignment conflicts between VIPs and physical hosts. Each approach requires different configuration options to be supplied to Cilium at installation, however these configuration options are not mutually exclusive to each other. Both approaches can be used in a single Kubernetes cluster simultaneously without conflict.

More information on using Cilium within Kubernetes is provided in other posts in this series:

mdhLabs

Thursday, April 30, 2026

Cilium and Kubernetes - Caveats / Concepts

Important Cilium Caveats

Traditional Hardware Based Load Balancing and SSL

Software Based Load Balancing and SSL