This post is installment #3 in a series of posts providing directions on installing and using Cilium for load balancing and SSL processing. Links to all of the posts in the series are provided below for convenience.
Cilium and Kubernetes - Caveats / Concepts
Cilium and Kubernetes - Installing Cilium Within Kubernetes
Cilium and Kubernetes - Configuring SSL and Load Balancing
Cilium and Kubernetes - Externally Accessing Services via ARP
Cilium and Kubernetes - Externally Accessing Services via BGP
This post in the series will explain the use of the newer Gateway and HTTPRoute objects provided by the CNI framework within Kubernetes for implementing SSL processing and load balancing. These newer resource object formats supercede older Ingress objects previously standardized within Kubernetes. Because this tutorial series is intended to illustrate how to access such services using both the ARP and BGP schemes supported by Cilium, this post will reflect the creation of two parallel sets of services, one that uses ARP to allow access from the physical host segment and the other using BGP to allow advertising of avialable service points via routing protocols. This will hopefully make it easier to understand how the approaches differ.
 |
Because this tutorial series is intended to explain BOTH the ARP and BGP approaches, the illustrations and instructions will reflect BOTH approaches being deployed within a single cluster simultaneously with different services being active simultaneously using both approaches. At first, this may appear to make naming conventions for files and Kubernetes objects more verbose / obtuse than necessary. However, such complexities will better illustrate where the two approaches differ and how those differences need to be accomodated outside the cluster. |
Creating a Deployment for a ReplicaSet
Both theARP and BGP examples start with an underlying deployment of a redundant set of pods executing the underlying Spring Boot web service. These will be named differently to ensure completely separate processing paths but in reality, the core deployment is unaffected by the adoption of Cilium to provide SSL and load balancing at higher levels.
Here is the content of the file cd-arp-deploy.yaml for the deployment that will be used by the ARP implementation.
apiVersion: apps/v1
kind: Deployment
metadata:
name: cd-arp-deploy
spec:
replicas: 3
selector:
matchLabels:
app: cd-arp
template:
metadata:
labels:
app: cd-arp
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- cd-arp
topologyKey: "kubernetes.io/hostname"
containers:
- name: cdtrackerapi
image: fedora1.mdhlabs.com:5000/cdtrackerapinossl:latest
imagePullPolicy: Always
startupProbe:
httpGet:
path: /cdtracker/api/readycheck
port: 6680
periodSeconds: 2
failureThreshold: 10
readinessProbe:
httpGet:
path: /cdtracker/api/readycheck
port: 6680
initialDelaySeconds: 0
periodSeconds: 60
livenessProbe:
httpGet:
path: /cdtracker/api/healthcheck
port: 6680
initialDelaySeconds: 10
periodSeconds: 60
envFrom:
- configMapRef:
name: cd-configmap
- secretRef:
name: cd-dbpass-secret
ports:
- containerPort: 6680
protocol: TCP
tolerations:
- operator: "Exists"
effect: "NoSchedule"
Here is the content of the file cd-bgp-deploy.yaml for the deployment that will be used by the BGP implementation.
apiVersion: apps/v1
kind: Deployment
metadata:
name: cd-bgp-deploy
spec:
replicas: 3
selector:
matchLabels:
app: cd-bgp
template:
metadata:
labels:
app: cd-bgp
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- cd-bgp
topologyKey: "kubernetes.io/hostname"
containers:
- name: cdtrackerapi
image: fedora1.mdhlabs.com:5000/cdtrackerapinossl:latest
imagePullPolicy: Always
startupProbe:
httpGet:
path: /cdtracker/api/readycheck
port: 6680
periodSeconds: 2
failureThreshold: 10
readinessProbe:
httpGet:
path: /cdtracker/api/readycheck
port: 6680
initialDelaySeconds: 0
periodSeconds: 60
livenessProbe:
httpGet:
path: /cdtracker/api/healthcheck
port: 6680
initialDelaySeconds: 10
periodSeconds: 60
envFrom:
- configMapRef:
name: cd-configmap
- secretRef:
name: cd-dbpass-secret
ports:
- containerPort: 6680
protocol: TCP
tolerations:
- operator: "Exists"
effect: "NoSchedule"
Creating the Inner LoadBalancer Service
Both the ARP and BGP examples include an inner load balancer which distributes requests to the set of pods without SSL encryption. The two examples below do reflect one key difference between them. The ARP version specifies a label of mdhlabs-arp: enable and the BGP version specifies a label of mdhlabs-bgp: enable. This label coupled with the environment name of prod or development will drive selection of the assigned load balancer virtual IP from the pools of IP space configured earlier. Other than that tag to drive IP selection, these two Service definitions are functionally identical.
Here is the content of the file cd-arp-svc.yaml for the deployment that will be used by the ARP implementation.
apiVersion: v1
kind: Service
metadata:
labels:
app: cd-arp
mdhlabs-arp: enable
name: cd-arp-svc
spec:
selector:
app: cd-arp
ports:
- protocol: TCP
port: 6680
targetPort: 6680
type: LoadBalancer
# externalTrafficPolicy controls how requests from OUTSIDE the
# cluster are distributed WITHIN the cluster
# Local = traffic is processed by first node / pod that attracted the request
# but does not undergo source NAT
# Cluster (default) = requests are balanced across all nodes/pods but source IPs are NATed
externalTrafficPolicy: Cluster
# internalTrafficPolicy controls how reqeuests originating from WITHIN the
# cluster are distributed:
# Local - requests stay within pods on same node
# Cluster (default) - requests are balanced across all nodes and pods
internalTrafficPolicy: Cluster
Here is the content of the file cd-bgp-svc.yaml for the deployment that will be used by the BGP implementation.
apiVersion: v1
kind: Service
metadata:
labels:
app: cd-bgp
mdhlabs-bgp: enable
name: cd-bgp-svc
spec:
selector:
app: cd-bgp
ports:
- protocol: TCP
port: 6680
targetPort: 6680
type: LoadBalancer
# externalTrafficPolicy controls how requests from OUTSIDE the
# cluster are distributed WITHIN the cluster
# Local = traffic is processed by first node / pod that attracted the request
# but does not undergo source NAT
# Cluster (default) = requests are balanced across all nodes/pods but source IPs are NATed
externalTrafficPolicy: Cluster
# internalTrafficPolicy controls how reqeuests originating from WITHIN the
# cluster are distributed:
# Local - requests stay within pods on same node
# Cluster (default) - requests are balanced across all nodes and pods
internalTrafficPolicy: Cluster
For both of these service definitions, more explanation of the externalTrafficPolicy and internalTrafficPolicy parameters is warranted. As referenced in the comment lines of the YAML files, the externalTrafficPolicy parameter controls whether externally arriving traffic should be balanced across ALL pods in the cluster (Cluster) or stick to pods running on the same node that accepted the traffic from outside the cluster (Local). Similarly, the internalTrafficPolicy parameter controls whether traffic originating from WITHIN the cluster (such as web service A calling web service B) are balanced across all pods in the cluster (Cluster) or stick to pods on the same node that originated the traffic. In general, if true load balancing is desired / required, these should be set to Cluster.
Creating the HTTPRoute Object
In the new CNI based solutions for layer 7 processing, the HTTPRoute object provides a structure for defining the hostnames appearing in incoming traffic and URI paths that should be routed to underlying Service objects that steer into ReplicaSets of pods. This layer of traffic routing involves matches on the hostname appearing into a URL such as https://api.mdhlabs.com/cdtracker/api/healthcheck which might be different across environments such as https://apidev.mdhlabs.com/cdtracker/api/healthcheck. As a result, this layer of the flow will typically require environment-specific configuration files. While these files may differ in content because of environment, this layer does not reflect any differences based upon the use of ARP versus BGP.
Here is the content of the file cd-arp-httproute.prod.yaml for the deployment that will be used by the ARP implementation.
# cd-arp-httproute.prod.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: cd-arp-httproute
namespace: prod
spec:
parentRefs:
- name: cd-arp-gw
hostnames:
- "api.mdhlabs.com"
rules:
- matches:
- path:
type: PathPrefix
value: /cdtracker/api
backendRefs:
- name: cd-arp-svc
port: 6680
Here is the content of the file cd-bgp-httproute.prod.yaml for the deployment that will be used by the BGP implementation.
# cd-bgp-httproute.prod.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: cd-bgp-httproute
namespace: prod
spec:
parentRefs:
- name: cd-bgp-gw
hostnames:
- "api.mdhlabs.com"
rules:
- matches:
- path:
type: PathPrefix
value: /cdtracker/api
backendRefs:
- name: cd-bgp-svc
port: 6680
Defining a Secret for the SSL Certificate and Private Key
The Gateway resource defined next identifies the hostname expected for incoming traffic and must identify where the public certificate and private key required for that SSL processing will be housed for use by the gateway in decrypting traffic. The following command creates a Secret of type TLS resource in the prod namespace referencing the certificate and key files needed.
mdh@fedora1:~/gitwork/webservices/cdtrackerapi $ kubectl create secret tls -n prod \
api-secrettls --cert=/containeretc/cdtrackerapi/api.mdhlabs.com.cert.pem \
--key=/containeretc/cdtrackerapi/api.mdhlabs.com.key.pem
secret/cdtrackerapi-secrettls created
mdh@fedora1:~/gitwork/webservices/cdtrackerapi $
|
Use of the basic Secret object within Kubernetes to supply private SSL key information to deployed resources likely has some security concerns associated with it, particularly with the lack of encryption on the wire between the etcd instance of the cluster and nodes that read the Secret object when starting Gateway objects. However, optimization of that aspect of SSL administration is beyond the scope of this tutorial. |
Creating the Gateway Object
In the new CNI based solutions for layer 7 processing, the Gateway object replaces the older Ingress component. Like the HTTPRoute object, the Gateway object will often contain references to hostnames and related SSL keys that will require distinct configurations per environment. Like the Service objects earlier, the two examples below do reflect one key difference between them. The ARP version specifies a label of mdhlabs-arp: enable and the BGP version specifies a label of mdhlabs-bgp: enable. This label coupled with the environment name of prod or development will drive selection of the assigned load balancer virtual IP from the pools of IP space configured earlier. Other than that tag to drive IP selection, these two Gateway definitions are functionally identical.
Here is the content of the file cd-arp-gw.prod.yaml for the deployment that will be used by the ARP implementation.
# cd-arp-gw.prod.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
labels:
app: cd-arp
service-name: cd-arp-svc
mdhlabs-arp: enable
# NOTE: The mdhlabs-arp=enable label above must match the label
# specified by a CiliumL2AdvertisementPolicy object to trigger
# advertisement via ARP. HOWEVER, this Gateway object generates
# a second Service object of type LoadBalancer that must also
# have this mdhlabs-arp=enable label to trigger the actual ARP advertisement.
#
# Cilium up to version 1.19.3 has a bug that fails to copy this
# label to that auto-generated Service which prevents the ARP advertisement
# from being generated. That label must be manually added after
# this gateway is created via this command:
#
# kubectl -n prod label service cilium-gateway-cd-arp-gw mdhlabs-arp=enable
#
# Once added, Cilium will attempt to generate the ARP for the VIP
# NOTE: This tag must be reapplied each time Cilium auto-generates
# the Service.
name: cd-arp-gw
namespace: prod
spec:
gatewayClassName: cilium
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: "api.mdhlabs.com"
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: api-secrettls
Here is the content of the file cd-arp-gw.prod.yaml for the deployment that will be used by the ARP implementation.
# cd-bgp-gw.prod.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
labels:
app: cd-bgp
service-name: cd-bgp-svc
mdhlabs-bgp: enable
# NOTE: For VIPs assigned for Gateway and Service objects, Cilium
# expects to match a label in the Gateway or Service to a label
# that triggers advertisement of the IP via ARP or BGP.
#
# THe label here (mdhlabs-bgp: enable) matches a label in a
# CiliumBGPAdvertisement config which SHOULD trigger the IP
# assigned here to be adverised. HOWEVER, this mechanism actually
# works on the SERVICE object and here, this Gateway auto-generates
# a SERIVCE object but Cilium does not label that auto-generated
# service with the (mdhlabs-bgp: enable) tag here so the IP
# is NOT advertised.
#
# This (mdhlabs-bgp: enable) tag must be MANUUALY added to the
# auto-generated Service for the Gateway EACH time the Gateway
# is deployed.
name: cd-bgp-gw
namespace: prod
spec:
gatewayClassName: cilium
listeners:
- name: https
protocol: HTTPS
port: 443
hostname: "api.mdhlabs.com"
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: api-secrettls
Manually Labeling the Auto-Generated Service for a Gateway
As referenced in the YAML file examples in the prior section that define a Gateway for SSL termination, Cilium has a known bug
in its advertisement functionality for both ARP and BGP that requires a manual workaround. The CONCEPT of the advertising mechanism
is that defining a special label such as mdhlabs-arp: enable or mdhlabs-bgp: enable that matches
a policy for L2 ARP or BGP will trigger actions that generate the advertisement. However, Cilium releases up to 1.19.3 fail to
copy this attribute from the Gateway resource specifiying it to the auto-generated Service object that creates the LoadBalancer
for the gateway. As a result, the ARP or BGP function is never notified to trigger its advertisement process and the IP
assigned to the LoadBalancer for the Gateway never gets advertised and is not reachable outside the cluster.
While waiting for a bug fix in a future Cilium release, this problem can be manually corrected by manually adding the desired label to the auto-generated Service resource after creating the Gateway in the cluster. The auto-generated Service object is always assigned a name of cilium-gateway-originalgatewayname. For the gateways defined as cd-arp-gw and cd-bgp-gw, the following commands would be required to attach the expected label to trigger ARP or BGP advertisement of the IP:
kubectl -n prod label service cilium-gateway-arp-gw mdhlabs-arp=enable
kubectl -n prod label service cilium-gateway-bgp-gw mdhlabs-bgp=enable
Verification of Components After Deployment
Despite what the words might imply, the command kubectl -n prod get all does NOT actually exaustively list all deployed components of all types in a given namespace. It only returns information on pods, services, deployments and replicacsets.
mdh@fedora1:~/gitwork/webservices/cdtrackerapi $ kubectl -n prod get all
NAME READY STATUS RESTARTS AGE
pod/cd-bgp-deploy-6675bf6bb5-cw6xd 1/1 Running 1 (4h31m ago) 16h
pod/cd-bgp-deploy-6675bf6bb5-nrkq9 1/1 Running 1 (4h31m ago) 16h
pod/cd-bgp-deploy-6675bf6bb5-tw5lx 1/1 Running 1 (4h31m ago) 16h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cd-bgp-svc LoadBalancer 10.100.125.156 192.168.77.128 6680:32391/TCP 16h
service/cilium-gateway-cd-bgp-gw LoadBalancer 10.101.163.26 192.168.77.129 443:32549/TCP 16h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cd-bgp-deploy 3/3 3 3 16h
NAME DESIRED CURRENT READY AGE
replicaset.apps/cd-bgp-deploy-6675bf6bb5 3 3 3 16h
mdh@fedora1:~/gitwork/webservices/cdtrackerapi $
In order to confirm the status of all resource types associated with a Cilium based load balancer configuration, the component types must be explciitly listed like this:
mdh@fedora1:~/gitwork/webservices/cdtrackerapi $ kubectl -n prod get pods,deployments,replicaset,services,httproute,gateway
NAME READY STATUS RESTARTS AGE
pod/cd-bgp-deploy-6675bf6bb5-cw6xd 1/1 Running 1 (4h36m ago) 17h
pod/cd-bgp-deploy-6675bf6bb5-nrkq9 1/1 Running 1 (4h37m ago) 17h
pod/cd-bgp-deploy-6675bf6bb5-tw5lx 1/1 Running 1 (4h37m ago) 17h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cd-bgp-deploy 3/3 3 3 17h
NAME DESIRED CURRENT READY AGE
replicaset.apps/cd-bgp-deploy-6675bf6bb5 3 3 3 17h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cd-bgp-svc LoadBalancer 10.100.125.156 192.168.77.128 6680:32391/TCP 17h
service/cilium-gateway-cd-bgp-gw LoadBalancer 10.101.163.26 192.168.77.129 443:32549/TCP 17h
NAME HOSTNAMES AGE
httproute.gateway.networking.k8s.io/cd-bgp-httproute ["api.mdhlabs.com"] 17h
NAME CLASS ADDRESS PROGRAMMED AGE
gateway.gateway.networking.k8s.io/cd-bgp-gw cilium 192.168.77.129 True 17h
Based upon all the configuration created to this point, both the cd-arp-deploy and cd-bgp-deploy
should be physically running on the cluster and they should be reachable from any of the three Kubernetes nodes kube1, kube2
or kube3. Note that an HTTPS service cnanot be reached with a simple curl command that specifies the host IP address instead
of the fully qualified domain name.
[root@kube1 ~]#curl -X GET https://api.mdhlabs.com/cdtracker/api/healthcheck{ "host": "cd-arp-deploy-b94f54dbf-s6ftj", "ready": "true" + "time": "2026-04-30 18:43:29" }[root@kube1 ~]# [root@kube1 ~]# [root@kube1 ~]#curl -X GET https://192.168.99.129/cdtracker/api/healthcheckcurl: (35) Recv failure: Connection reset by peer [root@kube1 ~]#
This is because the Gateway is testing the hostname in the request against the SSL certificate and finding the IP string does not match the host name in the certificate. To test both the ARP version (on 192.168.99.129) and BGP version (on 192.168.77.129), the local /etc/hosts file will need to be edited to flip between the two IP addresses mapped to api.mdhlabs.com.
The final point to note here is that with no other configuration being completed, even though these services are running WITHIN the cluster, the services cannot be accessed from OUTSIDE the cluster. The processes for providing external access into these services via ARP and BGP are covered in the final two installments of this series.
More information on using Cilium within Kubernetes is provided in other posts in this series:
Cilium and Kubernetes - Caveats / Concepts
Cilium and Kubernetes - Installing Cilium Within Kubernetes
Cilium and Kubernetes - Configuring SSL and Load Balancing
Cilium and Kubernetes - Externally Accessing Services via ARP
Cilium and Kubernetes - Externally Accessing Services via BGP
