Kubernetes has been using Docker overlay network as the default pod‑connecting mechanism in the cluster for years. While this solution is simple to configure, in production environments it reveals a number of performance and architectural limitations. In this article we will examine the internal mechanisms of Docker overlay, run brief latency and throughput tests, and present the most common pitfalls that DevOps engineers encounter. At the end you will find a practical checklist and guidelines on when to choose overlay and when to switch to another networking model.
How Docker overlay network works in Kubernetes
Overlay is a virtual layer built on top of an existing physical network (e.g., Ethernet). Each pod receives its own IP address from the 10.244.0.0/16 range (by default), and Docker uses libnetwork and iptables to encapsulate traffic in VXLAN tunnels. A packet leaving a pod is first handed to the virtual interface cbr0, then encapsulated in a VXLAN header and sent out through the host’s physical interface.
Key components:
- VXLAN (Virtual Extensible LAN) – provides a 24‑bit segment identifier (VNI), enabling isolation of multiple clusters on a single network.
- iptables NAT – translates pod addresses to host addresses, allowing communication outside the cluster.
- IPVS / kube-proxy – handles service load‑balancing, but in
iptablesmode adds an extra NAT chain, increasing the rule count.
Why does this matter? Every additional hop (VXLAN → iptables → IPVS) introduces latency overhead and raises CPU usage, which can become a bottleneck in large clusters.
Docker bridge vs. overlay – which to choose?
Docker bridge is a simple L2 bridge that operates on a single host. Packets are not encapsulated, eliminating VXLAN overhead, but it requires manual routing configuration between hosts. Overlay, on the other hand, works “out‑of‑the‑box” and automatically connects all nodes, at the cost of extra processing.
In practice:
- Bridge delivers lower latency (< 0.2 ms) and higher throughput (up to 10 Gbps) in small single‑node environments.
- Overlay adds about 0.5‑1 ms latency and limits throughput to 2‑3 Gbps when the CPU is fully utilized.
The choice depends on requirements: if you need a simple, high‑performance connection in a closed test environment – bridge. In a multi‑node cluster where automatic topology is crucial – overlay.
Performance testing – what to measure?
To benchmark overlay performance, iperf3 and ping are most commonly used between two pods placed on different nodes. Below is a sample command set:
# start iperf server on node A
kubectl run iperf-server --image=networkstatic/iperf3 --restart=Never -- -s
# start iperf client on node B
kubectl run iperf-client --image=networkstatic/iperf3 --restart=Never -- -c iperf-server -t 30
# measure latency
kubectl exec -it iperf-client -- ping -c 10 iperf-serverIn a typical 5‑node cluster (AWS m5.large) the average throughput was ~2.4 Gbps with latency of 0.8 ms using overlay. For comparison, with bridge latency dropped to 0.3 ms and throughput rose to 7.8 Gbps.
When to use overlay and when not to
When to use:
- Multi‑node environments with dynamic pod scaling.
- Network isolation between different teams or environments is required (e.g., dev vs. prod).
- A simple networking model without manual routing configuration is needed.
When not to use:
- Maximum bandwidth and minimal latency are critical (e.g., databases, high‑performance HPC).
- You have full control over the infrastructure and can manage L2/L3 routing manually.
- In a cluster with a very large number of pods (>10 000) iptables rules become unmanageable.
Typical production pitfalls and how to eliminate them
1. iptables table overflow – with a large number of services kube‑proxy generates thousands of rules, increasing packet‑processing time. Solution: switch kube‑proxy to IPVS mode, which uses hash tables and is much faster.
2. VXLAN fragmentation – incorrect MTU (default 1500) causes fragmentation, raising latency. Recommendation: set MTU to 1450 for the cbr0 interface and adjust the subnet‑layer MTU accordingly.
3. Lack of CNI monitoring – latency issues often stem from misconfigured CNI (e.g., Flannel vs. Calico). Use Prometheus + cAdvisor + kube‑network‑policy to collect metrics container_network_receive_bytes_total and container_network_transmit_errors_total.
“Overlay is convenience, not magic – without conscious optimization it becomes a bottleneck for any low‑latency application.”
Practical example – implementation checklist
Below is a ready‑made list of steps that should be included in every playbook preparing a Kubernetes cluster for production with Docker overlay network:
- Check Docker and kube‑proxy versions (minimum 1.20/1.21) – newer versions include fixes for VXLAN handling.
- Set MTU 1450 on the
cbr0interface and verify no fragmentation (ping -M do -s 1400). - Switch kube‑proxy to
IPVS(kubectl edit cm kube-proxy -n kube-system). - Enable CNI monitoring (Prometheus + node‑exporter) and set alerts on
network_latency_seconds> 1. - Consider alternative CNI plugins (Calico, Cilium) for high security or performance requirements.
After completing the above steps, most common latency and throughput issues will be eliminated, and the cluster will be ready to handle production workloads.
Conclusion and invitation to collaborate
Docker overlay network in Kubernetes is a powerful tool that simplifies networking in large clusters, but it requires conscious performance optimization. Understanding VXLAN, iptables, and IPVS mechanisms helps avoid the most common pitfalls, and applying a practical checklist ensures stability in production environments. If you need assistance with network auditing, migration to a more performant CNI, or implementing advanced monitoring, the Coderia.it team is ready to support your project – contact us today.
