Read on Twitter
Having seen some of the discussions around cloud-native network architectures yesterday, I wanted to present my counter-arguments. Doing this with one of two ad-hoc tweets is pointless so fasten your seatbelts, this is going to be a 
https://twitter.com/danieldibswe/status/1298566332703809542
https://twitter.com/danieldibswe/status/1298566332703809542
Specifically, I'm going to argue two points:
1. Cloud-native archs are messy and misdesigned
2. Network infra team would have done a better job
Note:, I'm not much familiar with Swarm (is that still a thing?) so for my examples I'll be using k8s which I'm more familiar with.
1. Cloud-native archs are messy and misdesigned
2. Network infra team would have done a better job
Note:, I'm not much familiar with Swarm (is that still a thing?) so for my examples I'll be using k8s which I'm more familiar with.
Point #1. I think that cloud-native archs are complex, not messy. Managing large fleet of dynamic workloads at (even small) scale is a complex problem, so you shouldn't expect a simple solution. Some design decisions may be suboptimal at first but they generally evolve over time
There are two main networking services that cloud-native archs implement. 1) Plain IP connectivity 2) Load-balancing. The first one is a very simply problem with super easy solutions. Each node (VM or baremetal) gets a unique range of IPs and allocates them to pods.
Reachability between nodes is established with either BGP (for on-prem case), static routes or secondary subnets (for public cloud case) or overlays. Overlay is the least preferred and the first two options create a very clean design that's easy to troubleshoot.
Service is the abstraction that's most commonly known in the infra world as the load-balancing. You have a large number of pods that represent a single "service" and you want to spread requests across them in a fair way.
This is where you start getting into the iptables and NATs on multiple nodes and this is what was perceived as messy. Look at it from this pov - you need to design an efficient LB solution in a highly dynamic and horizontally scalable environment.
You need to do it in a vendor- and cloud-independent way that would work with most LTS linux versions. Using iptables (and IPVS) is the only way to do it. It may not be very efficient but what you gain is extreme flexibility and scale and "out of the box" experience.
There are ways to optimise it further with eBPF (Cilium, Calico) so in a few years, you won't see iptables used as much. If you don't want to rely too heavily on iptables now, there's always the sidecar pattern. Extra complexity - sure! But there's always a huge gain in features.
Now let's address another aspect of complexity - troubleshooting. The packet forwarding pipeline looks messy and complex. Try comparing this with any BCM chip architecture. Does it look messy/complex? Yes! Do you need to know it? Some people will say yes ( @networkservice).
Do you need to troubleshoot it? Most likely not. The same applies to cloud-native designs. For every company there a choice - take on the complexity and mitigate the risks by hiring the right people or offload the risk by buying a support contract.
Another thing worth mentioning is that a lot of these designs can be run in public clouds (cloud-native, heh?). AKS, EKS, GKE will take care of all of this networking complexity for you.
You'd still need to have some basic troubleshooting skills (is it DNS?) but by and large you can rely on the vendor to give you the working combination of iptables rules.
Point #2. I know there are success stories of cross-team silo-breaking designs ( @cmsirbu do you know if that design is still running?). However in my experience most attempts to involve network infra team results in increasing delays, stalling, suggestions of stack of F5s.
One design I've seen suggested advertising a /19 full of /32s into the underlay and questioning whether the size of the FIB tables was sufficient.
Given that it's a complex solution to begin with, all network-integrated designs end up being complex and convoluted as well.
And what's the point in leaking that complexity into the underlay when you can stick to the minimal interface (BGP, static routes) and keep the failure domains contained?
