Read on Twitter
After witnessing the ordeal of setting up the AWS ALB ingress controller on EKS, I think there are a few important points that need to be made about running Kubernetes on AWS!
1/...
1/...
The ALB ingress controller brings some interesting features (for instance, the ability to route requests directly to pods when running in the so-called "IP mode").
It also lets you leverage other AWS features (VPC subnets etc) in a rather elegant way.
2/...
It also lets you leverage other AWS features (VPC subnets etc) in a rather elegant way.
2/...
It also brings *a lot* of complexity. For instance, it requires to set up (and understand) how to map together IAM policies, roles, and Kubernetes ServiceAccounts. It's not something to treat lightly (since we're talking about cluster permissions and security).
3/...
3/...
Most orgs won't need the above-mentioned "interesting features"; and/or they could also get them with other simpler alternatives.
If your k8s cluster has less than 10 nodes, or less than 10 ingresses - there is a good chance you don't need the ALB controller.
4/…
If your k8s cluster has less than 10 nodes, or less than 10 ingresses - there is a good chance you don't need the ALB controller.
4/…
Now, let's say that after thoughtful analysis, you decide that you absolutely definitely literally want to deploy
something like the AWS ALB ingress controller.
Then you need someone on your team who has *both* very strong k8s and AWS skills.
5/...
something like the AWS ALB ingress controller.
Then you need someone on your team who has *both* very strong k8s and AWS skills.
5/...
Following a tutorial is one thing; transforming these tutorials (either in AWS-land or Kubernetes-land) into reproducible deployments can be a whole different story, and quite a time-consuming one.
6/...
6/...
I suggest to start with the most popular options, like the NGINX ingress controller, or Traefik. These are a few orders of magnitude easier to set up; and they can be deployed on any compliant Kubernetes cluster, even local ones like kind, k3d, minikube, etc.
7/...
7/...
In fact, they will also save you money! By default the ALB ingress controller creates one ALB per ingress, which will cost you $15/mo (+LCU costs). NGINX or Traefik typically create a single NLB (so you pay that once per cluster, not once per ingress).
8/...
8/...
I'd also suggest to use as few proprietary annotations as possible in your ingress resources. Because if one day you decide to switch to another controller (for any reason!) that'll be more work for you. Don't ban them entirely, but keep that in mind.
9/...
9/...
A good litmus test: the controllers and operators that you add to your Kubernetes clusters should all be deployed automatically (feel free to use shell scripts, Cloud Formation, Terraform, or whatever). If my cluster install process has manual steps, that's a red flag!
10/...
10/...
To wrap up: some cluster add-ons (like the aforementioned ALB controller) are extremely powerful, but they are also the kind that contributes to Kubernetes' reputation of being so complex.
"This shouldn't be so hard!!!"
11/...
"This shouldn't be so hard!!!"
11/...
You're right, this shouldn't be so hard. But you don't have to complete a 5 year carpentry apprenticeship and cut your own tree to build a bookshelf. You could also get one from IKEA, build it by lunch time, and do whatever you want with the remaining 5 years of time :)
12/12
12/12
