THREAD:

Is there a best instance type for Kubernetes?

It depends, but yes — some instances are better than others.

1/7

Not all CPU and memory in Kubernetes nodes can be used to run Pods.

CPU and Memory are divided into:

- Operating system
- Kubelet, CNI, CRI
- Pods
- Eviction threshold

More: https://twitter.com/danielepolencic/status/1263046916184526848

2/7

So what percentage of memory and CPU is used to run Pods?

And how does that change in cloud vendors and across instances?

Let's have a look:

3/7

Smaller instances are inefficient.

If you use a 4GB memory 1vCPU instance for your nodes, you're probably wasting 20-30% of resources.

But there's more.

There's a limit on the number of Pods you can run per Node.

- GKE 110
- AKS 250
- EKS (depends on ENIs)

4/7

In GKE, when you provision a large (256GB) node, you can't deploy more than 110 Pods.

If after 110 pods there's memory left, it's wasted.

What's the efficiency for Pods with 1GB, 2GB, 4GB, 8GB or 16GB of memory?

5/7

In EKS, the number of Pods you can have on an instance is dictated by the ENIs limits.

Again, the chart shows the efficiency for Pods with increasing memory limits.

6/7

In AKS, the chart is similar to GKE, but the graph is shifted to the right since the pod limit is higher (250 Pods).

Notice how the efficiency drops after 128GiB because you hit the Pod limits.

7/7

Summary:

- run nodes with >=64GB of memory
- at least 2vCPU
- pay attention to pod limits per node

I will open-source the research so you can run the number yourself.

Did I miss something?
What would you like to see next?

Let me know!