Serverless workers: a thread about SQS and AWS Lambda. With visuals using a simulator I built :)
Let's start with just SQS: it's a queue. Like RabbitMQ. You can send (millions!) messages to SQS and they get… queued! They are stored until a script consumes them.

To consume messages, we can connect SQS to Lambda. The best part: we don't run a daemon that polls the queue. If the queue is empty, our code does not run! (and we don't pay) When a message arrives, Lambda will boot a Linux environment with our code, and our code will be invoked!

What if there are many messages? That's where Lambda shines: it will boot workers on the fly (up to 1000 by default). That means ultra-fast queue processing. And each worker is isolated from the others (separate Linux environments), meaning no concurrency or scaling to deal with.

What if we don't want to overload a downstream service (database, API, etc.)? We can limit the number of Lambda instances. SQS will act as a buffer.
Here is an example with a maximum concurrency of 10 Lambda instances:

We can also process messages in batch: e.g. process 10 messages per Lambda execution (instead of 1). That can help process messages faster, or reduce the load downstream (e.g. by batching SQL inserts). However it makes handling errors harder!
Why? I'll demo errors next time ;)

If you have any questions, feel free to ask! I will collect all the questions about SQS and Lambda to answer them with visual illustrations next time.
If you have feedback for the simulator, please share :)

ICYMI: the thread continues here with SQS error handling: https://twitter.com/matthieunapoli/status/1246069088306094086