Thread by @julian_wood, Such amazing knowledge in the Amazon Builder's Library. Just seen "Avoiding fallback [...]

Such amazing knowledge in the Amazon Builder's Library. Just seen "Avoiding fallback in distributed systems". https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/?did=ba_card&trk=ba_card > super useful info, even if you're designing an on-prem active/passive service. How should you think about retry/failover and fallback?

Building services that behave predictably during failures by avoiding fallback logic.

Also "Static stability using Availability Zones". How to load apps across multiple data centers (AZs in AWS speak). Meaning, when do you use active/active or active/passive, and how do you build multiple active/active services relying on each other. https://aws.amazon.com/builders-library/static-stability-using-availability-zones/?did=ba_card&trk=ba_card

Architecting to use multiple availability zones for high availability and ensuring systems are statically stable.

Then, failures always happen, how do you manage stuff not talking. "Timeouts, retries, and backoff with jitter". from @MarcJBrooker shows how we think about it, how we are cautious with retries. https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/?did=ba_card&trk=ba_card

Building resilient systems and dealing with failures by using timeouts, retries, and backoff with jitter.

A great way to connect separate services is with a queue for messaging. What happens when queues back up? "Avoiding insurmountable queue backlogs" from @dyanacek explains how we use prioritization, sidelining, and backpressure to minimize queue backlogs. https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/?did=ba_card&trk=ba_card

Prioritizing draining important workloads from queue backlogs quickly, and avoid backlogs in the first place.

Then @dyanacek has more with "Using load shedding to avoid overload". Avoid taking on more work when you know you can't cope. https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload/?did=ba_card&trk=ba_card

Strategies for maintaining predictable, consistent performance in the face of overload.

You can imagine AWS rolls out a heck of a lot of software changes globally to MASSIVE distributed systems. How do we rollback when things go wrong? "Ensuring rollback safety during deployments" explains how we do it: https://aws.amazon.com/builders-library/ensuring-rollback-safety-during-deployments/?did=ba_card&trk=ba_card.

Upgrading live systems with backwards compatibility, rollback, and state in mind so that changes are non-disruptive to customers.

Pair this with "Going faster with continuous delivery". How we reduce the risk that a defect will reach customers and how we drastically up the speed of execution. " https://aws.amazon.com/builders-library/going-faster-with-continuous-delivery/"

How Amazon automates software delivery including automated testing and deployments.

and diving even deeper into our deployment process with "Automating safe, hands-off deployments". @clare_liguori explains how we do staggered wave deployments to internal service "cells" across 24 Regions & 76 Availability Zones https://aws.amazon.com/builders-library/automating-safe-hands-off-deployments/

Strategies for continuously deploying to production while balancing safety and speed.

You can follow @julian_wood.

Tip: mention @twtextapp on a Twitter thread with the keyword “unroll” to get a link to it.

Latest Threads Unrolled: