Read on Twitter
Last week, CodeDeploy & CloudFormation added support for doing ECS blue-green / canary deployments as part of CFN stack updates, yay!
https://aws.amazon.com/about-aws/whats-new/2020/05/aws-cloudformation-now-supports-blue-green-deployments-for-amazon-ecs/
The way it works is pretty cool, so I'll dig into what happens during a stack update in this thread
https://aws.amazon.com/about-aws/whats-new/2020/05/aws-cloudformation-now-supports-blue-green-deployments-for-amazon-ecs/
The way it works is pretty cool, so I'll dig into what happens during a stack update in this thread
To switch to blue-green deployments in CFN, you need to make two changes to your template:
1) Use an ECS task set resource instead of specifying all the config in the ECS service resource. This lets the deployment create "blue" and "green" task sets in your ECS service.
1) Use an ECS task set resource instead of specifying all the config in the ECS service resource. This lets the deployment create "blue" and "green" task sets in your ECS service.
2) Add the CodeDeploy blue-green "Transform" and "Hook" configuration. This lets CodeDeploy orchestrate a blue-green deployment and canary traffic shifting during a CFN stack update, for example when you deploy a template with a new container image to your stack.
The hook splits the deployment into five steps that CFN will perform one at a time. You can think of this like a sequence of 5 templates progressively deployed as part of a single stack update, ending in the template that you provided for the deployment containing your new image.
Let's walk through an example ECS blue-green deployment of a new image in CloudFormation to see this in action!
Prior to the deployment, my ECS service looks something like this: A blue task set behind a blue target group receives all test and prod traffic from the load balancer
Prior to the deployment, my ECS service looks something like this: A blue task set behind a blue target group receives all test and prod traffic from the load balancer
I update the ECS task definition in my CloudFormation template with a new container image tag and deploy the updated template to my CloudFormation stack. My blue-green deployment is now in progress!
I only edited one line in my template, and I didn't have to manage switching anything from blue to green. The CodeDeploy blue-green transform takes care of updating my template to flip task set, task def, and target group references from blue to green when the deployment starts.
The first step orchestrated by the CodeDeploy hook is for CFN to create the "green" task def and "green" task set with the new image. Think of this as CodeDeploy adding new resources to your stack's current template (partial diff example below), and then CFN applying that change
The green task set is registered to the green target group, but doesn't receive any traffic yet. The "blue" task set is unchanged, running the old image and receiving all the traffic.
The next step is to change the test listener resource's configuration to point to the green target group, instead of the blue target group.
The green task set now gets traffic on a "test port", while the blue task set still gets production traffic. You can configure the CodeDeploy hook with a Lambda function that runs your integration tests against the test port to ensure the green task set works as expected.
The canary phase of the deployment now starts by shifting 20% of production traffic to the green target group. CodeDeploy then pauses the stack update to allow time (for example, 15 min) to monitor for production impact and rollback while the impact is limited to 20% of requests.
To take full advantage of CodeDeploy "wait time" after the canary traffic shift, the CloudFormation stack should have rollback triggers configured so that CFN monitors an alarm during the deployment and rolls back the stack update if there is any impact.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-rollback-triggers.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-rollback-triggers.html
Once the canary phase is finished, the next step shifts the rest of production traffic over to the green task set.
The blue task set is now standing idle, ready to quickly start taking traffic again in case of rollback. The CodeDeploy hook can be configured to again pause the stack update to allow time to monitor production impact and rollback.
The final step in the deployment brings the stack up-to-date with the template I deployed (post-transform). The "blue" task set and task def with the old image are removed from the stack.
The deployment is now complete! My new image is running in a green task set behind a green target group, receiving both test and production traffic.
The next time I deploy a new image to my CloudFormation stack, CodeDeploy will take care of orchestrating going from my currently-running "green" task set to a new "blue" task set running the new image. I only need to change one line in my template to make that happen!
To get started with blue-green deployments in CloudFormation, the official docs have a sample template. I also put one up on GitHub showing the 20% canary deployment example I used in this thread:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/blue-green.html#blue-green-template-example
https://github.com/aws-samples/aws-reinvent-2019-trivia-game/blob/master/trivia-backend/infra/codedeploy-blue-green-cfn/template.yaml
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/blue-green.html#blue-green-template-example
https://github.com/aws-samples/aws-reinvent-2019-trivia-game/blob/master/trivia-backend/infra/codedeploy-blue-green-cfn/template.yaml
And finally, if you're an AWS CDK user, subscribe to this GitHub issue to get updates on the work we have planned to add CodeDeploy blue-green deployment support for ECS to the CDK! https://github.com/aws/aws-cdk/issues/1559
