We started using Amazon Web Services EC2 Container Service recently at my day job, Engaged Nation. Docker is somewhat new to me in a production sense and AWS ECS can look incredibly daunting when looking at all the controls. I ran into this situation where my AWS ECS Agent AMI was outdated and it was warning me that I needed to update. If you are using AWS manually (no provisioning tools) then the easiest way is to goto the ECS Instance Tab, click on the instance in question, then click the “Update Agent” Ok no problem I thought, so I go an update my Terraform recipes and re-provision my AWS environment with the new updated AMI in us-west-2. Should all be good now right? Wrong

At least the way I have things setup, the old EC2 Instance just hung around with live containers running in it. I was assuming (first mistake) that the ECS Agent would somehow magically take care of this. After reading about the Agent, it really seems to be really isolated to only have control over its own server. Using cluster querying and other magic, when an EC2 instance in the ECS cluster spins up, it sees how many containers it should launch within itself. I could be kinda wrong about some of this so go read their docs for more details but what this means is you gotta do this more from the AWS console rather than this “automatically” happening which I guess you could maybe write some script to do but thats just crazy talk.

Step 1 : Drain the connections on the EC2 Instance

First thing, head over to your ECS cluster with the outdated AMI

Select the server you want to drain and click “Actions” then click “Drain Instances”

This will route any connections to a new EC2 instance so you can safely kill this old instance.

Step 2 : Detach the instance from the ALB

Now that your containers are not taking anymore connections, head over the the EC2 dashboard -> Auto Scaling Groups.

Once you are there, select the cluster in question, then click the “Instance” tab, check off the instance in question and click “Actions”, click “Detach”. After it is detached, you should see your cluster spin another instance up and recover.

Step 3 : Terminate the EC2 Instance.

“Astalavista, baby…”

Finally, last but not least (save money!!!) and lets go kill the instance. Head over to the EC2 Dashboard, check off the instance and click “Actions” -> Instance State -> Terminate

So in recap:

Steps : Drain->Detach->Terminate

Dashboards : ECS->ALB->EC2

Let me know if you guys know of a better way or this can be improved. Love to see an automated way to do this too.