A scale-in event on an EC2 Auto Scaling group terminates workers that are still processing long-running jobs. How do Auto Scaling lifecycle hooks let you drain that work before the instance goes away?

## The default is abrupt When Auto Scaling decides to remove an instance — a scale-in, a health-check replacement, an instance refresh, a maximum-instance-lifetime rotation — the default path is straight to `Terminating` and then gone. For a stateless HTTP server that is often survivable, because the load balancer drains connections. For a worker that has pulled a message off a queue and is thirty seconds into a two-minute job, it means the job dies mid-flight and reappears later as a duplicate or a stuck record. A lifecycle hook is the extension point that turns that abrupt transition into a negotiated one. ## The mechanics A hook is attached to one of two transitions: - `autoscaling:EC2_INSTANCE_LAUNCHING` — the instance pauses in `Pending:Wait` before being put into service. - `autoscaling:EC2_INSTANCE_TERMINATING` — the instance pauses in `Terminating:Wait` before being destroyed. ```bash aws autoscaling put-lifecycle-hook \ --auto-scaling-group-name worker-asg \ --lifecycle-hook-name drain-jobs \ --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \ --heartbeat-timeout 600 \ --default-result CONTINUE ``` When the hook fires, Auto Scaling publishes a lifecycle action notification. The modern route is an EventBridge rule matching the Auto Scaling lifecycle event and invoking something — commonly a Lambda function or an SSM automation — but the hook can also be configured with a notification target ARN pointing at an SNS topic or an SQS queue, along with a role that grants publish access. Alternatively, an agent already running on the instance can watch for its own impending termination and act locally, which is usually the simplest design for a worker that owns its own drain logic. Every notification carries a `LifecycleActionToken` along with the group name, hook name and instance id. That token, or the instance id, is what you pass back: ```bash # Still working — extend the wait aws autoscaling record-lifecycle-action-heartbeat \ --auto-scaling-group-name worker-asg \ --lifecycle-hook-name drain-jobs \ --instance-id i-0123456789abcdef0 # Done — release the instance now aws autoscaling complete-lifecycle-action \ --auto-scaling-group-name worker-asg \ --lifecycle-hook-name drain-jobs \ --instance-id i-0123456789abcdef0 \ --lifecycle-action-result CONTINUE ``` ## Timeouts, heartbeats and the ceiling `HeartbeatTimeout` is how long Auto Scaling waits before giving up on you; it defaults to one hour. Each heartbeat resets the clock, so a drain of unknown length is handled by a loop that heartbeats on a fixed interval while work remains. There is nevertheless an absolute ceiling on how long an instance may sit in a wait state, so a hook is a drain window, never a way to pin an instance indefinitely — for that you want scale-in protection instead. `DefaultResult` decides what happens when the timeout expires with no response. For a *terminating* hook the instance is terminated either way; `ABANDON` just skips any remaining actions and gets on with it, while `CONTINUE` proceeds through the normal path. For a *launching* hook the difference is much sharper: `ABANDON` means the new instance is considered failed and is terminated rather than put into service, which is exactly what you want when the warm-up step failed. Always call `CompleteLifecycleAction` when the work is done. If you only rely on the timeout, every scale-in costs you the full window in instance-hours and slows every rolling operation the group performs. ## Designing the drain itself A good terminating hook is short, ordered and idempotent: 1. **Stop taking new work** — deregister the consumer, stop the poll loop. 2. **Finish or hand back what is held** — either complete in-flight jobs, or return them so another worker picks them up promptly instead of waiting for a visibility timeout to lapse. 3. **Flush** — ship buffered logs, metrics and traces, since the instance and its ephemeral storage are about to vanish. 4. **Complete the action.** The drain must also survive being interrupted, because the ceiling exists and hardware fails. Design so that a worker dying mid-drain is recoverable rather than merely unlikely. ## What hooks are not They do not stop scale-in from choosing this instance — that is what instance scale-in protection is for, and what the group's termination policies influence. They also do not handle HTTP connection draining for a load-balanced tier; the target group's deregistration behaviour covers that, and the two are complementary. Hooks are for the work your application knows about and the platform does not. One more place hooks quietly matter: warm pools. Instances entering a warm pool pass through the launching transition too, so a launch hook that prepares an instance runs when the instance is prepared, not when it is finally handed to the group.