An EC2 security group's rules are not what your team expected. What do AWS CloudTrail, AWS Config and Amazon CloudWatch Logs each record, and which question is each one the right tool for?
answer
- three services, three different units
- calls, state, and consequence
- one has a change timeline
- one has request parameters
- one only has your log lines
basics
~20 sCloudTrail records the API calls made against AWS itself. AWS Config records the resulting configuration state of a resource over time. CloudWatch Logs stores log lines emitted by applications and services. Use CloudTrail for the call, Config for the state, CloudWatch Logs for the effect on your workload.
solid answer
~40 sThey answer three different questions. **CloudTrail** is the record of API activity: it captured the `AuthorizeSecurityGroupIngress` call, the principal and session that made it, the source IP, the request parameters and whether it succeeded — but not what the group looked like afterwards. **AWS Config** records configuration items: point-in-time snapshots of the resource, a change timeline, its relationships to other resources, and rules that evaluate compliance — so it answers "what were the rules at 03:00, and what exactly changed between then and now". **CloudWatch Logs** holds log lines your applications and many AWS services write; it tells you what the workload did as a consequence — the connections that started failing. In practice you correlate: Config narrows the change window, CloudTrail explains the call in that window, CloudWatch Logs shows the impact.
go deeper
Be able to say in one line that CloudTrail records who called which AWS API, while CloudWatch Logs holds application and service log lines — and not to use the two names interchangeably.
Explain what a Config configuration item is and why a change timeline answers a question CloudTrail structurally cannot, plus what delivering a trail into CloudWatch Logs actually buys.
Show the correlation workflow under time pressure: narrow the window with Config, explain the call with CloudTrail, confirm impact in the workload's logs, and know the gaps in each.
Decide which of the three is recorded where across the estate, since Config's per-item pricing and CloudWatch ingestion make full coverage a budget choice, and state which resource types are worth a timeline.
## Three different nouns The cleanest way to hold these apart is to notice they record three different kinds of thing. **CloudTrail records calls.** Its unit is an API request against an AWS service endpoint. Each record carries the event source and name, the time, the identity that made the call (including the assumed-role session), the source IP and user agent, the request parameters, sometimes response elements, and the error code if it failed. Note what is *not* there: the resulting state of the resource. CloudTrail tells you `AuthorizeSecurityGroupIngress` was called with a particular CIDR; it does not hand you the security group as it stands now. **AWS Config records state.** Its unit is a configuration item — a point-in-time snapshot of one resource's configuration, plus its relationships to other resources (this security group is attached to those instances). Config maintains a timeline of those items, so it can answer "what did this resource look like at 03:00?" and "what changed between these two versions?" as a structured diff. It also evaluates Config rules against those items, so it can tell you a resource is non-compliant with a standard you defined, and it delivers configuration history and snapshots to S3. **CloudWatch Logs stores log lines.** Its unit is a log event in a log stream inside a log group. Application stdout shipped by an agent, Lambda function logs, VPC Flow Logs, and many AWS services' logs land here. It is where you see the *consequence* of a change — the connection timeouts that began at 03:04. ## Working the security-group example 1. **Config** gives you the change timeline for the group. You see two configuration items minutes apart and the diff between them: an ingress rule added, another removed. Now you have a tight time window and know precisely what changed. 2. **CloudTrail** covers that window. You find the corresponding EC2 API calls and can read the request parameters, the principal and session, the source IP and user agent, and whether other calls in the same session failed. If the call went through the console, that shows in the user agent; the console still calls the same API underneath, so console-driven changes are not invisible. 3. **CloudWatch Logs** shows what it did to the workload — when the application's connection errors started and stopped, which tells you whether this change is even the one that hurt. CloudTrail can also *deliver into* CloudWatch Logs. That is a delivery option on a trail, not a merging of the two services: it puts CloudTrail events into a log group so that metric filters and alarms can be built over API activity, at the cost of CloudWatch Logs ingestion and storage on top of the trail. ## Where each one falls short **CloudTrail** has no notion of resulting state, so reconstructing "what did this look like last Tuesday" from calls alone means replaying every mutation — painful and error-prone. Its console Event history reaches back only 90 days, and data-plane activity is recorded only if you opted into data events beforehand. **Config** records only the resource types it supports and only in regions where the recorder is enabled; if it was not turned on before the change, there is no timeline to read. It is also priced per configuration item recorded and per rule evaluation, so "record everything everywhere" is a cost decision, not a free default. And it deliberately does not tell you *who* — the identity lives in CloudTrail. **CloudWatch Logs** knows nothing about AWS API activity unless something puts it there, and its cost is driven by ingestion volume, so it is the wrong place to dump everything on the off-chance. ## The interview answer The crisp formulation is: CloudTrail is the verb, Config is the noun, CloudWatch Logs is the consequence. Candidates who blur CloudTrail and CloudWatch into "the AWS logging service" get caught immediately by a follow-up like "so where do you see what the security group looked like before the change?" — a question CloudTrail structurally cannot answer and Config answers directly.
- Your trail already delivers to CloudWatch Logs. Does that make AWS Config redundant?No. Delivering a trail to CloudWatch Logs just puts API-call records into a log group so you can build metric filters and alarms on them. It still contains calls, not resource state — there is no configuration timeline, no structured diff between versions, and no relationship graph. If you need to know what a resource looked like at a point in time, that is Config's job regardless of where CloudTrail events are stored.
- A change appears in AWS Config with no corresponding CloudTrail event. What are the plausible explanations?Most often the search was wrong: the call landed in another region, or in us-east-1 because the service is global, or outside the 90-day Event history window when no trail existed. It can also be an AWS-initiated or service-linked action recorded differently, or a change made through a service that acts on your behalf. Widen the region and time window before concluding anything is missing.
- Why is AWS Config not simply enabled for every resource type in every account?Because it bills per configuration item recorded and per rule evaluation, and a chatty resource type in a large estate produces a lot of items. Teams typically record the resource types that matter for compliance and change forensics, enable the recorder in the regions actually in use, and accept that unrecorded types have no timeline.
saying these in an interview costs you the question
- Calls CloudTrail and CloudWatch the same service
- Expects CloudTrail to show prior resource state
- Thinks console changes bypass CloudTrail entirely
- Assumes Config records every resource type by default
- Believes CloudWatch Logs captures AWS API calls automatically