Logs Insights Querying

## What Logs Insights actually is CloudWatch Logs Insights is a query engine that runs over log events already stored in CloudWatch Logs. You do not create an index or a table first: you pick one or more log groups and a time range in the console (or pass them to the `StartQuery` API), and the engine reads the events in that selection and runs your query over them. The query language is a **pipeline**, closer to a shell one-liner than to SQL. Commands are separated by `|` and evaluated left to right; each one receives the rows the previous one emitted. ## The system fields that are always there Even for a completely unstructured log line, every event exposes a set of `@`-prefixed fields: - `@timestamp` — the event time as reported by whatever put the log there (the agent, the SDK, the Lambda service). This is what the console's histogram and `sort @timestamp` use. - `@ingestionTime` — when CloudWatch Logs accepted the event. It differs from `@timestamp` whenever the producer buffered, retried, or had clock skew, which is exactly why the two fields exist separately. - `@message` — the raw text of the log event, unparsed. - `@logStream` — the stream inside the log group. A stream is usually one instance, task or Lambda execution environment, not one service. - `@log` — an identifier of the form `accountId:logGroupName`, which is how you tell events apart when a query spans several log groups. - `@ptr` — an opaque pointer to the stored event, used by the console's "view surrounding events" and by the `GetLogRecord` API. If the log line is JSON, Logs Insights also auto-discovers its keys as queryable fields, flattening nesting into dot notation, so `{"http":{"status":500}}` becomes the field `http.status`. There is a cap on how many fields it will discover per event, so extremely wide JSON can lose its tail. ## The commands you will actually use ``` fields @timestamp, @message, @logStream | filter @message like /ERROR/ | sort @timestamp desc | limit 20 ``` - `fields` adds columns to the output and can compute new ones (`fields @timestamp, latency / 1000 as seconds`). `display` is its blunter cousin: it *replaces* the output columns with exactly what you name, and only the last `display` in the query takes effect. - `filter` keeps events matching a boolean expression. For substring or regex matching on text use `like` (`filter @message like /ERROR/` for a regex, `filter @message like "ERROR"` for a literal substring), and `not like` for the negation. For discovered or parsed fields you can use ordinary comparisons: `filter http.status >= 500 and path = "/checkout"`. - `sort` orders rows, `asc` or `desc`. - `limit` caps the number of rows returned. - `dedup` collapses rows that share the values of the fields you name, keeping the first one — handy for "show me one example per error type". - `parse` and `stats` extract and aggregate; they are the step up from simple searching. ## Order matters, and `stats` is a wall Because it is a pipeline, position changes meaning. `sort` before `limit` gives you the top 20 by that ordering; `limit` before `sort` gives you an arbitrary 20 events that then get sorted among themselves. More importantly, `stats` **collapses** the event stream into aggregate rows: after a `stats`, the only fields that still exist are the aggregates you computed and the keys you grouped by. Referring to `@message` after a `stats` is a common beginner error. ## What `limit` does not do `limit` is an output cap, not a cost or scan control. The engine still reads the log data in the selected log groups and time range before it can decide which 20 rows to hand back. Narrowing the *time range* and the *log group selection* is what makes a query cheaper and faster; `limit` only makes the result readable. As of 2026 a single query also returns at most 10,000 log events regardless of what you ask for, so a query that is really an export needs a different tool. ## Running it outside the console Queries are asynchronous over the API: `StartQuery` returns a `queryId`, and `GetQueryResults` polls it, returning a status (`Scheduled`, `Running`, `Complete`, `Failed`) plus a `statistics` block with `recordsMatched`, `recordsScanned` and `bytesScanned`. Those three numbers are the honest feedback loop on whether your query was well targeted.