The pipe query language
The full grammar of the new KQL-shaped Query tab: every stage, every operator and function, the field aliases, and the safety limits — a second engine, separate from classic search.
There are now two ways to query your events, and they are genuinely two different things, not two names for one mechanism. This article documents the new one: a pipe-shaped query language and its own compiler, sitting alongside — not replacing — the field-based DSL described in Searching your logs.
The grammar
<source> [| <stage>]*A query is a source followed by zero or more pipe-separated stages, each one operating on the output of the one before it. Today the source is always siem_log_events — your normalised event store, the same one classic search reads. Left-to-right pipe semantics are real: each stage is compiled as a fresh SQL layer wrapping the previous one, so the query behaves the way it reads, top to bottom, rather than being reassembled into one flattened statement behind the scenes.
Every stage
| Stage | Shape | What it does |
|---|---|---|
where | where <expr> | Filters rows. The expression language is described below. |
extend | extend <name>=<expr>[, ...] | Adds computed columns without dropping the existing ones. |
parse | parse <field> with regex "<pattern>" as (<name>, ...) | Extracts named capture groups out of a text field — typically raw_log — into new columns. |
summarize | summarize [<alias>=]<func>(<args>) [, ...] [by [<alias>=]<expr> [, ...]] | Aggregates. Replaces the available column set with whatever you named — see the note on symbol tracking below. |
join | join kind=inner|left|right (<subquery>) on <key>[==<rightKey>] | Joins the pipeline so far against an independently compiled subquery. See the note on joins below. |
sort / order | sort|order [by] <expr> [asc|desc] [, ...] | Orders rows. |
top | top <N> by <expr> [asc|desc] | Sort and limit in one stage. |
project | project [<alias>=]<expr> [, ...] | Chooses and renames the output columns. Like summarize, it replaces the available column set rather than adding to it. |
take / limit | take|limit <N> | Caps the row count, subject to the safety caps below. |
On join: the subquery compiles independently, with its own base layer and its own bounds, then the right-hand side is packed into a single joined column rather than requiring you to hand-alias every column to avoid a collision. Only one source (siem_log_events) is registered today, so in practice a join is a self-join — correlating, for example, authentication and process category rows on a shared user_id.
The operator and function vocabulary
| Group | Members |
|---|---|
| Comparison and text | has, !has, contains, !contains, startswith, endswith, matches regex, in(...), between(lo .. hi) |
| Scalar functions | ago(), now(), bin(), tostring(), toint(), todatetime(), case() |
Aggregates (inside summarize) | count(), sum(), avg(), min(), max(), dcount(), percentile(), make_set(), make_list(), arg_max() |
String, number, boolean and duration literals (5m, 1h, 7d, 30s) are all recognised, and and / or / not combine expressions with real precedence. Every literal value is bound as a query parameter rather than being interpolated into the generated SQL text — that is enforced by construction, not by a filter, and it is what makes the language safe to expose directly to you.
Field aliases
The pipe engine's field whitelist is the same fifteen columns the classic DSL already enforces — the two are kept in lockstep by construction, since the pipeline package sits underneath the search package rather than the other way round. On top of the full field names, the pipe language additionally accepts a set of short aliases for the ones you will type most often: time, src_ip, dst_ip, dest, user, asset, host, module, class, sev, category. Either the full name or the alias resolves to the same underlying column and the same value.
Safety limits
| Limit | Value | What happens when you hit it |
|---|---|---|
| Row cap, synchronous run | 1,000 rows | take/limit is capped at this value on POST .../siem-query/run. |
| Row cap, async job | 100,000 rows | The higher cap applies to a background job — see Async search jobs and the hunting library. |
| Time-window cap on a raw scan | 90 days | A query with no summarize stage — a raw row scan — spanning more than 90 days is refused outright with a coded error (search.window_too_wide), rather than silently clamped or left to time out. |
raw_log predicates | 7 days | A condition against raw_log additionally inherits the classic DSL's own 7-day cap, because it is the one column with no index. |
| Planner-estimate rejection | 5,000,000 estimated rows | Before running your query, the compiler asks Postgres's own planner (EXPLAIN, cost only, no table access) how many rows it expects to scan, and refuses execution if the estimate is too high — the backstop for a query whose window and field predicates should have made this unnecessary but did not. |
The Query tab in the console
Find it at SIEM → Event Search, next to the existing Basic tab. It is a plain textarea — there is no full syntax highlighting or tokenised editor, deliberately scoped out for this wave — with click-to-insert field chips that come from the schema endpoint (GET .../siem-query/schema), so you can build a query without memorising the alias table above. Results render in the same table component the rest of the console uses, with columns derived from whatever keys your query's rows actually carry — a project- or summarize-shaped query produces a different column set than a raw scan, so a fixed column layout would be wrong for most queries and none is imposed.
What is not in the console yet: submitting a query as a background job, polling it, and paging its results all exist as a tested API with no console surface — see Async search jobs and the hunting library for the raw endpoints.
What to read next
- Searching your logs — the classic DSL, still the default Basic tab and still the production search endpoint.
- Async search jobs and the hunting library — running a pipe query as a background job, and the 40 built-in hunts.
- Scheduled detection rules and templates — a different consumer of the same underlying event store, on a timer rather than ad hoc.