Collecting Cloudflare logs

Cloudflare sits in front of your public websites and sees every request to them, including the ones its web application firewall turned away. That is a useful stream to have next to your identity and endpoint telemetry, and cloudflare is an accepted parser kind on an ingest token.

It is still a shallow parser — it reads a dozen or so fields out of the hundred-odd Cloudflare can emit — but the fields it reads now reach useful places. This article says exactly which, so you can decide whether the source is worth connecting for what you want to do with it before you spend an afternoon on Logpush.

Getting the logs in

There is no Cloudflare connector and nothing polls the Cloudflare API. As with Okta, you forward.

1. 1

   Issue an ingest token with `parser_kind` set to `cloudflare`

   Requires siem:sources.write. One token for this source only — a token carries exactly one parser, and sharing it with another format runs the wrong parser over one of them. See Ingest tokens.

2. 2

   Send newline-delimited JSON to the collector

   POST /api/v1/siem/hec with Authorization: Bearer <token> and Content-Type: application/json. One JSON object per request record. A top-level JSON array is also accepted and split into one record per element.

3. 3

   Choose the HTTP requests dataset

   The parser is written against Cloudflare's HTTP request log. A firewall-events dataset or a Workers trace has different field names, and none of the fields listed below will be found in it — the record will still be stored, with nothing extracted.

4. 4

   Either timestamp format is fine

   RFC 3339 or Unix epoch, whichever your Logpush job is set to. Both are accepted; see the note below if you remember this being a problem.

The fields, and where each goes

What a Cloudflare event looks like in search

Every one of them is category = network. The action is derived from the response status, unless the web application firewall stated a verdict — in which case the verdict wins, because it is the more specific thing the edge actually said.

# Every error response the edge returned
category = network AND outcome = failure

# One client address, everything it did
source_ip = "203.0.113.5"

# Everything the WAF blocked — now a real count
action = waf_block

# Everything the WAF acted on at all
category = network AND severity >= 4

# One site behind Cloudflare
asset_id = "shop.example.com"

# Blocks against one site
asset_id = "shop.example.com" AND action = waf_block

# One URL path, through the raw record
raw_log contains "/wp-login.php"

# Requests from one country — no searchable column, so the raw record
raw_log contains "\"ClientCountry\":\"ru\""

Volume is the thing to plan for

A public website produces far more log lines than your identity provider and your endpoints combined, and most of them are of no security interest whatsoever. Three practical consequences.

• The ingest rate limit is 1,000 events per second per token, with a burst of 2,000. A busy site behind Cloudflare can exceed that. A throttled request is currently answered as 401 invalid token rather than a 429, which is a known defect documented in Ingest tokens — so a Cloudflare shipper reporting sudden authentication failures is very likely being rate limited rather than misconfigured.
• Retention is shared. Cloudflare volume competes with everything else you collect for the window your plan entitles you to. See Log retention.
• Filter at the Logpush end, not here. There is no ingest-side sampling or drop rule. Send the requests you care about — errors, or one hostname — rather than the whole stream, and reconsider only if the narrower set proves useful.

Replayed records duplicate, as with every source that carries no identifier of its own. Cloudflare's RayID is not read and is not used for deduplication, so a Logpush job that re-sends a batch produces a second copy of every record in it.

What to read next

1. Sending events with the HTTP Event Collector — the endpoint and the exact request shape.
2. What a firewall log actually becomes — the same analysis for Cisco ASA, FortiGate and Palo Alto, with the same conclusions about raw_log.
3. Searching your logs — the fifteen fields, and the limits of contains.