Skip to main content

A log source is ingesting nothing

Token, endpoint, payload, tenant and rate-limit refusals when events are not arriving — plus why events can arrive perfectly and still raise no alerts.

Written for whoever runs IT9 min readUpdated

A collector is running, you believe it is sending, and nothing is showing up. This article works through the refusals in the order they occur — the token, the endpoint, the payload, the tenant, the limits — and gives the exact response for each so you can match what your collector logged.

Then there is a second half, and for many readers it is the part that actually answers the question. Events can be arriving perfectly and still produce no alerts. If your real question is why am I getting no alerts, skip to that section before you touch your token.

What success looks like

The two push endpoints and their success responses. Neither uses a JSON error envelope.
ReceiverPathSuccess
HTTP Event CollectorPOST /api/v1/siem/hec202, body {"accepted": <n>, "text": "Success"}
OTLPPOST /api/v1/siem/otlp/v1/logs200, body {"partialSuccess": {}}

Three details trip people up. HEC has one path — there is no separate batch endpoint, and batching is a matter of payload shape on the same URL. OTLP answers 200, not 202. And the accepted header forms are Authorization: Splunk <token> (so existing Splunk collectors work unchanged) or Authorization: Bearer <token>. There is no X-Qorionix-Ingest-Token header; if your collector sends one, that is the whole bug.

The token was refused

Token refusals
401 missing bearer token
401 invalid token

missing bearer token means no Authorization header arrived, or it used a scheme other than Splunk or Bearer. invalid token means the token was presented and did not resolve — unknown, revoked, or rate limited. Unknown and revoked answer identically so the response does not tell an attacker which tokens exist.

  • Tokens begin qxi_ if minted recently, or zqi_ if minted before the rename. Both are valid indefinitely and the prefix carries no authority — resolution is on a hash of the whole string. If yours begins with neither, it is not an ingest token: it may be an enrolment token (zqe_), a provisioning token (zqscim_) or an API key, which has no prefix at all.
  • There is no expiry. A token is valid until revoked, so one that stopped working did not time out.
  • Revocation takes effect on the next request. There is no cache to wait out.
  • There is no rotate operation. Issue a new token, cut the collector over, delete the old id. The list shows active and revoked rows together during a cutover for that reason.

The plaintext is shown once, at issue, and cannot be recovered — if it is lost, revoke and issue a new one. When issuing, the refusals are {"error":"organization_id and name are required"} and {"error":"unsupported parser: <value>"} at 422, and revoking an unknown or already-revoked id gives {"error":"ingest token not found"} at 404. See Ingest tokens.

The rate limit, and why it lies

Each token is limited to 1,000 events per second steady with a burst allowance of 2,000, counted per token and per process. Cross it and the resolver refuses — and, as above, the refusal is reported as an authentication failure with no Retry-After, so a well-behaved collector cannot back off intelligently.

Tell the two apart by pattern rather than by message. An invalid token fails every request. A rate limit fails only the requests above the line, so you see a partial trickle that correlates with load. Intermittent 401s under load with steady success when quiet means you are being throttled and your token is fine.

The wrong endpoint or method

Both receivers accept POST only and answer 405 method not allowed to anything else, including the GET people use to test a URL by hand. A 405 from a browser is not evidence of a problem.

The OTLP path is the one most often written wrongly. The mounted path is /api/v1/siem/otlp/v1/logs, with the OTLP-conventional /v1/logs on the end of the Qorionix prefix — an exporter configured with a base URL that already ends in /v1/logs produces a duplicate and a 404. OTLP also accepts HTTP with a JSON payload only: gRPC is not offered and protobuf over HTTP is not accepted.

The payload was rejected

Payload refusals, by receiver
400  unsupported Content-Type: <value>        (HEC)
400  empty payload                            (HEC)
400  <the raw JSON decoder message>           (HEC)
400  read failed                              (OTLP)
400  empty OTLP payload                       (OTLP)
503  queue unavailable                        (HEC)
503  ingest sink unavailable                  (HEC)

HEC accepts three shapes when the content type is application/json or absent: a JSON array, a concatenated stream of objects, or a single object. A Splunk envelope is unwrapped, so {"event": {...}} yields the inner object. Any text/* type takes newline-delimited raw lines. Anything else gives the unsupported Content-Type refusal, usually a collector defaulting to a form encoding.

Welcome the two 503s rather than resenting them: they mean the receiver could not hand your batch on and said so instead of claiming success. That is a deliberate repair — the previous code discarded the failure and still answered 202, which is how ingestion works stayed believed for as long as it did. Retry; your collector's buffer is doing its job.

Size and batch limits

  • HEC has no explicit maximum body size. A large single POST is not refused on size grounds.
  • A single line may be up to 1 MiB on HEC's text/* path and on syslog over TCP and TLS. A longer line is not accepted.
  • Syslog over UDP reads 64 KiB per packet, with a two-second read deadline. Framing is newline-delimited only — RFC 6587 octet counting is not supported, so an octet-framed appliance produces garbage rather than events.
  • On the service path, an empty batch gives events array must not be empty, and a single event needs both fields: source_id and raw_log are required.
  • Registering a log source gives name and source_type are required, then source_type must be one of: syslog, cloud_api, agent, webhook, and a log source named "<name>" already exists in this organization at 409.

Tenant resolution, which only affects syslog

Push receivers take the tenant from the token. Syslog cannot: a syslog line carries no credential, so the tenant is resolved from the sending appliance's IP address, and a line whose peer address belongs to no tenant is dropped rather than guessed at. Dropping is right — an ambiguous resolution on an ingest path is a cross-tenant leak waiting to happen — but it means a correctly configured appliance sending to a correctly bound listener can still produce total silence.

siem ingest: syslog line dropped — no tenant owns this peer address (add a siem_syslog_sources row)
Worker log, rate-limited to once per minute per peer

Matching is longest-prefix, so a single-address carve-out beats the enclosing subnet, and ranges may not overlap between tenants: syslog peer_cidr <range> overlaps a range owned by organization <id>.

Events are arriving and no alerts appear

This is the most important section here and it is not a configuration problem. There are two separate write paths into the event store, and only one of them runs detections.

The service path — detections fire
The authenticated, organisation-scoped ingest endpoints. An event written here is stored and then evaluated, and a rule that fires becomes an alert immediately.
The receiver path — detections do not fire
HEC, OTLP, syslog, Microsoft 365 and Entra, AWS CloudTrail, Google Workspace, and Windows Event Log. Everything arriving this way is normalised, stored, indexed and fully searchable — and raises no alert. Nothing re-reads the stored events later and evaluates them.

Your logs are not wasted: they are searchable, retained, and the evidence an auditor asks for. What you must not do is read no alerts as nothing happening. Until this closes, alerting comes from the endpoint sensor and the posture assessment, and log-based detection is a search-and-review exercise — see Searching your logs, and build saved searches for the things you would most want to be told about.

Connectors that will not produce anything yet

  • Kafka and Kinesis are accepted as receiver types and are stubbed. No collector runs for them.
  • CEF and ECS are accepted as parser kinds and not implemented — a token issued for either falls through to the generic JSON parser, so your CEF stays unparsed rather than refused.
  • Syslog over TLS exists in the code and is not reachable from the running collector. Plain TCP and UDP are.
  • Okta has a parser and a mapper and no collector; the only route is posting to HEC with a token whose parser kind is okta.

What to do

  1. Establish whether anything is arriving at all

    Search for events from that source over the last hour. Arriving-and-silent and not-arriving are different problems, and everything depends on which you have.

  2. Read your collector's response code, not its dashboard

    202 with an accepted count is success; 200 with partialSuccess is success on OTLP, with the caveat above. 401, 400, 405 and 503 each point at one section here.

  3. Before rotating a token, check whether you are throttled

    Intermittent 401s tracking your event volume are the limiter misreporting itself. Steady 401s on every request are a genuinely bad token.

  4. For syslog, confirm your peer address is mapped

    A listener with no tenant mapping drops every line silently. Ask us to confirm the range rather than assuming the listener is broken.

  5. If the question was really about alerts, stop here

    Read the two-paths section, then A detection rule never fires for the rule-side reasons a detection stays quiet even on the path that does evaluate.

Was this article wrong?

If a procedure here does not match what you see, or a limit we described has changed, tell us and we will fix the page. Email us about this article, or see how to get help if you need an answer rather than a correction.

Everything in troubleshooting