How your logs are normalised

Every event that reaches Qorionix is rewritten into a single shared schema before anything reads it. This article explains what that schema is, which field names you can use when you write a search or tune a rule, and a few decisions inside the mapping that will otherwise look strange the first time you open a raw event.

It is written for whoever runs IT. You do not need to know the schema to use the product, but you do need to know the field names the moment you want to search for something specific or narrow a noisy rule.

Why one schema matters more than it sounds

A failed sign-in looks completely different depending on who reported it. Microsoft 365 calls the person UserId and the outcome an error number. A Windows domain controller calls the same person TargetUserName and reports event 4625 with a SubStatus code. AWS calls them userIdentity.arn. A firewall might not name them at all and give you only an address.

If the platform stored those as they arrived, every detection rule would have to be written once per product. A rule for failed sign-ins from one address would become five rules that drift apart, and the sixth source you connect next year would be invisible to all of them until somebody wrote a sixth. That is the ordinary state of an untended log collection, and it is the reason so many of them collect without detecting.

So normalisation happens once, at the front door. One rule then covers every source that can express the concept, and a source you add later is covered on the day it starts sending.

The schema: OCSF 1.6.0

The target schema is the Open Cybersecurity Schema Framework, an open, vendor-neutral standard for describing security events. Qorionix maps to version 1.6.0, and the attribute names and enumerated values are taken from that published version rather than invented. The version is recorded on every stored row, so a future upgrade does not leave you guessing which shape a given event is in.

OCSF has many event classes. Qorionix maps to five of them, and that is the whole set — a short true list rather than a long claim.

Five sources have dedicated mappers written against their exact record shapes: Microsoft 365 and Entra, Windows Event Log, AWS CloudTrail, Google Workspace and Okta. Everything else — syslog lines, arbitrary JSON posted to the collector, network appliances — goes through a generic fallback, so every stored event is OCSF rather than only the ones somebody wrote a mapper for.

Where the original record is still in hand, the source-specific mapper runs on the raw bytes first, because nothing has been flattened yet: a Windows 4625 keeps its SubStatus, and a Microsoft 365 sign-in keeps its RiskLevelDuringSignIn. Only if that fails does the pipeline fall back to mapping the already-parsed version, which cannot recover detail the parser dropped.

One rescue is worth naming because it looks like a bug when you first see it. An Authentication event has to name a principal. When a source names none — a malformed sshd line, an appliance reporting only auth failed from this address — the event is not discarded. The user is recorded as the literal unknown and flagged as unresolved, because dropping it would lose the source address that brute-force-by-address rules need. The flag is there so nobody mistakes unknown for a real account.

Every event is written twice, on purpose

One arriving event produces two database rows inside one transaction, sharing the same identifier so they always join:

• The OCSF document. The complete, unflattened OCSF event as a JSON document, plus a set of typed columns — time, class, severity, actor, target, addresses, hostname, operation — that exist only because a database cannot index a path inside a JSON document as cheaply as a real column. Those columns are projections, not a second version of the truth.
• A flat projection. A single wide row in the platform's own event table, with a fixed set of columns and a metadata document. This is what the detection engine evaluates, what search reads, and what every console view displays.

That is not duplication for its own sake. Writing only the OCSF document would have normalised your data and simultaneously disconnected it from everything that consumes it. Writing only the flat row would have thrown away the detail that makes the raw event worth keeping for an investigation.

The field names you will actually type

A small number of fields sit at the top level of an event: category, action, outcome, severity, user_id, source_ip, dest_ip, host, actor_type, raw_log and classification. Everything more specific lives under metadata., and that set is not open-ended — it is a declared contract of named fields, each with a stated availability. You can read your own copy at any time from the field-contract endpoint, which the rule-authoring interface and the ingest side both read, so the two cannot drift apart.

A representative sample, to show the shape rather than the whole list:

A handful of contract fields are declared but marked pending: the pipeline does not populate them yet. Two are geographic coordinates, which no connected source reports. The rest are computed inside the privileged-access, identity and AI modules, none of which currently writes an event into the SIEM. The consequence — that the rules reading those fields stay quiet — is covered in Shadow detections, and why some rules stay quiet.

Where each field's value comes from

There are exactly three ways a metadata field gets its value, and each site in the code is marked with which one it used. Knowing the difference tells you how much to trust a field on an unfamiliar source.

Read from the standard

OCSF has an attribute that means precisely this, so the value is read from it. metadata.command_line comes from the process command line, metadata.geo_country from the source endpoint's country, metadata.bytes_out from the traffic counters. These are the most trustworthy, because two different sources both had to satisfy the same published definition.

Lifted from the unmapped block

OCSF has no attribute for the concept at all, so the value is taken out of the event's unmapped block — which is exactly what the standard reserves that block for. Authentication in OCSF 1.6.0 defines nothing for which client application was used, whether Conditional Access applied, or how a multi-factor challenge ended, so metadata.client_app, metadata.ca_status and metadata.mfa_result are lifted rather than forced onto a standard attribute that means something else.

Computed

Derived from OCSF attributes plus what the pipeline knows about the event. metadata.auth_requirement is computed from whether multi-factor was involved; metadata.file_operation from the Windows event code; metadata.file_extension from the file path; metadata.dns_query_length from the length of the query. A computed field is only as good as its inputs, which is why the derivation is recorded rather than assumed.

Why the OCSF document is nested inside metadata

The metadata document holds the flat contract fields at its top level, and the entire OCSF event nested underneath a single ocsf key. That looks like an odd shape until you meet the reason, which is a genuine collision between two vocabularies that use the same word for different things.

{
  'identity_provider': 'entra',
  'auth_protocol': 'device_code',
  'risk_level': 'high',
  'geo_country': 'RU',
  'ocsf': { 'class_uid': 3002, 'auth_protocol': 'OAUTH 2.0', 'user': {...} }
}

Both keys are called auth_protocol and they mean different things. OCSF's is a protocol caption from a fixed enumeration — OAUTH 2.0, Kerberos, SAML. The detection contract's is the OAuth grant flow: device_code, ropc, saml, oauth2, wsfed. That distinction is not a nicety. It is the whole basis of the device-code phishing rule NES-SIEM-100 and the ROPC rule NES-SIEM-101, and the OCSF 1.6.0 enumeration simply cannot express it.

Flattening the OCSF document into the same namespace would have made one of those two meanings win silently. Nesting keeps them apart, still lets you address any OCSF attribute directly as metadata.ocsf.<path>, and still lets a rule that matches on the whole metadata document with a substring or pattern see everything.

The category vocabulary the projection had to invent

Rules and searches scope themselves with a category, and that vocabulary has members OCSF has no category for — because, as above, mail, files, the registry, DNS and cloud control-plane calls all arrive as one class. Leaving them all labelled api is what made whole rule families unable to match. So the projection resolves a category from the workload the source itself states, in this order:

1. A registry key on the event makes it registry.
2. The stated workload or service name decides next: anything containing exchange becomes email; sharepoint or onedrive becomes file; Azure Active Directory or Entra becomes iam. This is checked first because it is the one label the source itself asserts — the Microsoft 365 audit log genuinely says Exchange, and CloudTrail genuinely names the service.
3. A recognisable cloud product key becomes cloud. This test is deliberately narrow, so that a cloud-scoped rule cannot be satisfied by a Teams audit record that merely happened to come from Microsoft.
4. A file path with nothing better to go on becomes file.
5. Anything left over stays api.

The full set of values you can use is authentication, iam, process, network, email, file, registry, cloud, api and other. Alongside it, action carries the specific verb — login_failed, login_success, mfa_failed, logout, user_created, password_reset, api_key_created, role_assigned, process_launch, connection_refused, dns_query, and so on.

One small fix in that path is worth knowing about if you ever compare a raw record with its normalised form. The Microsoft 365 audit log spells directory operations with a trailing full stop — Consent to application. — while Microsoft's own documentation, which the rule library follows, has none. An untrimmed operation matched no rule at all, so the trailing stop is stripped during projection.

An absent field is not an empty one

A metadata key is written only when the event genuinely carries evidence for it. Empty strings and nulls are dropped rather than stored.

This is the same principle that governs posture checks in When a check cannot run, applied at the level of individual fields. Rules use exists and not in conditions, so writing a guessed or blank value would turn we did not observe this into we observed nothing — and that is precisely how a rule ends up firing on the absence of telemetry rather than on an attack. The behavioural verdicts follow the same discipline: a verdict is stamped only when it is anomalous, because writing false on every event would make an exists condition on it meaningless.

The practical consequence when you search: do not assume a field is absent because the value was negative. It is absent because there was nothing to say.

What to read next

1. Searching your logs — the field names above, put to work.
2. The detection library — how a rule is built on top of this vocabulary, and the contract that stops a rule reading a field nothing populates.
3. Connecting your first log source — including the honest account of which sources alert today and which only collect.