Testing a rule before you trust it

Changing a detection threshold is a change to what you will and will not be told about, and the two ways it goes wrong are opposite. Set it too low and you bury the real alert under forty from a backup service account. Set it too high and the rule is decoration. Neither failure announces itself, which is why the rule editor has a section that replays the rule over the events you actually received before you commit to the number.

That section is at the bottom of the rule editor, under Detection Rules at /siem/rules, and it is called Test against your real data. Its own description is exact: pick what the rule should match, then run it over the events you actually received. The rule's own threshold, grouping and time window are replayed exactly as they are set on the left, so this is what would have fired.

First: what you are actually testing

A shipped rule's conditions are fixed. What you can change, and therefore what the tester is for, is the firing behaviour: severity, how many events, over what window, how many distinct values, and how long it stays quiet afterwards. The editor renders your settings back to you as an English sentence, and reading that sentence is half the review:

The tester then answers the question that sentence raises: given the events you actually received, how often would that have happened? Tuning a detection rule covers the full set of fields you may change and the two that are coupled to each other.

A rule you wrote is different in one respect: its conditions are yours as well, so the tester is replaying a pattern you can still edit rather than only a number you can still move. Everything below about verdicts, limits and shadow mode applies to both kinds of rule; the section Writing a rule of your own covers the part that is specific to yours.

Running a test

1. 1

   Set the firing behaviour on the left first

   The tester reads the thresholds, grouping and window from the form as they currently stand, not from what is saved. So you can try a number without committing to it — which is the point.

2. 2

   Choose a look-back

   The last hour, the last 6 hours, the last 24 hours, or the last 7 days. Start at 7 days for a rule you are unsure about; a quiet hour proves very little.

3. 3

   Press Run the test

   The tester restates what it is about to do in words — count the events in the range that match the conditions, group them the way this rule groups them, and report every group that crosses the threshold — so there is no ambiguity about what the result means.

4. 4

   Read the verdict, not just the number

   There are four possible verdicts and they are not interchangeable. The next section is about telling them apart, because one of them is routinely misread.

The four verdicts

Would have alerted, N times

The groups that crossed the threshold are listed, so you can look at which user or host would have tripped it. This is the verdict to reason from. If the list is dominated by one service account, you have found your exception before it found you.

Would not have alerted

Events matched, and no group crossed the threshold. The hint gives you the arithmetic — how many matching events across how many groups, against the threshold and window — which is exactly what you need to decide whether the threshold is right or merely untested.

No matching telemetry

The one that gets misread. Nothing in the range matched the conditions at all, so there is nothing to test the threshold against. The console says so: connect a log source, or widen the conditions. This is not a passing test.

The tester cannot query these fields

Some of the rule's conditions read fields the search index has no column for. Everything else is tested and the result is a lower bound. See below.

What the tester cannot see, and why

This is the honest limit, and the editor labels it per field rather than hiding it. Each field a rule reads is shown with one of three states.

The distinction in the middle row follows from something explained in Searching your logs: detection rules can read a much larger vocabulary of metadata fields than the search language can filter on. That is a genuine mismatch rather than an oversight, and the tester is built on the search side of it. So a rule keyed on, say, a command line or a consent scope may be partly untestable, and the editor tells you which of its conditions were left out rather than quietly ignoring them.

Read a lower bound accordingly. Would have alerted three times on a partly-untestable rule means at least three, and the real figure could be higher. It never means fewer.

A small number of condition types also cannot be expressed in this quick replay: matching against a list of values, a pattern or regular-expression match, a none of these test, and any negated condition. Where a rule uses one, the tester names the conditions it could not test and reports the result from the rest — again a lower bound. It does not silently drop them and present the number as complete, which is the failure mode worth guarding against: a test that quietly ignored half a rule would read as a clean result.

Shadow mode: the third state a rule can be in

Beyond on and off there is shadow mode, which the editor describes as: records what it would have done without raising alerts. It is a rule's status set to testing rather than active.

Two ways a rule ends up there.

1. It shipped that way, because it reads a field nothing populates yet. The editor says so and tells you what would change it: this rule is in shadow mode because nothing is sending those fields yet, and it will start alerting on its own once that telemetry arrives. You do not have to do anything.
2. You put it there, which is the right move for a rule you have just retuned and do not yet trust. It contributes to risk scoring and stays out of your alert queue.

Shadow detections and why some rules stay quiet is the fuller account, including how a shadow rule still contributes to an incident.

Writing a rule of your own

You can write a detection rule from scratch. Go to Detection Rules → New rule, or use the New rule button on the rule library screen. It needs the same permission as tuning an existing rule, so if you are allowed to move a threshold you are allowed to author a rule.

This is new. Earlier versions of this article said there was no route to create a detection rule and no route to delete one; both now exist. What has not changed is that the rules which shipped with the product are still not editable in place and cannot be deleted — you tune those, exactly as described above and in Tuning a detection rule. Authoring is a separate path for rules that are yours.

The form is in five parts and fills top to bottom.

1. 1

   Name it, and say what it detects

   A rule name, the severity a match should carry, the kind of log the rule reads, and a description of what it detects. The description is not decoration — it is the first thing somebody reads when the rule wakes them at two in the morning — and the form will not save without it.

2. 2

   Say what it looks for

   One or more conditions, each a field, a comparison and a value. Every condition has to be true of one single event for that event to match; they are combined with and, not or. Any individual condition can be negated, which is how you express anything except this. A rule with no conditions at all matches every event that reaches it, so the form refuses to save one.

3. 3

   Say how often before it counts

   A threshold — how many matching events it takes — with a time window they have to fall inside, and a cooldown so that one incident does not arrive as fifty alerts. Leave the threshold at zero and you are told about every single match. You can optionally count separately per group (per user, per host) or per distinct value of a field. A one-line note explaining why you chose that number is required, which is the same discipline the shipped rules are held to: a threshold nobody wrote a reason for is a number nobody can safely move later.

4. 4

   Give it context

   Which ATT&CK technique and tactic the rule maps to; what legitimate activity could look similar, so the rule gets narrowed rather than switched off wholesale the first time it is noisy; and whether it is switched on or in shadow mode — logging its matches without alerting anyone, which is how you try a rule safely before you trust it.

5. 5

   Try it on your real data

   The same replay step described in this article, reading your conditions and your firing behaviour as they currently stand in the form. Run it before you save, and read the verdict rather than the number — the four verdicts and their traps are above, as are the two limits on what the replay can see.

What the form refuses to save, and why

The checks run before you can save, and each one exists because the rule it would have produced is broken in a way you would not have noticed:

• No name, or no description of what it detects — the two things a person triaging the alert reads first.
• No conditions at all. A rule with none matches every event, which is not a detection.
• No log-source kind chosen — without it the rule has nothing to be evaluated against.
• No ATT&CK technique chosen. Every shipped rule carries a mapping and yours does too, so your coverage view stays truthful.
• No note on what could produce a false positive. The whole argument for this being mandatory is in The detection library: a rule without one is a rule you switch off in week two rather than narrow.
• A threshold with no time window, or with nothing chosen to count. Five events is not a threshold until you say five events of what, inside how long.
• Arithmetic that can never be satisfied. The clearest example: counting distinct values of the very field you are also grouping by. Every group can then only ever hold one distinct value, so no group can reach a threshold above one, and the rule would sit there looking healthy and never fire. The form catches that and explains it rather than saving it.

That last check is the same instinct as the field contract described in The detection library. A rule that can never match is the most expensive thing a detection library can contain, because it is indistinguishable from a quiet estate.

Editing, deleting and the ceiling

A rule you wrote yourself you can edit freely and delete outright. A rule that shipped with the product you can neither edit in place nor delete — its tuning controls are the whole of your influence over it, and that boundary and the reasoning behind it are in Tuning a detection rule.

One hundred rules ship today, and that ceiling is deliberate too — a test in the codebase fails the build above one hundred, with the instruction to retire weak rules rather than accumulate them. Your 200 are counted separately and do not eat into it.

If a detection you need is genuinely absent, it is still worth telling us even though you can now write it yourself — knowing which gap a real customer hit is the input that gets a rule written properly, maintained, mapped and shipped to everybody. How to get help.

A method that works

1. 1

   Test at seven days before you touch anything

   Establish what the rule as shipped would have done in your environment. If the answer is no matching telemetry, stop and fix the log source; nothing downstream of that is meaningful.

2. 2

   Look at the groups, not the count

   The names in the would have alerted list tell you whether you have a tuning problem or an exception problem. One repeat offender is an exception. Twenty different users is a threshold.

3. 3

   Add the exception before you raise the threshold

   Excluding the backup service account keeps the rule sensitive for everybody else. Raising the threshold to survive that one account makes the rule less sensitive for the whole estate, permanently. The editor suggests values already seen in your logs as you type, so the exception is one you know exists.

4. 4

   Retest, then put it in shadow mode for a working week

   Not an afternoon. You are watching for the weekly job, the month-end batch and the one person who works Sundays.

5. 5

   Switch it to active, and write down what you changed and why

   The reason field is recorded in the audit trail, so use it properly. It is also the only place the next person will find out that the threshold is five because of a backup account.

What to read next

1. Tuning a detection rule — every field you may change, and the two that are coupled.
2. A detection rule never fires — the diagnostic path when a test says one thing and reality says another.
3. The detection library — what ships, how a rule is put together, and who wrote it.