When a check cannot run

Some checks in your report will say they could not be evaluated. This article explains why, what each reason means, and what you can do about it. It is a short article about a single rule, and the rule is the most important one in the product.

The five reasons

Every failed read during discovery is classified into one of five causes. The distinction is not cosmetic. Two of them are yours to act on, one is usually nothing at all, one resolves itself, and one is our fault — and collapsing them together would hide our own bugs behind your licence tier.

The {X} in each sentence is the name of the dataset in the same words the interface uses elsewhere: users and guests, multi-factor authentication registration, administrator roles, groups, Conditional Access policies, licences, devices, application consents, sign-in logs, mailbox rules. So a real gap reads: We were not permitted to read your mailbox rules, so the checks that depend on it could not run.

What a skipped check looks like

Skipped checks are reported in the report, not swallowed. Each one carries its identifier, its title, the coverage gap it is waiting on, and a reason in this form:

could not be evaluated: the sign-in log data was not available for this tenant
could not be evaluated: the mailbox rule data was not available for this tenant
could not be evaluated: the multi-factor authentication registration data was
  not available for this tenant

The count is shown alongside the findings. Six checks could not be evaluated because your licence does not emit sign-in risk is an honest summary of your posture. Quietly passing those six is not, and the difference between the two is the difference between a report you can rely on and one you cannot.

The gaps you are most likely to see

Four are common enough to name individually.

Last sign-in is unavailable

We could not read when each account was last used, so we have not reported any account as dormant. Reading last sign-in requires Microsoft Entra ID P1 and the AuditLog.Read.All permission. When this attribute is the reason a user query fails, we retry without it rather than losing your entire staff list — the directory matters more than the timestamps — and record the gap.

Sign-in risk is hidden

Microsoft hides its sign-in risk scoring on your licence tier, so we could not check whether any sign-in was flagged as risky. Sign-in risk detail requires Microsoft Entra ID P2. Graph reports the risk level as the literal string hidden in this case; treating that as a risk level would invent findings and treating it as none would invent an all-clear, so it is normalised away and recorded as a gap.

A list was longer than one pass

Any paged enumeration stops after 200 pages. You are told: Your directory is larger than this assessment enumerates in one pass, so the account list is incomplete — with the equivalent wording for devices and sign-ins — and the enumeration continues in the background. A truncated inventory that looks complete is the dangerous outcome, so hitting the cap is always reported.

Only the important mailboxes were checked

We checked mailbox rules on the 200 most important accounts rather than all N. Administrators are scanned first, then accounts with a mailbox ordered by most recent sign-in, so the bound degrades coverage in the least harmful order. The remainder continues in the background.

Absent is not the same as clean

The rule at the top of this article has consequences throughout the inventory, not only at the level of whole checks. Three examples of the same principle applied to individual facts:

• A missing signInActivity is a coverage gap, never has not signed in. An account whose last sign-in we cannot read is not reported as dormant. Disabling accounts on the strength of an absent timestamp would lock people out of their own jobs and would look, from the inside, exactly like the product working.
• A user absent from the authentication-methods report is not assessed, not no second factor. A missing signal must never manufacture a finding, and telling you that a director has no second factor when in fact we could not see their registration is the kind of error that costs a report all of its credibility at once.
• A risk level of hidden is neither risky nor safe. It is unknown, and it is recorded as unknown.

Sometimes zero is the finding

The converse trap is treating an empty result as missing data. A tenant with no Conditional Access policies at all returns an empty list, and that is not a gap — it is one of the most important findings in the report, because it means nothing is requiring a second factor no matter who has registered one. An empty policy list is deliberately not recorded as missing data, for exactly that reason.

The same applies in the other direction to the platform's own configuration. The gaps that describe what Qorionix has switched on for you — the tenant audit log, the activity feed, mailbox auditing, retention, the baseline detections — default to assumed not configured. A pessimistic default produces an offer to fix something that may already be fixed. An optimistic default produces a silent claim that something is done when it is not, and there is no contest between those two mistakes.

What to do about a gap

1. 1

   Read the cause, not only the message

   The five causes above have completely different remedies. Not set up on this tenant usually needs nothing from you. We were not permitted needs about two minutes in the Entra admin centre. Your licence does not make it available needs a decision about money, and we will tell you what the money buys before you make it.

2. 2

   For a permission gap, add the permission and consent

   Find the dataset in the table in Every permission we ask for, and why, add that application permission to your app registration, and grant admin consent. You do not need to reconfigure anything in Qorionix: the permission is attached to the app registration, not to the connection record.

3. 3

   Start a fresh run yourself

   For a permission gap the report says we will re-run this part automatically. Take that as a statement of intent rather than a promise: nothing currently watches for a gap being closed, so a newly granted permission is picked up on your next assessment run and not before. Start one once the consent is in place. We would rather tell you this than have you wait a fortnight for a re-run that was never scheduled.

4. 4

   For a licence gap, treat it as a decision

   Read What your Microsoft licence lets us see. For many organisations of 40 people the honest answer is that the upgrade is not worth it and the changes already in front of you matter more. That is a legitimate answer and the report is built to let you give it.