Skip to main content

Live response and approval

The most powerful thing in the product: what a live-response ticket can do to a machine, who has to approve it, how it is signed, and what is recorded.

Written for whoever runs IT9 min readUpdated

Live response is the mechanism by which Qorionix does anything at all to one of your machines. Every quarantine, every isolation, every process terminated and every file collected goes through it. It is the most powerful capability in the product and it is the one that deserves the most scrutiny before you turn it on, so this article is written to be read sceptically.

The short version: a live-response ticket is a cryptographically signed instruction to one named machine to run one named command from a list of four, valid for fifteen minutes, requiring a named human to approve it, with the result recorded. There is no interactive shell and no arbitrary command execution. If you were worried that installing our sensor gave somebody a remote terminal on every computer you own, it does not — and the reasons why are below.

What a ticket is

A ticket names five things: your organisation, one sensor, one command, whoever requested it, and when it expires. It is created through the platform, approved through the platform, and collected by the machine — the platform never connects to your machines, which is why none of this requires you to open a firewall port inbound.

The machine holds a long-lived connection open asking for work. When there is work, it receives at most sixteen tickets at a time, and only tickets that are approved and have not yet expired. Each ticket handed over is immediately marked as dispatched, so the same instruction cannot be collected and executed twice.

The sensor then runs the command, and reports back a status and a detail string. That detail is stored on the ticket and is exactly what you read in the console as the outcome — including when the outcome is the sensor refusing.

What a live-response session can actually do

Being explicit here matters more than being reassuring, so this is the whole surface. Four commands do something on a machine today.

`fetch_file`
Reads a named file off the machine and brings it back to the platform, with its SHA-256 digest. This is a read of arbitrary file contents from a machine you own, bounded only by a size limit — 10,485,760 bytes on Linux, with an equivalent cap on Windows and macOS. Treat the permission to run it as the permission to read any file on any endpoint, because that is what it is.
`kill_process`
Terminates one process by its process ID. Irreversible. The sensor refuses to terminate the operating system's own root process — process 1 on Linux and macOS, process 4 on Windows — because doing so takes the machine down.
`isolate`
Rewrites the machine's firewall to block everything except the control plane, after saving the existing configuration. See Isolating a machine, which is written for whoever decides.
`quarantine_file`
Moves a named file into a protected vault directory on the machine, renames it to its own SHA-256 digest, and strips all access to it. The original bytes survive, which is what makes it reversible.

Two further commands — unisolate and restore_file — exist to reverse the last two. They are implemented on all three operating systems and are currently not deliverable over the long-lived connection; the defect and the workaround are in Response actions and repeated in the isolation article, because it matters most there.

So the honest boundary is: read any file, kill any process, cut the network, vault a file. That is a serious set of powers over a computer, and it is a short and enumerable one. Nothing in it can install software, change a password, read memory, or run a command of your choosing.

The approval requirement

Every ticket is created with the status pending_approval, unconditionally. There is no path in the product that creates an already-approved ticket — not for an administrator, not for the platform's own automation, not for us.

One approval is required by default, and the threshold can be raised. Reaching it signs the ticket and moves it to approved. Approving requires the response-actions write permission; un-isolating and undoing require the higher execute permission, because those change a host's state immediately rather than queueing a request for somebody else to consider.

There is exactly one exception in the whole module, and it is narrow: a file-quarantine ticket raised by a threat-intelligence hash match can approve itself, if the autonomous capability is enabled, the rule is set to tier 2, and the indicator's confidence is at or above 0.95. The recorded approver in that case is the literal string sentinel-q-tier2-auto, so an automatic approval is never mistakable for a person's.

The one-tap un-isolate and undo controls also create and approve in the same call. That is not an exception to the rule so much as an application of it: the operator pressing the button is the human in the loop. What is skipped is the waiting, not the recording — the ticket row, the approval row naming the operator, and the signature are all written exactly as normal.

Ticket signing, and the separate secret

A ticket carries a detached signature: a keyed hash computed over a canonical string, so that a sensor can prove the instruction came from the platform and has not been altered in transit.

What is signed
<ticket_id>.<sensor_id>.<command>.<expires_rfc3339>

The machine recomputes that string from the ticket it received and refuses anything whose signature does not verify. It then separately refuses anything already past its expiry. Both checks happen before the command reaches any code that could act on it.

The important part is which key does the signing. It is not the platform's session-signing key. On the platform the value is called TICKET_SECRET; on a sensor it is the environment variable ZQ_TICKET_SECRET. The two must hold the same value, and that value must not be the platform's JWT_SECRET.

The two used to be the same value, which meant root on any one endpoint yielded a key that could mint platform JWTs for any user in any tenant.
Sensor source, on why the ticket secret is separate

That is worth sitting with. A sensor has to hold the ticket key in order to verify tickets, and a sensor lives on a laptop that somebody might eventually own. When the ticket key and the session key were the same value, compromising one endpoint yielded the ability to issue valid sessions for anyone in any organisation on the platform. Separating them turns a total compromise into a contained one.

Three protections follow from that, and they are worth checking on your own deployment:

  • The ticket secret is derived one-way from the session secret when it is not configured explicitly, so it cannot be reversed back into it. Leaving it unset is safe; setting it carelessly is not.
  • Setting the ticket secret equal to the session secret — or to any other platform secret — is a boot-time configuration error. The platform refuses to start rather than running in the state that caused the original problem.
  • The residual risk is stated rather than hidden. The signature is a symmetric keyed hash, which means a sensor holds the key it verifies with. A fully compromised endpoint can therefore forge tickets for itself. It cannot forge tickets for any other machine, and it gains nothing it did not already have by being compromised — but it is not zero, and we would rather write it down than let somebody discover it in an assessment.

What is recorded

Every stage leaves a record, and the records are designed to be readable by somebody who was not there.

What each stage of a live-response ticket writes down
StageWhat is recorded
CreationThe organisation, the sensor, the command, its arguments, the blast radius, and who or what requested it. Tickets raised by the detection engine record sentinel-q-autoresponse as the requester.
ApprovalA separate row naming the approver, and the signature over the canonical string. An automatic tier-2 approval also records the reason, naming the indicator, the confidence and the floor it cleared.
DispatchThe moment the machine collected the ticket, which is also what prevents a second collection.
CompletionThe status the sensor reported and its detail string — the same text that appears as the outcome in the console, including refusals.
EvidenceFor a quarantine: the file's original path, its vault path, its digest and when it was performed. For an isolation: the literal command that reverses it. This evidence is what an undo is derived from, rather than anything the caller supplies.

Alongside the ticket, decisions taken by Sentinel-Q are appended to a hash-chained audit log — an append-only record in which each entry is bound cryptographically to the one before it, so that a deleted or edited row shows up as a break rather than as nothing. What is hashed is deliberately a fixed, separate structure rather than the whole record, so that adding a display field in a future release cannot silently change every historical hash.

Each of those entries carries an undo handle where one exists — and where none exists, that absence is itself meaningful: callers treat a missing handle as this action must not be taken autonomously. Undoing a change Qorionix made covers what the handles contain.

The limits, in one place

  • One machine per ticket. There is no fleet-wide live-response action. Containing ten machines is ten tickets and ten approvals.
  • Sixteen tickets per collection, approved and unexpired only.
  • Fifteen-minute validity from creation.
  • No shell, no arbitrary command. Four verbs, listed above.
  • File collection is size-capped at 10,485,760 bytes on Linux, with an equivalent cap elsewhere. A large database file will not come back.
  • Isolation refuses without a control-plane range, so a machine cannot be cut off from the platform that would restore it.
  • Restoring a quarantined file validates the digest before using it — sha256 must be a 64-character lowercase hex digest — because anything else is a path-traversal attempt into the vault. It also refuses to overwrite an existing file at the destination.
  • Undo of an unfinished ticket is refused, with the ticket's actual state named in the message.

Errors and what they mean

Server-side, with the HTTP status you will see in a log
liveresponse: ticket not approved                                    409
liveresponse: ticket expired                                        410
liveresponse: not found                                             404
liveresponse: this action has no counter-command and cannot be
  undone automatically                                              422
un-isolate requires an identified operator                          401
undo requires an identified operator                                401
request carries no organization context                             403

The two requires an identified operator messages are not a bug to work around. A reversal performed by nobody in particular is a reversal nobody can be asked about later, so the call is refused rather than attributed to the system.

On the machine's side of the connection, two refusals indicate an identity problem rather than a permission one:

Sensor authentication on the ticket connection
missing sensor auth context                     401
sensorID does not match authenticated sensor    403

The second means a sensor asked for tickets belonging to a different sensor. It is worth knowing about a historical version of this problem, because it produced a silence that looked exactly like nothing being wrong: a machine that re-enrolled used to walk away believing it was a sensor identifier that did not exist in the database. Telemetry kept working, so the machine looked healthy in every list — and its request for tickets asked about a sensor nobody had, so it silently never received any. Approvals appeared to succeed and nothing ever happened on the host. That is fixed. If you see a live-response ticket that is approved and never collected, that is still the shape of failure worth reporting to us.

A refusal that mentions sensor auth: — missing bearer token, unknown enrollment token, sensor revoked — is about the machine's credential rather than about the ticket. A revoked sensor is refused at the door and will not collect anything.

Who should hold this

Given what the four commands do, the permission to approve them deserves the same care as an administrator account. Our own default grants give an analyst the execute permission — containing a host is the core analyst job — and give a viewer nothing beyond read.

Two suggestions from experience rather than from the source code. Make sure more than one person can approve, because the fifteen-minute expiry means an approval that waits for one specific person on holiday is an approval that does not happen. And review who holds the execute permission whenever somebody leaves, since fetch_file alone is the ability to read any file on any machine in the business.

Roles and permissions has the full grant list, and Response actions covers what each action does once approved.

What to do next

  1. Check who in your organisation currently holds the response-actions execute permission. If the answer surprises you, fix it before you need it.
  2. Confirm with whoever runs your deployment that ZQ_TICKET_SECRET is set from the platform's ticket secret and not from its session secret.
  3. Read Isolating a machine — it is the one action whose reversal you should understand before the first time you approve it.

Was this article wrong?

If a procedure here does not match what you see, or a limit we described has changed, tell us and we will fix the page. Email us about this article, or see how to get help if you need an answer rather than a correction.

Everything in endpoints and response