The sensor update channel and staged rollout

The platform has a real sensor update channel: signed releases, three channels, per-tenant pinning, a staged rollout that reaches roughly five per cent of a fleet first, and automatic rollback when an updated sensor goes quiet. The server side of that is built and has been verified against a migrated database.

And the sensors you have installed do not use any of it, because no build script stamps the verification keys into the binary and the sensor refuses to poll without them. This article describes both halves, because the machinery is real, the gap is specific, and knowing which is which is the difference between planning around it and being surprised by it.

The gap, precisely

The sensor verifies an update against Ed25519 public keys compiled into its own binary, deliberately not fetched from the platform, so that a compromised platform cannot authorise an update. The value is a build-time substitution, and the update loop's first action is to check it:

if embeddedPublicKeys == "" || os.Getenv("ZQ_DISABLE_AUTO_UPDATE") == "1" {
    return
}

No build script or pipeline in the repository sets that value. The release script stamps the version and nothing else. So every binary the shipped build produces has no keys, the loop returns before starting its ticker, and the sensor never asks whether an update exists.

This is fail-closed, which is the right direction for the failure to point. A sensor with no keys cannot be tricked into applying an unsigned update; it simply does nothing. It is still a gap, and it is a gap in the half that is hardest to fix retroactively — the machines already deployed carry the binary they were installed with.

The three channels

stable, beta and nightly. Three, enforced both in the code and as a database constraint, and the default for a tenant that has never been assigned one is stable.

There is no channel called canary. That word means a percentage in this product, not a channel, and a test exists whose only job is to reject canary as a channel value. If you have seen the term in a conversation about rollouts, the next section is what was meant.

Five platform strings are published: linux-amd64, linux-arm64, windows-amd64, darwin-amd64 and darwin-arm64. Note there is deliberately no darwin-universal — the universal binary is what the macOS installer package ships, but a running sensor polls with its own single architecture, so macOS is published as two releases pointing at two artefacts.

Staged rollout: a stable 5% cohort, enforced

This is the part most worth understanding, because it is the control that limits the damage from a bad build.

Every sensor is hashed into one of a hundred buckets — a plain FNV hash of its lowercased identifier, taken modulo 100. A release carries a rollout percentage, and a sensor is offered it only when its bucket number is less than that percentage. The hash is pure, so the same host lands in the same bucket every time: a 25% rollout is stable across polls rather than re-drawing the cohort every four hours.

The default is 5. A release published without a percentage is rewritten to five before it is stored, and the reason is recorded: a bad sensor pushed to a whole fleet is an outage nobody can recover from without physical access to every endpoint. Going straight to a full rollout is possible, and it requires setting the percentage to 100 explicitly rather than leaving it out.

The intended operator rhythm is to publish once at 5%, then 25%, then 100%, watching the endpoint list between steps. Yanking a release is a single action.

How a release is signed and verified

Ed25519, with a division of labour that is the interesting part: the platform holds public keys only and never signs anything. Signing happens offline with a separate tool, and the platform refuses a release that arrives unsigned — signature required — the platform never signs a release, the offline release workflow does.

signed payload:  version|sha256|platform|fingerprint
fingerprint:     the first 16 bytes of SHA-256 over the public key, in hex

Binding the platform and the artefact into the same signature matters: a signature over the hash alone could be replayed to offer a Linux build to a Windows host, or an old version to a sensor that had moved past it.

1. 1

   The operator signs offline and publishes

   The platform verifies the supplied signature against the stored public key before the release is stored at all. A mismatch is refused with signature verification failed — the supplied signature does not match (version, sha256, platform, fingerprint).

2. 2

   A sensor asks whether there is a newer build for it

   It sends only its platform and its current version. Its identity and its organisation come from the enrolment credential it authenticates with, never from a query parameter.

3. 3

   The sensor verifies the signature before downloading anything

   The manifest names the key fingerprint; the sensor selects the matching compiled-in key. An unrecognised fingerprint is refused with manifest signed with unknown key. A compromised platform therefore cannot authorise an update.

4. 4

   Then it downloads and checks the hash

   A mismatch deletes the temporary file: hash mismatch: got …, want … (artifact tampered or corrupt).

Key revocation is worth understanding because it is not a revocation list. Each release is bound to the signing key that signed it, and each key has a validity window. The offerable-release query joins the key and filters on that window — so the platform declining to offer a release whose signing key has expired is the whole of revocation enforcement. A release with no key binding at all is not offerable, and existing ones were retired by the migration that introduced the binding.

What a tenant can set, and what only a database can

Six per-tenant settings exist. Two are reachable through the API; four are not.

Setting a channel or a pin is PUT /api/v1/organizations/{orgID}/sensor-channel, and the body accepts exactly those two fields. The other four have no route, no console control and no command-line tool; changing one means an operator changing a row. That is worth knowing before you plan a rollout around a minimum version.

Automatic rollback works like this. When a sensor is offered an update, a deadline is armed. If the sensor has not reported back by the deadline it is presumed broken, and the previous known-good version is offered instead. Where a rollback is not possible — because there is no previous version to go back to — the recorded error says so rather than claiming a rollback that did not happen. That distinction was a real defect once: the failure path used to report a rollback it had not performed.

A sensor that was rolled back is also quarantined from the version that broke it, individually, so it is not offered the same bad build on its next poll. One exception worth knowing: a pin is not subject to that quarantine. If you pin a version, you get it, including one a sensor previously rolled back from. A pin is an instruction, and overriding it silently would make it useless.

Who can publish a release — a boundary we are not happy with

Publishing a release and yanking one are fleet-wide actions: they affect every customer, not one tenant. Today both are gated on xdr:endpoints.write, which is a tenant-scoped permission that every tenant administrator holds.

The code says what that is: an immediate stop-gap, not the right boundary. It stops a viewer. It does not stop a tenant administrator from pushing a release to the whole fleet. A narrower pair of permissions — a read and a publish, specifically for sensor updates — is defined and seeded, and no route uses it yet.

We are documenting this rather than waiting until it is fixed, because you are entitled to know the shape of the trust boundary you are inside. Its practical exposure today is limited by the fact that the update channel is not live, which is a mitigation by accident rather than by design.

macOS: the signature, not the ticket

macOS is the platform where all of this gets genuinely interesting, and where the constraint comes from Apple rather than from us.

The update artefact is the raw executable, never the installer package. And a raw executable cannot be notarisation-stapled. Apple, verbatim: Although tickets are created for standalone binaries, it's not currently possible to staple tickets to them. Stapling supports disk images, signed installer packages and code-signed bundles, and nothing else. That is why the distributable artefact is a package, and it has a direct consequence for the update channel.

So the update path is authorised by three things, none of which is a notarisation ticket:

1. The Ed25519 manifest signature is the authorisation, checked before the download, against keys compiled into the sensor.
2. Gatekeeper is not in this path at all. Gatekeeper assessment is triggered by a quarantine attribute that the operating system attaches to downloads made by a browser or an installer. A file written by the sensor's own HTTP client carries no such attribute, so executing the replaced binary is never assessed. Notarisation buys nothing here — it is entirely a property of the downloadable installer.
3. The Developer ID signature still matters, on Apple Silicon, because macOS on that architecture refuses to execute an unsigned executable at all. This is why the update artefact must be the signed binary and not the raw build output.

One detail with a bearing on how sign-and-publish must be ordered: signing rewrites the executable, so the hash in a release manifest must be taken after signing. Publish the pre-signature hash and every sensor correctly refuses the update.

And the blast radius, stated as the sensor team states it: a macOS build that will not execute costs about five per cent of the macOS fleet one restart cycle. That is the 5% cohort earning its keep.

One known defect in applying an update

When the sensor applies an update it starts the new binary as a child process rather than replacing itself. The consequence is that two sensors run, and on the next poll the old one is offered the same update again. It is recorded in the code as a known defect and it is not fixed.

A related macOS-specific problem was fixed, and it is worth knowing about because the symptom was alarming: the updater used to write the new binary over the running one in place. macOS validates a signed executable's pages lazily, so the kernel would kill the running process partway through. The updater now stages the new file and renames it into position.

What to do with all this

1. Keep upgrading manually, and verify from the endpoint list rather than from your deployment tool's success count. Keeping the sensor up to date.
2. Do not build a compliance statement on automatic patching of the sensor. If a questionnaire asks whether endpoint agents self-update, the answer today is no.
3. When it does go live, expect to be in the 5% first for about one machine in twenty, and expect a rollout to take a few days rather than an afternoon. That is the design working.
4. If you need to switch it off on a particular machine, ZQ_DISABLE_AUTO_UPDATE=1 in the sensor's environment is the documented kill switch, and it will keep working when the channel goes live.

What to read next

1. Keeping the sensor up to date — the manual procedure you will use for now, including fleet verification.
2. Installing the sensor on macOS — the signing position on that platform in full.
3. The endpoint list and endpoint health — the view every rollout is verified against.
4. Sensor rollout rings — the newer, host-group-scoped rollout object, and the same not-yet-wired caveat that applies to it too.