Credential rotation overview

Rotation means Qorionix changes the password on the target machine on a schedule and stores the new one in the vault. It is the control that turns 'nobody knows how old this password is' into a date, and it is the only thing that makes a departed employee's memory of a shared password stop mattering.

This article covers how the schedule works, what happens when a rotation fails, and a defect from earlier in this product's life that reported success while leaving the vault holding the old password. That last part is uncomfortable to publish. It is here because the failure mode it produced — customers locked out of their own systems — is exactly the thing you would want to have been warned about, and because you should know how to check for yourself rather than take our word that it is fixed.

Why the interval matters more than the password

Password advice usually concentrates on strength: length, character classes, entropy. For a privileged account that advice is close to irrelevant, because almost nobody breaks a privileged password by guessing it. They obtain it — from a former colleague, a spreadsheet, a chat history, a reused password on a breached website, a script with credentials in it, or a support ticket where somebody pasted it.

Against that threat, strength does nothing and age does everything. A forty-character random password that has never changed is a permanent key, and every copy ever made of it still works. A twenty-character password that changes every thirty days is a key with an expiry date: whatever leaked stops working, without anybody having to notice that it leaked.

Qorionix generates the replacement secrets itself, so strength is not a thing you have to think about. The engines produce between 18 and 24 bytes of cryptographically random material depending on the platform — for databases, 24 bytes rendered as 32 characters spanning upper case, lower case and digits, which is shaped specifically so that a SQL Server instance enforcing its own complexity policy will accept it. Your only real decision is the interval.

How the schedule works

There are two separate numbers and confusing them is the most common source of 'why has nothing rotated'.

The scheduler tick — 60 seconds

How often Qorionix looks for work. Every minute it selects up to 25 schedules whose next run is due and processes them. Overridable at deployment with the environment variable ROTATION_SCHEDULER_INTERVAL. This number has nothing to do with how often your passwords change.

The credential interval — 30 days by default

How often this password changes. Set per credential when you enable rotation, in days. This is the number you care about.

The frequent tick is what makes a rotation you schedule for 'the next run' happen within a minute rather than tomorrow.

One more operational constraint, and it is a real one: run exactly one worker. The query that claims due schedules does not take a row lock, and the next-run date only advances after the rotation finishes. Two worker replicas would therefore pick up the same credential at the same moment and rotate it twice concurrently, with the second rotation authenticating using a password the first one has already replaced. Do not scale this process horizontally until we publish that it is safe to.

Turning rotation on for a credential

You choose an interval. You do not choose an engine — the engine is derived from the target's platform plus the credential's type, so there is no way to build a schedule using a rotation method that cannot work on that machine.

1. 1

   Attach the credential to a target first

   Rotation needs an address to connect to. Without one you get this credential is not attached to a target yet, so Qorionix does not know which machine to rotate it on.

2. 2

   Record the account name

   Rotation needs to know whose password to change. Without it: this credential has no account recorded, so Qorionix does not know whose password to change.

3. 3

   Run the connection test and get outcome ok

   On the four platforms where a real sign-in is possible, this proves the stored password is the one the target currently expects — which is a precondition for rotation, because the engine authenticates with the current secret before setting a new one. See Targets and devices.

4. 4

   Choose the interval, and when the first run happens

   An interval of zero or less becomes 30 days. A first run of zero means the next scheduler tick — within about a minute. Setting the first run a few hours out is a reasonable habit on a production machine, because it gives you a window to change your mind.

One refusal is ours rather than yours: rotation is not available: the service was started without a rotation store returns HTTP 500 and means the deployment is not configured for rotation at all. Nothing you can do in the console will fix it.

How a failure is recorded

Every attempt writes two things: it updates the schedule with the outcome, the reason and the next run date, and it appends a row to the rotation history. The history row carries whether the new secret was verified and the raw driver text, which is kept for support and is never the message you are shown first.

In the vault list, every credential carries a rotation summary written as a sentence rather than a status code, so one page answers is-it-on, when-did-it-last-run, did-it-work, when-is-it-next and why-did-it-fail. The states and their exact wording:

Two blocker sentences appear in place of a schedule where something is missing: Attach this credential to a target first. and No account is recorded for this credential, so Qorionix does not know whose password to change.

Times are rendered in words — just now, 3 hours ago, due now, in 12 days, never, not scheduled — with correct pluralisation, on the grounds stated in the code itself: '1 minutes ago' in a security product reads as carelessness about everything else.

The two scheduler error strings

Most failure text comes from the target itself and is explained the same way a connection test result is. Two messages come from the scheduler and are worth recognising:

Qorionix has no rotation engine called %q, so this schedule can never run.
  Recreate it choosing the platform of the target.

Qorionix could not open the stored credential for this schedule, so it did not
  attempt to contact the target.

The first means the schedule names an engine that does not exist — almost always a schedule created by hand against the API rather than through the console, since the console derives the engine for you. It will never succeed; delete it and enable rotation from the credential instead. The second means the stored secret could not be decrypted. No connection to your target was attempted, so nothing on the machine changed; the problem is on our side of the wire.

There is no retry, and what to do about it

A failed rotation is not retried. The next run date is set to now plus the interval on success and on failure identically — so a rotation that fails on a thirty-day schedule will not be attempted again for thirty days. There is no attempt counter, no backoff, no jitter.

We are not going to dress this up as a design decision. It is a gap. The practical consequences for you are two:

• A failing rotation will not fix itself, even if the cause was transient. A firewall change that broke connectivity for an hour costs you a full interval of rotation on that credential unless someone intervenes.
• Failures have to be noticed by a person. The credential's own summary says Last attempt %s failed. with the reason, so noticing takes a glance at the vault list rather than an investigation — but the glance has to happen. Put it in whoever's weekly routine owns this.

The remedy is the Rotate now action on the schedule, which runs the rotation immediately, in the API process, and records the result the same way a scheduled run would. Use it after fixing the cause; do not use it repeatedly against an unfixed cause on a production account.

The defect we are telling you about

Earlier in this product's life, rotation had a defect with two halves that combined into the worst outcome available. The component that loaded a schedule read the encrypted credential from the database and then discarded it, so every rotation engine received an empty current password. And the component that recorded the result never wrote the new password back to the vault.

The consequence: a rotation could change the password on the target, record success, and leave the vault holding the old secret. The console showed a green result and a fresh timestamp. The next person to use that credential — or the next brokered session, or the next connection test — authenticated with a password the machine no longer accepted. Customers were locked out of their own systems by a control they had switched on to make themselves safer, and the product told them everything was fine.

It is fixed, on both halves, and the fixes are structural rather than careful:

• The current secret is now actually decrypted before being handed to the engine.
• The new secret is sealed into the vault before a result may be written as success. If sealing fails, the status is downgraded to failed and the reason says exactly what has happened rather than something vague.
• The worker refuses to start if it has no means of opening and storing credentials, rather than running in a degraded state. The reasoning, in the code's own words: a rotation scheduler that cannot open or store a credential can only produce failures or, worse, desynchronised vault rows — and running it in that state is not better than not running it.
• The API path refuses too, with rotation cannot run in this process: it was started without the credential sealer.
• A build test enforces the wiring. If the scheduler ever loses its call site again, the build fails, on the stated grounds that scheduled rotation would never fire and every rotation schedule row would be decorative.

If the vault and the target ever do diverge, you will be told in these words rather than shown a success:

the target's password was changed but Qorionix could not store the new secret: the
  credential sealer was not installed at start-up. The vault and the target now
  disagree.

the target's password was changed but Qorionix could not store the new secret (%v).
  The vault and the target now disagree; rotate again or reset the account
  manually.

Why publish this at all? Because the defect is the kind that a vendor can quietly fix and never mention, and because you are about to let software change the passwords on your most important machines. Knowing what went wrong before, and knowing the specific test that proves it is not happening now, is worth more to you than an assurance. Our own documentation records overstating capability as this repository's dominant historical defect; the correction has to include the awkward parts or it is not a correction.

Two residual defects worth knowing

Both are live, both are narrow, and both are better in your hands than in ours alone.

A rotation can fail after the password has already changed, and lose the new secret. Three engines are able to return a new secret and an error at the same time — the AWS IAM key engine when it cannot clean up the old key, the Entra service principal engine when its cleanup is incomplete, and the Palo Alto engine when the change was written but not committed. In that situation the error is recorded and the new secret is discarded rather than stored. The Palo Alto case is the sharpest: the new password hash is sitting in the device's candidate configuration, so whoever next commits a change on that firewall activates a password that exists nowhere. Details in Rotating cloud credentials and Rotating network device credentials.

Palo Alto rotations are reported as unverified even though they were verified. The engine does re-authenticate after committing the change. The flag that records verification is computed from a different list which does not include it, so every successful Palo Alto rotation shows the sentence Qorionix changed the password but could not sign in with the new one to confirm it. when in fact it did. Cosmetic, misleading, and named here so you do not chase it.

The rotation jobs view

The jobs list answers 'what is about to happen and what just happened' in one chronological view, by merging schedules that are due with runs that have completed. There is deliberately no jobs table behind it — a job is not a thing the system stores, it is a view over two things it already stores, and inventing a third record to keep in step with them would be a source of drift rather than clarity.

• Each entry has a kind of scheduled or completed.
• Identifiers are prefixed schedule: or run: so the two identifier spaces cannot be mistaken for each other.
• Only active schedules contribute a pending entry. A paused schedule has no next run, and showing one would claim a rotation is queued when it is not.
• The failure case returns HTTP 500 with could not list rotation jobs: followed by the detail.

What to do next

Read the article for the platform you are about to rotate. Each one names what breaks, which is the part you cannot get from the console.

1. Rotating Linux and SSH credentials — the best place to start, because it verifies itself.
2. Rotating database credentials — and the connection-pool consequences.
3. Rotating Active Directory and Windows credentials — the highest-risk group; read it before, not after.
4. Rotating cloud credentials — including what is only mock-tested.