Rotating database credentials

Five rotation engines cover databases: postgres_role for a PostgreSQL role, sqlserver_login for a SQL Server login including sa, mysql_user for a MySQL or MariaDB user, oracle_user for an Oracle database user, and generic_db, which routes to one of the others based on an engine name.

Databases are the best-tested rotation surface in the module. Both named engines change the password and then reconnect with the new one to prove it works, and both have been exercised against real database servers in containers rather than only against mocks. If a database rotation reports success, the password in your vault is the password the server accepts.

The wiring state, said plainly

You may encounter an internal Qorionix document — the parity programme tracker — that lists PostgreSQL and SQL Server rotation as built but not wired, and that enumerates only six rotation engines. That document is out of date. We are naming it rather than quietly correcting it, because a stale internal claim that undersells a capability is still a claim you might act on.

So: use postgres_role and sqlserver_login, by choosing the right platform on the target. They are wired, they are selectable, they verify their own work, and they are the ones we have run against real servers.

generic_db is functional but is not derivable from any platform in the catalogue, which means there is no way to select it in the console. Reaching it requires posting the adapter name directly to the rotation schedules API. If you do that, be aware that the adapter name on a schedule is not validated against the registry in the application layer — the only guard is a database constraint that is deliberately not enforced retroactively. A misspelled adapter name will therefore be accepted at creation and fail at every run, with Qorionix has no rotation engine called %q, so this schedule can never run. Recreate it choosing the platform of the target. There is no capability in generic_db that the two named engines do not have, so there is no reason to take that risk.

The 'no executor configured' history

This matters if you are searching for an error you saw, or reading older notes. There was a period in which generic_db was registered with no engine behind it, and a comment saying that start-up would supply one — which nothing did. Every rotation through it failed with:

Because PostgreSQL rotation ran through that same adapter at the time, this broke database rotation entirely, and it broke it in the most expensive way: the schedule looked correct and every run failed. It is fixed. The engine router is now installed when the adapter is constructed rather than afterwards, so an adapter cannot exist in a non-functional state, and a build test fails if one ever can again. The stated reasoning behind that change is worth quoting because it is the principle the whole module was rebuilt on: an adapter that is present in the API and fails at every invocation is worse than an adapter that is absent.

PostgreSQL (`postgres_role`)

Prerequisites, on a target whose platform is postgresql:

• Port — from the target, defaulting to 5432.
• metadata["database"] — the database to connect to, defaulting to postgres. This is the connection database, not the database whose users you are changing; roles are cluster-wide in PostgreSQL.
• metadata["sslmode"] — defaulting to prefer. Set it to require or stronger if your server supports TLS, because prefer will silently accept an unencrypted connection.
• Permission — either the role can alter itself, or you supply an administrator.
• The connection timeout is 15 seconds.

There are two ways the change can be authenticated, and which one you get depends on the target's metadata:

Self-service (the default)

Qorionix connects as the role being rotated, using the current vault secret, and the role changes its own password. The role needs ALTER ROLE on itself. Simple, and it means no additional privileged credential has to exist anywhere.

Administrator path

If metadata["admin_user"] and metadata["admin_password"] are present on the target, they are used to connect instead. That account needs CREATEROLE or superuser. Use this when the role being rotated cannot alter itself, or when you are rotating a role that cannot log in at all.

The change itself is ALTER ROLE <role> WITH PASSWORD '<new secret>', with the role name quoted as an identifier and the password checked against a character allow-list before it is interpolated. The engine then opens a fresh connection as the rotated role and runs SELECT 1. Only if that returns does the rotation count as successful.

postgres: target host is required
postgres: %w
postgres connect as %q: %w
postgres ALTER ROLE %q: %w
postgres: password changed but the new secret does not authenticate for %q: %w
postgres: verification query failed for %q: %w
postgres: verification query returned %d

Three more come from the password validator, which refuses to build unsafe DDL rather than trusting itself:

refusing to set an empty password
generated password is %d characters, beyond the 128 this engine will interpolate
generated password contains an unsafe character %q at position %d; refusing to
  interpolate it into DDL

You should never see the validator errors, because Qorionix generates the passwords itself and generates them within the rules. They exist as a belt-and-braces refusal: if a future change to password generation ever produced something that could alter the meaning of the statement, the rotation fails loudly instead of executing it.

postgres connect as %q: is the one you are most likely to meet, and it means the connection or the authentication failed before anything was changed. Nothing on the server has been altered when you see it. postgres: password changed but the new secret does not authenticate is the serious one: the ALTER ROLE reported success and the reconnect failed. Treat that as an incident on the role — reset it by hand, correct the vault entry, and find the cause before rotating again.

SQL Server (`sqlserver_login`)

Prerequisites, on a target whose platform is sqlserver:

• Port — from the target, defaulting to 1433.
• metadata["database"] — the initial catalog for the connection.
• metadata["trust_server_certificate"] — defaults to true. The server's certificate is accepted without validation unless you set this to false. Worth knowing and worth changing where your instance presents a certificate you can validate.
• metadata["allow_unencrypted"] — defaults to false, and see the callout below.
• Permission — self-service (the login alters itself) or an administrator via admin_user and admin_password, exactly as with PostgreSQL.
• The timeout is 25 seconds.

The change is ALTER LOGIN [login] WITH PASSWORD = '<new secret>', followed by a fresh login as the rotated account to prove it. CHECK_POLICY is deliberately left at the server default, which means your instance's own password policy still applies — the engine does not bypass it. The generated secret is shaped to satisfy a standard complexity policy: 24 random bytes rendered as 32 characters spanning upper case, lower case and digits.

sqlserver: target host is required
sqlserver: target account (login name) is required
sqlserver connect as %q: %w
sqlserver ALTER LOGIN %q: %w
sqlserver: password changed but the new secret does not authenticate for %q: %w
sqlserver dial %s: %w
sqlserver %s returned unknown encryption option 0x%02x
sqlserver TLS handshake: %w
sqlserver: login was neither acknowledged nor refused; the server sent %d bytes
  of tokens
sqlserver exec: %w

The protocol-level messages — the encryption option, the TLS handshake, the acknowledgement one — come from Qorionix speaking the SQL Server wire protocol directly rather than through a vendor driver. If you meet one, it is worth a support ticket with the raw text, because they indicate a server behaviour we have not seen rather than something you have misconfigured.

MySQL and MariaDB (`mysql_user`)

This engine speaks the MySQL wire protocol directly rather than going through a client library. It supports both mysql_native_password and caching_sha2_password authentication — the fast-path variant of the latter always, and the full-authentication variant only over a TLS connection, because that path sends the password in the clear otherwise.

The change is ALTER USER ... IDENTIFIED BY, verified with a DO 1 statement over a fresh connection as the rotated user rather than a SELECT 1 — the engine does not decode a full result set, so it uses the smallest statement that proves a successful authentication.

Oracle (`oracle_user`)

Oracle's own wire protocol (TNS/Net8) has no public specification, unlike PostgreSQL, SQL Server and MySQL — which is why this engine departs from the others' hand-rolled-client approach and shells out to sqlplus instead. Oracle Instant Client must be installed and on the PATH of the machine running the rotation worker, not the target database server. If it is missing, the rotation fails with a clear, actionable error telling you to install it, rather than a cryptic one.

Engine routing, if you use the API path

generic_db chooses its destination from metadata["engine"] on the target. The accepted values are exact:

Anything else is refused with generic_db: unsupported engine %q (registered: %s), and the message lists what is available.

What breaks: pools and restarts

Database credentials are almost never used by people. They are used by applications, and applications hold them in two places that behave differently.

The connection pool

Already-authenticated connections are not re-checked when the password changes. Your application keeps working, with no error and no sign that anything happened. The rotation appears to have been free.

The configuration

The old password is still in the configuration file, environment variable or secret store the application reads at start-up. Nothing has updated it.

The consequence is the most misleading failure pattern in this entire section: the rotation succeeds, everything works, and the application fails hours or days later. It fails the moment the pool needs a new connection — because it grew, because a connection was dropped by an idle timeout, because the network hiccuped, or because the application was restarted or redeployed. At that point the failure looks like a deployment problem, and nobody connects it to a password change from last Tuesday.

It gets worse under load. A pool that cannot authenticate typically retries hard, so one failed connection becomes hundreds of failed authentication attempts a minute. On SQL Server, where a login can be subject to a lockout policy, that retry storm can lock the login out entirely — and then even the correct password stops working.

The accounts worth rotating today, then: sa and other administrative logins that people use interactively, the account your database client is configured with, the migration account, the reporting account somebody set up in 2021. Those are also, conveniently, the ones most likely to be shared and least likely to have changed since installation.

What to do next

1. Register the database as a target with the right platform, and run the connection test — on both database platforms the test is a real sign-in.
2. Read the rotation overview for how failures are recorded and why there is no retry.
3. Start with Linux if you want an easier first rotation.