Rotating Linux and SSH credentials

The Linux engine, linux_ssh, changes the password of a local account over SSH and then reconnects with the new password to prove the change took effect. That last step makes it the most trustworthy rotation engine in the module and the right place to start if you are new to this. If a Linux rotation reports success, the password in your vault is the password the machine accepts — it has been demonstrated, not assumed.

This article covers what the engine needs, what it does not do, and the things in your estate that break when a Linux password changes.

Prerequisites

• A linux target with the address of the machine and a reachable SSH port. The port comes from the target and defaults to 22.
• A password credential attached to that target, with the account name recorded. SSH keys are a different case and are covered below.
• The current password in the vault must be correct. The engine signs in as the account using the stored secret before it changes anything.
• The account must have a usable shell. An account whose shell is /sbin/nologin or /bin/false will authenticate over SSH and then be unable to run anything, so rotation cannot work on it.
• chpasswd must be available on the host — it is part of the standard shadow utilities on every mainstream distribution — and the account must be able to run it, either as root or through sudo.

Run the connection test before you enable rotation. On a linux target the test is a genuine SSH password login, and it goes one step further than authenticating: it opens a session without running a command. That extra step exists precisely to catch the nologin case, because an account that authenticates and then cannot open a session would give you a green test followed by a failing rotation, which is the worst combination available.

How the rotation works

1. 1

   24 random bytes become the new password

   Generated by Qorionix. You do not choose it and no route in the product will show it to you.

2. 2

   Qorionix signs in over SSH as the account itself

   Password authentication, using the current secret from the vault. There is no separate administrative credential — the account changes its own password.

3. 3

   The new password is written to standard input, not to a command line

   The engine opens a session, runs chpasswd, and writes the single line account:newsecret to the command's standard input. This matters more than it looks: a password passed as a command argument is visible in the process table to every user on that machine for as long as the command runs. Standard input is not.

4. 4

   Optionally through sudo

   If the target's metadata sets sudo, the command becomes sudo -n chpasswd. See the note on sudo below, because the -n changes what your sudo rule has to look like.

5. 5

   The engine reconnects with the new password

   A fresh SSH connection, authenticating with the secret it just set. Only if that succeeds is the rotation recorded as successful and the new secret sealed into the vault.

Three optional metadata keys change the behaviour, and all three are set on the target:

`sudo`

Run the password change through sudo -n rather than directly. Needed whenever the account being rotated is not root and cannot write to the shadow file itself.

`change_command`

Override the command used to change the password. For unusual hosts where chpasswd is absent or behaves differently. Whatever you supply must accept account:password on standard input.

`skip_verify`

Skip the reconnect-and-prove step. Do not set this. It removes the single property that makes this engine worth trusting, and it turns a Linux rotation into a Windows rotation. The only defensible use is a host where a successful password change legitimately prevents an immediate second login, and if you have one of those, tell us about it.

SSH keys versus passwords

A linux target accepts two credential types: password and ssh_key. Both now rotate, through two different engines — linux_ssh for a password, linux_ssh_key for a key — and it is worth understanding how the key engine avoids the obvious way this could lock you out.

How SSH key rotation proves itself before it removes anything

linux_ssh_key generates a fresh ed25519 keypair and appends the new public key to authorized_keys, authenticated with the current (old) key. At that point the account has two working keys. It then opens a completely fresh connection using only the new key to prove it actually works — the same prove-before-you-trust discipline the password engine uses when it reconnects with the new password. Only after that connection succeeds does the engine remove the old key's line from authorized_keys.

If retiring the old key fails after the new key has already been proven live, the rotation does not report a plain failure: it returns a distinct partial-rotation state saying the new secret is live and the old one was not retired, so you know to go and remove the stale key by hand rather than assuming the account is back to one key. This is the same honest partial-outcome reporting the rest of the rotation module uses elsewhere — a rotation is never silently recorded as fully successful when it was not.

Before this engine existed, an ssh_key credential in the vault was stored, encrypted, access-controlled, audited and usable for a brokered session, but its rotation state said Qorionix can store credentials for Linux server but cannot rotate them automatically. That gap is closed for ed25519 keys.

The practical recommendation is unchanged for anything this engine does not cover: use keys for automation, rotate the password of the interactive administrator account with linux_ssh, and prefer a proxied session so nobody needs either — see Proxied SSH sessions.

What happens to sessions that are already open

Nothing. Changing a Unix account password does not affect connections that have already authenticated. An SSH session open at the moment of rotation stays open, keeps its shell, and keeps whatever it was doing. The password is checked at login and not again.

That has a pleasant consequence and an unpleasant one. Pleasant: a rotation cannot interrupt an engineer mid-task. Unpleasant: a rotation does not evict anybody. If you are rotating because you believe a password has leaked, the rotation stops the next sign-in — it does not close a session an attacker already has. Closing that requires terminating the session on the host, or terminating the brokered session in Qorionix if it came through the platform.

The same applies to a brokered session in progress: it was authenticated when it opened and continues on the connection it already has.

What breaks

Linux is kinder than Windows here, because far less caches a password. The things that do:

• Scripts that authenticate with a password. Anything using sshpass, an expect script, or a password in a configuration file. These fail at next run, and if that run is a nightly backup you find out days later.
• Ansible or configuration management using password authentication. Key-based playbooks are unaffected; --ask-pass workflows with a stored password are not.
• Monitoring or backup agents signing in as the account. Same failure, same delay in noticing.
• SFTP or SCP integrations with a partner. The password is often on the other party's side, where you cannot update it and may not know it exists.
• A saved password in somebody's SSH client or file manager. Trivial, and the person will tell you.

Nothing on that list is visible to Qorionix, so nothing on that list can be checked for you. The remedy is the same as everywhere in this module: find out what uses the account before you rotate it, and prefer accounts used by people over accounts used by machines.

The errors, verbatim

linux_ssh: host and account are required
linux_ssh dial: %w
linux_ssh session: %w
linux_ssh %s: %w
linux_ssh: the password was changed but the new secret does not authenticate
  for %q: %w

`linux_ssh: host and account are required`

The credential is missing its target or its account name. Attach it to a target and record the account; the credential's own rotation summary will say the same thing in longer words.

`linux_ssh dial:`

The SSH connection itself failed — no route, connection refused, timeout, or the credential was rejected. The wrapped text underneath distinguishes them, and note that authentication failures are classified as authentication failures rather than as network problems, so the message you are shown will not send you to open a firewall port that is already open.

`linux_ssh session:`

Authentication succeeded and the account could not open a session. Almost always a shell of /sbin/nologin or /bin/false, or a hard resource limit on the account.

`linux_ssh %s:` with the command name

The password-change command ran and returned a failure. The host's own message follows. The two common causes are the account lacking permission to change the password (fix the sudo rule) and the host's password quality rules rejecting the generated secret.

`linux_ssh: the password was changed but the new secret does not authenticate for %q`

The important one. The command reported success and the reconnect failed, so the machine and the vault may now disagree. The rotation is recorded as failed rather than as success, which is correct — but the password on the host may have changed anyway. Treat this as an incident on that account: reset it at the console, update the vault entry, and find out why the new password was not accepted before rotating it again.

One honest note on our own test coverage: this engine's automated tests are live tests, which means they only run against a real SSH host supplied by our own test environment rather than on every build. The verification step inside the engine is what actually protects you day to day, and it is the strongest such protection in the module.

What to do next

1. Rotate one non-critical Linux account and read the result. A Last rotated sentence with no verification caveat is the outcome you want.
2. Run a connection test with the stored credential afterwards for an independent end-to-end confirmation that the vault and the machine agree.
3. Set up proxied SSH so your team stops needing the password at all.
4. Move on to databases, which verify themselves in the same way.