Rotating network device credentials

Network devices are the part of an estate that most often keeps the password it was given on the day it was installed. Nobody wants to touch the firewall. This article covers the device families Qorionix can rotate automatically — Cisco IOS and IOS-XE switches and routers, Palo Alto PAN-OS firewalls, Juniper Junos devices, FortiOS firewalls, and VMware vCenter appliances — and it is specific about the device behaviours that will bite you: enable mode on the Cisco, and the configuration commit on the Palo Alto and the Juniper.

Read Credential rotation, and what it actually changes first if you have not. This article assumes you know what a target is and how a rotation schedule is attached to a credential.

The two device families

When you create a target you pick a platform, and the platform determines which rotation engine runs. Note that the platform key you will see in the API for the firewall is panos, while the engine behind it is named paloalto_panos — they are not a typo for each other.

All five accept only a password credential. None accepts an SSH key or an API token as the rotated secret itself, because none of the rotation paths knows how to replace one. If you hold an API key for a device, store it in the vault as a credential on a generic target and rotate it by hand.

How the Cisco rotation works

Qorionix signs in over SSH as the account whose password is being changed, using the password currently in the vault. It offers both password authentication and keyboard-interactive, because older IOS images will refuse the first and accept the second, and it deliberately re-enables several legacy key-exchange and cipher algorithms that modern SSH clients have dropped. That is not carelessness: a switch running a 2015 image cannot negotiate a 2026 cipher suite, and a rotation engine that cannot talk to your actual estate is decoration.

1. 1

   Sign in and take a terminal

   The engine requires the device to grant it a pseudo-terminal. IOS drives everything through an interactive CLI, so there is no way to run the change without one. A device configured to refuse PTY allocation fails here.

2. 2

   Enter privileged EXEC mode

   This is enable mode — the elevated shell on an IOS device, reached with the enable command and a separate secret. Qorionix uses the value in the target's enable_secret metadata. If you leave that field empty it falls back to the account's own password, which is correct on plenty of devices and wrong on plenty of others.

3. 3

   Turn off paging and enter configuration mode

   It runs terminal length 0 so the device stops emitting --More-- prompts, then configure terminal.

4. 4

   Set the new password

   It generates a fresh random secret and runs username <account> privilege <n> secret 0 <secret>, where <n> is the target's privilege metadata and defaults to 15. Setting this wrong is how you accidentally demote your own administrator account.

5. 5

   Leave configuration mode and save

   It runs end, then the save command — write memory unless you have overridden save_command in metadata, or set skip_save to skip it entirely.

6. 6

   Prove the new password works

   It opens a completely fresh SSH connection and signs in with the new secret. Only then is the rotation recorded as successful and the new secret written into the vault.

One implementation detail is worth knowing because it explains an error message you may see. IOS returns a successful exit status even when it has rejected the command you sent, so the engine cannot rely on exit codes. It matches the device's prompts and scans the returned text for IOS error patterns instead. When it finds one, you get the device's own words back:

cisco_ios rejected "username svc-net privilege 15 secret 0 ...": % Invalid input detected
cisco_ios: enable did not reach privileged EXEC mode; check metadata.enable_secret
cisco_ios: enable did not prompt for a password
cisco_ios: metadata.privilege must be numeric, got "fifteen"
cisco_ios: password changed but the new secret does not authenticate for "svc-net"
cisco_ios pty: ssh: pty-req failed
cisco_ios: no exec prompt after login
timed out after 30s waiting for the enable prompt

Secrets are stripped out of error text before it is stored or shown, so a rejected command in your rotation history does not become a place your new password is written down in plaintext.

How the Palo Alto rotation works

PAN-OS has no shell step. Everything happens over the appliance's XML API on port 443. The sequence is unusual in one respect that is worth understanding, because it is the reason this engine is safer than it might sound.

1. Qorionix exchanges a username and password for a short-lived API key using the keygen call. By default it does this as the account being rotated; if you supply admin_user and admin_password in the target's metadata it uses that administrator instead.
2. It generates a new secret and asks the device to hash it, using the request password-hash operational command. The plaintext password is never written into the device configuration — only the hash the device itself produced.
3. It writes that hash into the candidate configuration at /config/mgt-config/users/entry, against the administrator's name.
4. It issues a commit, then polls the resulting job until the device reports <status>FIN</status>. The commit budget is five minutes, polled every five seconds.
5. After the commit it runs keygen again with the new password. If that succeeds, the rotation is recorded as successful.

Every parameter travels in the POST body rather than the query string, specifically so the password does not end up in the appliance's own access log. That is the kind of detail that separates a rotation that improves your security from one that just moves the secret somewhere new.

The skip_commit metadata option exists and you should treat it as a trap rather than an option. With it set, the engine writes the hash into the candidate configuration and then returns an error:

Because that is an error, the scheduler takes its failure path and never stores the new secret. The hash, however, is already sitting in the candidate configuration. Whoever commits next — for any unrelated reason — activates a password that exists nowhere: not in the vault, not in anyone's notes. This is a real defect in the current release, not a subtlety of PAN-OS, and the safe response is simple: do not set skip_commit.

paloalto_panos: the device returned an empty password hash
paloalto_panos keygen as "admin": response carried no key
paloalto_panos commit job 4271 finished with FAIL: Commit failed
paloalto_panos commit job 4271 did not finish within 5m0s
paloalto_panos: password committed but the new secret does not authenticate for "admin"
PAN-OS returned status "error": Invalid credentials

One further limitation to record: the HTTPS client this engine uses does not verify the appliance's TLS certificate. Most PAN-OS management interfaces present a self-signed certificate and rejecting those would make the feature unusable for almost every customer, but the honest consequence is that the connection is protected against passive eavesdropping and not against an attacker who can already intercept traffic to your firewall's management address. Keep that address off any network you do not control.

Juniper Junos (`juniper_junos`)

The Juniper engine speaks NETCONF over the SSH netconf subsystem (RFC 6242), using NETCONF 1.0 framing. It issues an edit-config with the new password and then a commit, the same two-step change-then-activate shape the Palo Alto engine uses on its own configuration model.

FortiOS (`fortios_admin`)

The FortiOS engine changes an administrator's password through the appliance's REST API (PUT /api/v2/cmdb/system/admin/{name}). Because changing your own password mid-session over a password-based login does not work on FortiOS, this engine authenticates using a delegate metadata.api_token rather than the account's own credential. It verifies the change through FortiOS's own /logincheck endpoint, using the account's new password.

VMware vCenter appliance local accounts (`vcenter_local_account`)

This engine targets the vCenter appliance's own local OS accounts — root and similar — through a vSphere Automation API session login followed by a PATCH /api/appliance/local-accounts/{username}. It is deliberately narrower in scope than "VMware vSphere" generally: it does not touch accounts in an SSO domain, only the appliance's own local account store.

What a connection test can and cannot tell you

Before you enable rotation you can test the target. The two platforms behave differently here, and the difference matters.

For a Cisco target, the test is a real sign-in: Qorionix authenticates over SSH and opens a session. A successful result reads Connected to sw-core-01 on port 22 and signed in as "svc-net". A rejected password reads sw-core-01 answered, but rejected the credential for "svc-net". The password or key stored here is not the one the target expects.

For a Palo Alto target there is no pre-flight sign-in at all. The test falls back to checking that something is listening on the port, and says so plainly:

That is deliberate. A test that always passes is worse than no test, because it turns an unknown into a false assurance. If you need to know the firewall password is right before you trust Qorionix to rotate it, sign in to the web interface yourself once.

The honest status of each engine

Both engines are wired, reachable from the console and run on the schedule you set. They are not equally well proven, and you are entitled to know which is which.

Neither engine retries. A failed rotation is not attempted again until the next scheduled run, which is thirty days later on the default interval. If a rotation fails, use the rotate-now control rather than waiting.

Before you turn it on

• Have a second way in. A local console cable, a second administrator account that Qorionix does not manage, or an out-of-band management path. Rotating the only credential on the only path into a firewall is a bet you do not need to take.
• Fill in enable_secret on Cisco targets rather than relying on the fallback. Getting into enable mode is where most Cisco rotations fail, and the failure message tells you exactly that.
• Check the privilege level. privilege 15 is full administrative access and is the default. If the account is meant to be a level-7 operator, set it, or the rotation will quietly promote it.
• Rotate one device first. Attach one non-critical switch, run rotate-now, and read the history entry before you schedule the estate.

What to do next

1. Add the device as a target and run a connection test.
2. Store the credential in the vault and attach it to the target.
3. Enable rotation and read the schedule — start with a thirty-day interval and one device.