Upgrading pre-26.06.1 API tokens

Applies to Arc Enterprise clusters that carry API-created tokens from before 26.06.1. Before that release, every node kept its own SQLite auth database, so a token created through the API existed only on the node that issued it. 26.06.1 moved token creation and revocation onto the replicated Raft path, but pre-existing tokens were not migrated. They keep working on their home node until an operator replaces them.

Plan a maintenance window and preserve administrative access throughout. Bootstrap tokens configured via ARC_AUTH_BOOTSTRAP_TOKEN with the same value on every node are unaffected.

Maintenance procedure

1. Inventory active tokens on every node before upgrading: record node, local token identity, name, permissions, owner, and consumers. Preserve enough non-secret identity to distinguish legacy rows from their replacements; a numeric ID alone is insufficient. Do not export hashes or plaintext into the inventory.
2. Back up each node's auth database with the node stopped: the file configured by auth.db_path, including SQLite -wal and -shm sidecars when present. Keep the backup access-restricted. Never copy or delete the live database while the node is writing.
3. Upgrade all nodes to 26.06.1 or later. Restore stable leadership and full membership before issuing replicated tokens.
4. Validate the replicated path with a short-lived token created through POST /api/v1/auth/tokens. Confirm it authenticates on every node and that arc_cluster_auth_apply_create_total increments to match on every node. Revoke this new token through POST /api/v1/auth/tokens/:id/revoke and verify it is rejected everywhere.
5. Re-issue each inventoried token through POST /api/v1/auth/tokens on any node, preserving its scope and permissions. Capture the returned plaintext once into the approved secret store. Verify it on every node before rotating any consumer. If an ID collision prevents materialisation, stop and resolve the divergence (see below) before proceeding.
6. Rotate downstream consumers (CI secrets, SDKs, dashboards) to the replacement. Verify each consumer against multiple nodes.
7. Remove the old token locally on every node that holds it. Stop the node, take a fresh backup, and remove only the positively identified legacy rows from its local auth database, or use the supported rebuild from replicated FSM state. Do not select rows by ID alone, and do not remove a replicated replacement. If identity cannot be established, stop and seek support rather than guessing. Restart and verify that replacement tokens still authenticate and legacy tokens do not. Keep quorum available during node maintenance, or use a planned full-cluster outage. Repeat steps 5 through 7 one token at a time.
8. Verify completion: every active consumer uses a replacement, apply counters converge across nodes, arc_cluster_auth_rejected_total is stable, and every legacy token is rejected on every node.

Divergence and rollback

On a collision or a growing arc_cluster_auth_rejected_total, stop the affected node and preserve its backup. Reconcile only the conflicting legacy rows, or rebuild the node's local auth database from authoritative FSM state before it rejoins. Never remove Raft state to repair the SQLite cache, and do not invent SQL cleanup by token ID.

Restoring a pre-upgrade database can re-enable legacy credentials and diverge from replicated state. Treat rollback as a coordinated maintenance operation: keep the affected node out of client traffic until its state and credentials have been reconciled and verified. Retain backups until the completed migration is accepted.

Background

The 26.06.1 release notes cover the auth replication design, ID-collision detection, and the related metrics. The source issue is #457; its original suggestion to revoke legacy tokens through the API does not apply to unmigrated rows.