Monitoring & Ops Playbook
Operational metrics for Arc Enterprise clusters: replication lag, tiering, governance, audit, per-role expectations, failover signals, and endpoints for inspecting or killing queries.
Arc Enterprise exposes everything in the OSS monitoring playbook plus the cluster, replication, governance and query-management signals documented here.
Read the OSS page first
Ingest, query, storage, buffer, WAL and compaction metrics are identical in both editions, as are /health and /ready semantics. This page covers only what Enterprise adds.
Per-role expectations
A cluster runs three roles, and a metric that is healthy on one is a problem on another. Scrape all of them, but alert per role.
| Metric | Writer | Reader | Compactor |
|---|---|---|---|
arc_ingest_records_total | Rising | Flat | Flat |
arc_query_requests_total | Low | Rising | Flat |
arc_compaction_jobs_total | Flat | Flat | Rising |
arc_buffer_records_buffered | Meaningful | See below | Near zero |
arc_buffer_deferred_buffers | Meaningful | See below | Near zero |
arc_wal_* | Meaningful | Recovery only | Near zero |
Ingest arriving at a reader, or queries served by a compactor, usually means a load balancer is routing to the wrong pool. The compactor cannot serve queries locally and forwards them.
A reader's ingest buffer is not idle (up to v26.09.3)
Applies up to v26.09.3 only. From v27.01.1 readers no longer receive the WAL replication stream, so a reader's ingest buffer is idle and arc_buffer_records_buffered reads near zero (arc#888).
Up to v26.09.3, arc_buffer_records_buffered on a reader is not expected to be near zero when WAL replication is enabled. A reader applies replicated entries into its own ingest buffer, which means it carries the writer's write volume through its own buffer, flush queue and storage backend.
On those versions, two consequences when alerting:
- A reader whose storage was slower than the writer's accumulated a backlog of its own, visible on
arc_buffer_records_bufferedandarc_buffer_deferred_buffers, even though the reader accepted no client writes. - The reader had no way to push back. Client writes were forwarded to a writer rather than admitted locally, and replicated entries were applied unconditionally. So a reader that could not keep up grew its backlog rather than shedding load.
On those versions, alert on both gauges for readers as well as writers, and size reader storage against the writer's sustained ingest rate rather than against its own query load.
Cluster state
arc_cluster_manifest_rejected_paths_total # counter: manifest entries refused
arc_cluster_auth_rejected_total # counter: auth FSM commands refused
arc_cluster_rbac_rejected_total # counter: RBAC FSM commands refused
arc_cluster_rbac_cascade_rejected_total # counter: cascade limit exceeded
arc_cluster_local_delete_pending # gauge: manifest deletes not yet unlinked locally (per-node storage with replication)Rejection counters are security canaries
increase(arc_cluster_manifest_rejected_paths_total[1h]) > 0
increase(arc_cluster_auth_rejected_total[1h]) > 0
increase(arc_cluster_rbac_rejected_total[1h]) > 0These fire when a node proposes a Raft command the FSM refuses — a malformed path, an invalid auth mutation, an RBAC change that exceeds cluster.rbac.max_cascade_descendants. In a healthy cluster they stay at zero forever, so any movement is worth investigating rather than tuning.
The arc_cluster_auth_apply_* and arc_cluster_rbac_apply_* counters record applied FSM commands (create, update, delete, revoke, rotate, and the RBAC organization/team/role/permission operations). They are per-node and every healthy node converges to the same value, because all nodes apply the same Raft log.
Divergence between nodes is the signal
The absolute values matter less than their agreement. A node whose arc_cluster_*_apply_* counters lag the others is not applying the log — check it before trusting anything else it reports.
max(arc_cluster_auth_apply_create_total) - min(arc_cluster_auth_apply_create_total) > 0Replication
Removed in v27.01.1
v27.01.1 removes the WAL replication stream, because the readers and compactors it reached wrote every replicated row a second time (arc#888, arc#1181). The three metrics below are no longer exported (nor replication_entries_dropped_total in the JSON metrics), and cluster.replication_lag_limit, cluster.replication_buffer_size and cluster.replication_ack_interval are ignored: remove alerts built on them. Rows still reach every node through peer file replication. The section below applies to v26.09.3 and earlier.
arc_replication_lag_entries{peer="<reader>"} # gauge: writer entries not yet acknowledged by that reader (v26.09.3+)
arc_replication_lag_seconds{peer="<reader>"} # gauge: age of the oldest of those entries since the writer appended it (v26.09.3+)
arc_replication_entries_dropped_total # counter: sender-side dropsThe two lag gauges are exported by the writer, one series per connected WAL replication reader, read from the live connections at scrape time: a reader that disconnects leaves no stale series, and a caught-up reader reads 0 on both. The age is taken on the writer's clock at both ends, so it does not depend on clock agreement between nodes. They need wal.enabled = true; without the WAL there is no replication stream to measure.
arc_replication_lag_seconds > 30
arc_replication_lag_entries >= <cluster.replication_buffer_size>
rate(arc_replication_entries_dropped_total[5m]) > 0How to read them:
- A healthy reader's
secondssawtooths between0andcluster.replication_ack_interval(100 ms by default). Set the threshold well above that, and above any non-default value of the interval. - Lag is measured from the moment a reader attaches. The sender does not replay entries accepted before that, so a reader that restarts rejoins at
0and counts only the entries that follow; the data it missed while away arrives through file replication, not this stream. - Once a reader is more than
cluster.replication_buffer_sizeentries behind, the writer no longer holds the timestamp of the oldest outstanding entry, andsecondsreports the age of the oldest it still holds: a lower bound, tight while the buffer is full.entriesincludes entries the full buffer dropped, which that reader will never receive on this stream. That is why the second and third rules belong next to the first: they are the saturation signals, and they are never omitted.
A sender drops entries when the replication buffer overflows — the receiver is not keeping up, or the link is saturated. Tune cluster.replication_pull_workers and the fetch/serve timeouts, and check reader health.
arc_replication_sequence_gaps_total was removed in v26.09.2
This metric was exported but nothing detected gaps, so it read 0 regardless of whether entries went missing. It was removed rather than implemented, because the condition it claimed to measure cannot occur silently: the receiver checks checkpoint sequence equality and a cumulative payload hash, so a gap drops the connection instead of passing unnoticed. The counter could only ever report 0, which reads as "no gaps" and is indistinguishable from "not measured". See arc#810.
Detect replication problems from the lag gauges above, arc_replication_entries_dropped_total, reader catch-up state in GET /api/v1/cluster, and 503s from readers with cluster.query_gate_on_catchup enabled.
Readers serving stale results
cluster.query_gate_on_catchup defaults to false. With it off, a reader that has not finished pulling replicated files answers queries anyway, silently returning incomplete results. With it on, the reader returns 503 until it has caught up.
Enable the catch-up gate on any reader that serves correctness-sensitive queries
Silent under-reporting is worse than a retryable error for most workloads. With the gate enabled, alert on readers stuck in 503:
probe_success{job="arc-reader",path="/ready"} == 0 for 10mCheck catch-up progress at GET /api/v1/cluster.
Query governance
arc_governance_rate_limited_total # counter: queries rejected by rate limit
arc_governance_quota_exhausted_total # counter: queries rejected by quota
arc_governance_queries_capped_total # counter: results truncated at the row cap
arc_governance_policies_active # gauge: policies currently loaded# Users are hitting limits
rate(arc_governance_rate_limited_total[5m]) > 0
rate(arc_governance_quota_exhausted_total[5m]) > 0arc_governance_queries_capped_total is the subtle one: those queries succeeded, but returned truncated results. A client that does not check the rows_capped response field will silently under-report. Watch it when onboarding a new tenant, where an unexpectedly low cap looks like missing data rather than an error.
Query management
Requires an Enterprise license with the query_management feature and query_management.enabled = true.
arc_query_mgmt_active_queries # gauge: queries currently running
arc_query_mgmt_cancelled_total # counter: queries killed via the API
arc_query_mgmt_history_size # gauge: entries in the completed-query ringThe two gauges only update when the list API is polled
arc_query_mgmt_active_queries and arc_query_mgmt_history_size are refreshed as a side effect of GET /api/v1/queries/active and /history. There is no background sampler, so if nothing polls those endpoints the values are stale or zero regardless of what is actually running.
Do not treat arc_query_mgmt_active_queries as a concurrency gauge in Prometheus. Either poll the list endpoint on the same interval as your scrape, or use it only for ad-hoc inspection.
Inspecting and killing queries
| Endpoint | Purpose |
|---|---|
GET /api/v1/queries/active | Running queries; SQL truncated to 200 chars |
GET /api/v1/queries/history | Completed queries; ?limit= default 50, max 1000 |
GET /api/v1/queries/:id | One query, full SQL |
DELETE /api/v1/queries/:id | Cancel a running query |
All require admin auth plus the license feature. DELETE returns 409 if the query has already finished, 404 if the ID is unknown.
curl -H "Authorization: Bearer $ARC_TOKEN" https://arc:8000/api/v1/queries/active
curl -X DELETE -H "Authorization: Bearer $ARC_TOKEN" https://arc:8000/api/v1/queries/abc123def456Tiering
Tiering moves aged files from hot to cold storage. There are no dedicated Prometheus counters; observe it through /health, the migration history in the metadata database, and storage-level counters.
Only daily-compacted files ever migrate to cold storage
Tiering skips any file whose name does not end in _daily.parquet. Raw ingest files and hourly-compacted output stay on hot storage indefinitely.
If daily compaction is disabled, or never reaches its file threshold for a low-volume measurement, that measurement's data never tiers — and nothing reports it. You pay hot-storage prices forever with no error.
# Daily compaction must actually be running
increase(arc_compaction_jobs_success_total[24h]) == 0Before v27.01.1 this counter always read 0, so the alert fired all the time; it only means something from v27.01.1 (#1343).
A partition too wide to compact in a job's memory is left uncompacted from v27.01.1, so it never tiers either:
arc_compaction_partitions_too_wide > 0GET /api/v1/compaction/too-wide on the compactor node lists them; see File Compaction.
Also confirm the cold tier appears in /health under storage, alongside hot.
Cold-tier objects are written with no storage class or access tier set — S3 Standard, or the Azure storage account's default tier — and are read in place, so a query touching cold data pays object-store latency and a round trip per file, not a restore. Size tiered_storage.default_hot_max_age_days (default 30) against your actual query patterns, not just storage cost.
On a cluster with peer file replication, each node's tier metadata follows its own disk (since 26.09.3): the puller registers what it pulls, and a copy removed because another node migrated the file is marked cold once the object is confirmed in the cold tier. Two signals say the path is working, and both should agree across nodes within seconds of a file arriving:
tier_registerednext topulledinreplication_catchup_statusonGET /api/v1/cluster— trackspulledon a tiering node, zero on a node without tiering.replication_eventsonGET /api/v1/tiering/status—{applied, dropped, failed};droppedis zero on a healthy node. Non-zerodroppedorfailedmeans rows wait for the next tier scan, and the node loses partition pruning for the affected measurements until then.
GET /api/v1/tiering/files counts converging across nodes is the end-to-end check. The startup tier scan logs Startup tier scan completed with its counts on every boot; Startup tier scan did not complete means the node is relying on the next migration cycle.
Audit logging
Enterprise deployments are the ones most likely to run audit_log.enabled. The counters are documented on the OSS page; the operational point bears repeating here:
increase(arc_audit_events_dropped_total[5m]) > 0Arc drops audit events when its fixed 1000-event queue is full. For a compliance deployment, a gap in the audit trail is a reportable event, so treat any non-zero value as an incident rather than a tuning signal.
Cluster endpoints
| Endpoint | Auth | Returns |
|---|---|---|
GET /api/v1/cluster | Standard | Coordinator status, leader, catch-up progress, license tier |
GET /api/v1/cluster/nodes | Standard | All nodes; ?role= and ?state= filters |
GET /api/v1/cluster/nodes/:id | Standard | One node |
GET /api/v1/cluster/local | Standard | This node's identity and role |
GET /api/v1/cluster/health | Standard | Cluster-wide health |
GET /api/v1/cluster/files | Admin | File manifest (exposes schema and paths) |
DELETE /api/v1/cluster/nodes/:id | Admin | Remove a node — destructive |
Valid ?role= values are writer, reader, compactor, standalone; valid ?state= values are healthy, unhealthy, dead, unknown, joining, leaving. An invalid value returns 400 listing the valid set.
Failover
cluster.failover_enabled defaults to false in the binary, though the Enterprise Helm chart sets it to true. When enabled, a writer that stops heartbeating is replaced after cluster.failover_timeout (default 30s), with a cluster.failover_cooldown (default 60s) before another can occur.
Watch for failover through node state transitions at GET /api/v1/cluster/nodes, and alert on a writer sitting in unhealthy or dead:
probe_success{job="arc-writer",path="/ready"} == 0 for 2mNeither Helm chart ships a PodDisruptionBudget
A three-writer cluster maintains Raft quorum with two nodes. Without a PDB, a node drain or a cluster upgrade can evict two writers at once and stall the Raft log.
Add a PDB with minAvailable: 2 for the writer StatefulSet before running a node drain in production. The chart also rejects writer.replicas: 2 outright, because a two-node Raft group has no failure tolerance.
Deployment notes
Coordinated restarts for wire-format changes. Some releases change the coordinator or replication handshake. A rolling restart then leaves cross-version nodes marking each other unhealthy, which can trigger an unwanted failover. Check the release notes; when flagged, stop all nodes, upgrade every binary, then restart.
Enable server.storage_credentials_fail_ready on reader pools. It defaults to false, so expired storage credentials do not affect readiness and a reader keeps taking traffic while every query fails. See the OSS page.
Match server.shutdown_timeout to your grace period. It bounds the whole graceful shutdown; when it expires an unfinished buffer flush is abandoned. Writers with large buffers and slow object storage need more than the 30-second default, and terminationGracePeriodSeconds should match.