Monitoring & Ops Playbook

Arc exposes operational metrics at /metrics in Prometheus text format, plus a set of JSON endpoints and a built-in time-series ring buffer for deployments without a Prometheus.

This page is the operator's reference: what to scrape, what to alert on, and the handful of metrics that are exported but not yet populated — so you do not build a dashboard on a signal that can never move.

Endpoints

Scrape configuration

scrape_configs:
  - job_name: arc
    metrics_path: /metrics
    static_configs:
      - targets: ['arc:8000']

Neither Helm chart ships a ServiceMonitor or prometheus.io/* annotations, so add scrape configuration yourself. For the Prometheus Operator:

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8000"
  prometheus.io/path: "/metrics"

The signals that matter

Start with these. Each one is verified to move in response to the condition it describes.

Ingest throughput and errors

arc_ingest_records_total      # counter: records accepted
arc_ingest_bytes_total        # counter: bytes accepted
arc_ingest_batches_total      # counter: write requests turned into batches
arc_ingest_errors_total       # counter: rejected writes

# Ingest rate, records/sec
rate(arc_ingest_records_total[5m])

# Ingest error rate
rate(arc_ingest_errors_total[5m])

Per-protocol breakdowns exist as arc_msgpack_* and arc_lineprotocol_* counters.

Query latency and errors

# p95 latency across all HTTP endpoints
histogram_quantile(0.95, rate(arc_http_latency_seconds_bucket[5m]))

# Error rate across every endpoint, including writes and health checks
rate(arc_http_requests_error_total[5m]) / rate(arc_http_requests_total[5m])

There is no arc_query_latency_* histogram; query latency is available only as a lifetime average from GET /api/v1/metrics. Use the HTTP histogram above for percentiles.

For a query-specific error rate, use the query counters:

rate(arc_query_errors_total[5m]) / rate(arc_query_requests_total[5m])

All four query entry points — /api/v1/query, /api/v1/query/msgpack, /api/v1/query/:measurement and /api/v1/query/arrow — count requests consistently, so success + errors equals requests.

arc_query_timeouts_total and arc_slow_queries_total are wired on both paths and are safe to alert on. arc_slow_queries_total only moves when query.slow_query_threshold_ms is set (it defaults to 0, disabled).

Storage

arc_storage_writes_total       # counter
arc_storage_write_bytes_total  # counter
arc_storage_reads_total        # counter
arc_storage_read_bytes_total   # counter
arc_storage_errors_total       # counter — all backends

All four counters are wired on every backend: local filesystem, S3, and Azure.

rate(arc_storage_errors_total[5m]) > 0

Two gauges describe the most recent backup rather than live traffic:

arc_backup_skipped_files          # gauge — files the last backup inventoried but could not store
arc_storage_unaddressable_files   # gauge — files the last backup could not list at all

arc_backup_skipped_files counts files that were unreadable when the backup copied them (a file compaction or retention removed after the listing) or whose backup destination key would exceed the storage key limit (a source key of 983 bytes or more, which must be renamed). It spans every file group the backup copies and equals the backup status endpoint's skipped_files; the completed backup's manifest and the status endpoint name up to 32 of the skipped data and Iceberg metadata files in skipped_sample (a compaction recovery manifest or an outside-root warehouse file that was skipped is counted here but named only in the log). It is set by every backup that finishes its copy phases, including one the skip ratio then fails, and a clean backup sets it to 0. arc_storage_unaddressable_files counts files present in source storage that no listing can return, so the backup never inventoried them; it is set by every backup whose unaddressable check succeeds, before any copy. Both are gauges held in memory: fixing the files and running the next backup clears them, and a restart resets them to 0 until the next backup runs.

Buffer and flush health

This is where ingest backpressure shows up: records Arc has accepted but not yet written to storage.

arc_buffer_records_buffered       # gauge: records in a buffer, not handed to a flush yet
arc_buffer_records_queued         # gauge: records in flush tasks waiting for a worker
arc_buffer_records_inflight       # gauge: records being written to storage
arc_buffer_queue_depth            # gauge: flush tasks waiting for a worker
arc_buffer_flushes_total          # counter: completed flushes
arc_buffer_records_written_total  # counter: records written by flushes
arc_buffer_flush_failures_total   # counter: flushes that did not reach storage

arc_buffer_flush_failures_total is the most important single alert in an OSS deployment. Every increment means a batch of records did not reach storage and is being held in the WAL for recovery.

# Data is not reaching storage
increase(arc_buffer_flush_failures_total[5m]) > 0

# Backlog is growing: records are arriving faster than they flush
arc_buffer_records_buffered + arc_buffer_records_queued + arc_buffer_records_inflight > 500000

The three record gauges split what ingest holds in memory by stage, and their sum is the total. Alert on the sum: a record handed to a flush leaves arc_buffer_records_buffered, so under backpressure, when the flush queue is full, that gauge alone can read near zero while millions of records wait in the queue.

• A climbing arc_buffer_records_queued with a flat arc_buffer_records_buffered means storage is the bottleneck: flushes are produced faster than they are written.
• arc_buffer_records_buffered also grows when a full buffer finds the queue full and stays where it is; arc_buffer_deferred_buffers counts those buffers.
• arc_buffer_records_inflight also counts synchronous flushes: an explicit flush, the flush before a DELETE, and shutdown.

arc_buffer_records_buffered is sampled once per second; the queued and in-flight gauges change as tasks move. A sawtooth that returns to near zero is normal.

arc_buffer_records_queued and arc_buffer_records_inflight were added in v27.01.1. On an earlier version, arc_buffer_queue_depth counts queued flush tasks, not records.

Flush-queue backpressure

arc_ingest_flush_deferred_total    # counter: flushes deferred because the queue was full
arc_buffer_deferred_buffers        # gauge: buffers currently holding deferred records

arc_ingest_flush_deferred_total counts the events, not the records. It increments every time a buffer crosses max_buffer_size while every flush-queue slot is taken, which under sustained backpressure is close to once per write to a hot measurement — so treat the rate as a saturation signal, not a volume.

arc_buffer_deferred_buffers is the one to alert on. It is the number of buffers whose records no worker could take, sampled once per second. A value that climbs and stays up means storage is not keeping up with ingest:

# Flush workers cannot keep up with ingest
arc_buffer_deferred_buffers > 0 and rate(arc_ingest_flush_deferred_total[5m]) > 0

# Saturated for a sustained period
avg_over_time(arc_buffer_deferred_buffers[10m]) > 10

A deferred buffer is re-enqueued as soon as a flush worker frees a queue slot, so in a healthy system this gauge returns to zero within roughly one flush duration. If it stays high while arc_buffer_queue_depth is below ingest.flush_queue_size, flushes are failing rather than queueing — check arc_buffer_flush_failures_total.

Deferred records are held in memory, and Arc does not bound that: max_buffer_size is a per-measurement trigger, so worst-case held memory is roughly active_measurements x max_buffer_size x bytes_per_record. Tuning it is yours — lowering ingest.max_buffer_size or ingest.max_buffer_age_ms makes flushes smaller and more frequent, at the cost of more Parquet files for compaction to merge. Watch arc_buffer_records_buffered alongside the process's resident memory, and note that the records gauge is not a memory figure: bytes per record vary by more than 100x between a narrow numeric schema and a wide one with strings.

WAL

arc_wal_records_preserved_total            # records that fell back to WAL instead of flushing
arc_wal_dropped_entries_total              # WAL buffer full — data loss risk
arc_wal_failed_writes_total                # WAL entries that could not be written
arc_wal_disk_full_writes_total             # ... of which the disk was full (v27.01.1+)
arc_wal_recovered_writes_total             # WAL writes that failed, then succeeded on a new file (v27.01.1+)
arc_wal_disk_pressure                      # 1 while writes are refused for low WAL volume space (v27.01.1+)
arc_wal_disk_free_bytes                    # free space on the WAL volume
arc_wal_disk_pressure_rejections_total     # requests refused for WAL volume space
arc_wal_disk_stat_errors_total             # failed free-space reads (writes are let through)
arc_wal_oversized_payloads_total           # payload exceeded WAL limits
arc_wal_recovery_total                     # recovery runs
arc_wal_recovery_records_total             # records replayed
arc_wal_quarantined_files_total            # WAL files renamed to *.wal.failed after repeated replay failures
arc_wal_quarantine_released_entries_total  # unflushed entries the node stopped waiting for because their file was quarantined (v27.01.1+)

A rising arc_wal_quarantined_files_total means some WAL entries could not be replayed: they are only in the *.wal.failed files. See Files that keep failing are quarantined.

# Data is being dropped before it reaches the WAL
rate(arc_wal_dropped_entries_total[5m]) > 0

A rising arc_wal_records_preserved_total means flushes are failing or the flush queue is full, and records are surviving only because the WAL is on. If wal.enabled is false, those records are simply lost.

Compaction

arc_compaction_jobs_total
arc_compaction_jobs_success_total
arc_compaction_jobs_failed_total
arc_compaction_jobs_interrupted_total
arc_compaction_partitions_skipped_too_wide_total
arc_compaction_partitions_too_wide
arc_compaction_files_compacted_total
arc_compaction_bytes_read_total
arc_compaction_bytes_written_total
arc_compaction_manifests_recovered_total
arc_compaction_manifests_parked_unparseable_total
arc_storage_invalid_path_quarantined_total

rate(arc_compaction_jobs_failed_total[1h]) > 0

arc_compaction_partitions_too_wide (v27.01.1+) is a gauge of the partitions this node leaves uncompacted because they fail at the smallest batch (see the compaction page); arc_compaction_partitions_skipped_too_wide_total counts each cycle that left one alone. A partition that never compacts never moves to cold storage:

arc_compaction_partitions_too_wide > 0

The job counters count attempts, as /api/v1/compaction/stats does:

• A batch that fails and is retried at half size counts twice, so one partition that cannot be compacted can add several failures per cycle.
• arc_compaction_jobs_total is succeeded + failed + interrupted.
• An attempt cut short by a shutdown or a cancelled cycle is interrupted, not failed.
• A staged job (clusters with compaction commit) counts as succeeded once its output is staged, before the cluster commits it.

arc_compaction_manifests_parked_unparseable_total (v26.09.2+) should sit at zero. Growth means recovery parked a crash-recovery manifest whose body did not decode (typically a zero-length file left by a crash) under the .quarantined suffix in _compaction_state/, so it stopped blocking compaction without being completed. The parked file name gives the tier, database and job; check that partition for a zero-length _compacted file or for duplicate rows.

arc_storage_invalid_path_quarantined_total (v26.09.2+) is broader: it counts any entry dropped from a compaction, tiering, reconciliation or replication work set because a stored key (a compaction input or manifest, a cluster manifest entry, an edge sync ledger row) names something no storage backend can address. The condition is permanent, so the entry is not retried; find the Error log line naming the key and act on it.

Sustained compaction failure degrades query performance as small files accumulate, and on Enterprise it silently blocks tiering — only _daily.parquet files migrate to cold storage.

Memory and runtime

arc_memory_alloc_bytes
arc_memory_heap_alloc_bytes
arc_memory_sys_bytes
arc_goroutines
arc_gc_cycles_total

arc_memory_sys_bytes is the figure to compare against a container limit. Note that Go returns memory to the OS lazily, so RSS lags real usage.

DuckDB connection pool

arc_db_connections_max
arc_db_connections_open
arc_db_connections_in_use
arc_db_connections_idle
arc_db_wait_count_total
arc_db_wait_seconds_total

Saturation is arc_db_connections_in_use / arc_db_connections_max. Sustained near 1 means queries are queueing for a connection rather than executing.

The wait counters are the signal that saturation is actually costing you something:

rate(arc_db_wait_seconds_total[5m]) / rate(arc_db_wait_count_total[5m])

This is mean time blocked per waiting query. A rising arc_db_wait_count_total with a flat in_use well under max points at a pool sized below its configured maximum rather than at query load.

arc_db_connections_max reflects the effective pool size, which Arc derives from CPU count unless you set it, so scrape it rather than assuming the configured value.

Health and readiness

The two probes answer different questions and are not interchangeable.

/health always returns 200, deliberately, even when storage credentials have expired. The reasoning: restarting a pod does not fix expired credentials, it just loops the node through restarts and hides the problem. The response body still carries the detail:

{
  "status": "ok",
  "storage": { "hot": { "backend": "s3", "credentials": "static", "state": "ok" } },
  "uptime_sec": 3600.5
}

Alert on storage.*.state from the body, not on the HTTP status.

/ready returns 503 in exactly two cases:

1. The node is starting up (before WAL recovery completes) or shutting down.
2. Storage credentials are expired and server.storage_credentials_fail_ready is true.

Alerts worth paging on

Every rule below uses a metric verified to move. Thresholds are starting points — tune them to your workload.

Audit logging

When audit logging is enabled (audit_log.enabled), three counters cover it:

arc_audit_events_total          # counter: events committed to the audit table
arc_audit_write_errors_total    # counter: events that failed to persist
arc_audit_events_dropped_total  # counter: events discarded before queueing

increase(arc_audit_events_dropped_total[5m]) > 0

Metrics removed in v26.09.2

These were exported with HELP and TYPE strings but had no increment path, so they read 0 forever regardless of what the system was doing. A counter that cannot rise is worse than an absent one: it answers "is this happening?" with a confident, permanent no. They were removed rather than left in place.

One metric is wired everywhere but has a deliberate exclusion:

Without Prometheus

For air-gapped or edge deployments, Arc keeps its own in-memory history — no external system required.

curl 'http://arc:8000/api/v1/metrics/timeseries/application?duration_minutes=30'

Valid types are system, application, and api only; anything else returns 400 with the valid list. duration_minutes accepts 1–1440 and defaults to 30.

Retention is governed by metrics.timeseries_retention_minutes (default 30) and metrics.timeseries_interval_seconds (default 5), giving 360 sample points.

Operational traps

Things that make a dashboard look healthy when it is not.

A query for a measurement that does not exist is not an error. It returns HTTP 200 with success: true and zero rows, because Arc resolves measurements as a path glob at read time. No error counter moves. A dashboard that silently goes empty may be a typo in a measurement name, not an outage.

/api/v1/metrics/query-pool has two similar error fields. query_errors is the API-level count (real); query_errors_total maps to a DuckDB connection-level counter that is never incremented. Pick the wrong one and the panel reads zero forever. The endpoint's queries_total is likewise always 0: query execution does not route through the handle it counts. Its connections_* fields are populated as of v26.09.2, but prefer /metrics over this endpoint — the Prometheus surface has the pool gauges plus the wait counters this endpoint does not expose.

/api/v1/metrics/endpoints has no per-route breakdown despite the name. It groups counters by subsystem.

Daily compaction and retention share a cron slot. Both default to 0 3 * * *, so they contend for I/O at 03:00. Stagger one if that window is tight.

HTTP write timeout is silently raised at startup. If query.timeout (default 300s) exceeds server.write_timeout (default 30s), Arc raises write_timeout to match and logs a warning — so the effective value is not the configured one. Setting query.timeout = 0 disables that sync and leaves long queries to be cut off at 30s.

WAL is off by default in the binary. wal.enabled defaults to false for backwards compatibility, and a crash without it loses everything buffered since the last flush. Every shipped Docker Compose file and both Helm charts now enable it, but a hand-written config or manifest must set ARC_WAL_ENABLED=true explicitly. Confirm with arc_wal_recovery_total — a node that has never recovered and never written a WAL file is a node running without one.

Raise server.shutdown_timeout if your storage is slow. It bounds the whole graceful shutdown: when it expires, a buffer flush that has not finished is abandoned. Match it to terminationGracePeriodSeconds in Kubernetes rather than leaving both at 30s if flushes to your object store can take longer.

Probe configuration

What the Helm charts ship:

startupProbe:
  httpGet: { path: /health, port: http }
  periodSeconds: 10
  failureThreshold: 30      # 5 minutes for WAL recovery

livenessProbe:
  httpGet: { path: /health, port: http }
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet: { path: /ready, port: http }
  initialDelaySeconds: 5
  periodSeconds: 5

The startupProbe matters: WAL replay happens before the HTTP listener binds, so without a startup gate a slow recovery looks like a liveness failure and the pod crash-loops.

Related

• Backup & Restore
• Telemetry
• Profiling
• Data model