BasekickLabs

Clustering & High Availability

Configure an Arc Enterprise cluster: assign writer, reader, and compactor roles, set seeds and Raft addresses, tune peer file replication and catch-up, and enable writer failover.

Scale Arc horizontally with multi-node clusters. Separate write, read, and compaction workloads across dedicated nodes with automatic failover.

Choose a deployment pattern first

Arc Enterprise supports two cluster topologies: shared object storage and local storage with peer replication. See Deployment Patterns to choose the right one for your environment before configuring a cluster.

Overview

Arc Enterprise clustering uses a role-based architecture where each node in the cluster serves a specific purpose:

Arc Enterprise Architecture

Node roles

RolePurposeCapabilities
writerHandles data ingestion and WALIngest, coordinate
readerServes queries from shared storageQuery
compactorRuns background file optimizationCompact
standaloneSingle-node mode (default)All capabilities
  • Writers receive data via the ingestion API, buffer it, and flush Parquet files to shared storage. The local WAL makes unflushed rows survive a restart.
  • Readers query Parquet directly from shared storage. Scale readers horizontally to handle more concurrent queries.
  • Compactors run hourly and daily file compaction in the background without impacting write or read performance.

Configuration

TOML configuration

[cluster]
enabled = true
node_id = "writer-01"           # Unique identifier for this node
role = "writer"                 # writer, reader, compactor, standalone
cluster_name = "production"     # Cluster identifier
seeds = ["10.0.1.10:9000", "10.0.1.11:9000"]  # Seed nodes for discovery
coordinator_addr = ":9000"      # Address for inter-node communication
health_check_interval = 10      # Health check interval (seconds)
heartbeat_interval = 5          # Heartbeat interval (seconds)
replication_enabled = true      # Peer file replication between nodes
replication_pull_bytes_per_second = 0         # Cap on this node's inbound replication, bytes/s or "10MB"; 0 = unlimited (v27.01.1+)
query_gate_on_catchup = false   # See "Query gating during replication catch-up" below
replication_catchup_enabled = true            # Walk the manifest on startup and pull what is missing
replication_catchup_barrier_timeout_ms = 30000 # Bound on the pre-walk sync with the leader (see below)
stale_local_sweep = true                      # Remove local copies of files deleted while this node missed it (v27.01.1+)
stale_local_sweep_max_fraction = 0.5          # Refuse a sweep pass that would remove more than this share of the node's files
raft_pre_vote = true                          # Raft pre-vote (v27.01.1+); see "Raft pre-vote" below before an upgrade

Environment variables

ARC_CLUSTER_ENABLED=true
ARC_CLUSTER_NODE_ID=writer-01
ARC_CLUSTER_ROLE=writer
ARC_CLUSTER_CLUSTER_NAME=production
ARC_CLUSTER_SEEDS=10.0.1.10:9000,10.0.1.11:9000
ARC_CLUSTER_COORDINATOR_ADDR=:9000
ARC_CLUSTER_HEALTH_CHECK_INTERVAL=10
ARC_CLUSTER_HEARTBEAT_INTERVAL=5
ARC_CLUSTER_REPLICATION_ENABLED=true
ARC_CLUSTER_REPLICATION_PULL_BYTES_PER_SECOND=0
ARC_CLUSTER_REPLICATION_CATCHUP_ENABLED=true
ARC_CLUSTER_REPLICATION_CATCHUP_BARRIER_TIMEOUT_MS=30000
ARC_CLUSTER_QUERY_GATE_ON_CATCHUP=false
ARC_CLUSTER_STALE_LOCAL_SWEEP=true
ARC_CLUSTER_STALE_LOCAL_SWEEP_MAX_FRACTION=0.5
ARC_CLUSTER_RAFT_PRE_VOTE=true

Raft pre-vote (v27.01.1+)

From v27.01.1 a node runs the Raft pre-vote round before it starts an election (arc#1272). A writer that cannot hear the leader, for example a restarted pod whose new address the leader cannot resolve yet, no longer raises its term on every election timeout. Before, when the leader reached it again, the higher term made a healthy leader step down.

Releases before v27.01.1 do not answer pre-vote. While a cluster runs both, a leader that dies can leave one old and one upgraded survivor with no leader until the restarted node rejoins on v27.01.1 (only when the upgraded survivor has the more recent log). Upgrade writers one at a time and wait for each to rejoin. To rule the window out, set cluster.raft_pre_vote = false on every node for the upgrade and remove it, with one more rolling restart, once all nodes run v27.01.1.

Peer replication bandwidth (v27.01.1+)

A node that rejoins with a large backlog pulls every file it is missing at once, and can saturate the network (arc#397). cluster.replication_pull_bytes_per_second caps the inbound bandwidth of that node's peer file replication. All of the node's pull workers share one budget, so new files, the startup catch-up and the periodic recheck together stay under it. It is a per-node setting, so it can be set only where it is needed, such as on a reader that rejoins often.

  • The value is bytes per second, or a size such as "10MB" (1 MB = 1024 KB). 0, the default, is unlimited.
  • A negative or unparseable value fails startup.
  • The cap does not apply to the staged compaction commit, which fetches each compacted output from the start every time and would otherwise never deliver a large one.

A file can take longer than one fetch under a low cap. Each fetch is still bounded by cluster.replication_fetch_timeout_ms (default 60000) on the pulling node and by cluster.replication_serve_timeout_ms (default 120000) on the node serving the file. When either ends a fetch, the next attempt resumes from the bytes already received, including the first attempt of a later pull, so a large file arrives in pieces instead of starting over. A file arrives sooner, with fewer interruptions, when both timeouts are longer than its transfer time at the cap divided among the pull workers (cluster.replication_pull_workers, default 4).

Helm: cluster.replication.pullBytesPerSecond.

Query gating during replication catch-up

In a local-storage cluster with peer replication, a reader node may serve queries before its background puller has finished pulling all the Parquet files the cluster manifest references. Without gating, those queries silently return partial results: the manifest knows about the missing files, but read_parquet() globs against local storage and only finds what's already on disk. Every row reaches a reader as a pulled Parquet file. From v27.01.1 there is no live WAL stream to readers, so a reader sees a row once the writer has flushed it and the puller has fetched the file (arc#888).

cluster.query_gate_on_catchup (added in 26.06.1, off by default) closes the remaining gap. When enabled, all user-facing read endpoints return 503 Service Unavailable until peer file replication has fully converged on this node.

When to enable it

Turn this on if you'd rather a reader return 503 for a few seconds at startup than serve incomplete results. Leave it off if your application can tolerate eventual consistency during catch-up and you'd rather queries always succeed (the existing pre-26.06.1 behavior). Either choice is defensible; this is a correctness-vs-availability knob.

What "fully converged" means

The gate is scoped to the startup catch-up batch only — not to all pull activity on the node. This distinction matters: in a busy cluster, steady-state ingest constantly puts new files in flight, and a naive "wait for everything to settle" predicate would mean the reader returns 503 every few seconds in normal operation. The gate's job is "the reader has finished bootstrapping its view of the manifest as of startup," not "no pulls are happening anywhere right now."

A node is considered ready when all of the following are true:

  1. The startup catch-up walker has finished its pass over the manifest.
  2. No paths the walker tagged are still in flight (catchup_inflight == 0). Steady-state pulls from reactive FSM callbacks are deliberately excluded.
  3. No catch-up-batch pulls failed after retries (catchup_failed == 0).
  4. No catch-up-batch pulls were dropped due to queue saturation (catchup_dropped == 0).

Since 26.09.2 the walker also checks the files this node originated instead of assuming it still holds them: a node that comes back with an empty data disk under the same cluster.node_id pulls its own files back from a peer's replica as part of the same batch, so the gate covers them too. On such a cluster the reconciler's orphan-manifest sweep waits for this convergence before it proposes anything and reports a held sweep as manifest_sweep_held in the run (#959).

Failures and drops outside the catch-up window do not keep the gate red. They're operational concerns surfaced via puller stats but not correctness blockers — by the time the catch-up batch has settled, the reader has reconciled its view of the manifest as of walker start. Steady-state failures are handled by reactive FSM callbacks (which re-enqueue), the Phase 5 reconciler, and operator alerting via the cumulative failed / dropped counters.

Self-heal: catch-up failures and drops both clear without a process restart, in two ways. When a later pull succeeds for a previously-affected path (a reactive FSM callback re-enqueueing after the underlying issue resolves, or a subsequent catch-up scan), the corresponding scoped counter decrements and the gate re-opens automatically. And when the entry is removed from the cluster manifest, whether by retention, compaction, the reconciliation sweep, or an operator, the reader forgets it: a recorded failure or drop is cleared, a pull still queued or in flight for it is dropped from the catch-up batch, and the gate re-opens at once (since 26.09.2, #759). That second path is the remedy for an entry no peer can serve, for example a file that was lost on every node, which no later pull will ever fix: the reconciliation sweep on the origin writer (reconciliation.enabled, or POST /api/v1/reconciliation/trigger?dry_run=false&act=true) removes such entries. A key no backend can address has no automated remover yet (retention cannot read it and the sweep only reports it); that class waits on the operator delete endpoint tracked in #794. The puller tracks affected paths in dedicated sets so it can attribute either event back to the original failure or drop. Pulls abandoned because their entry left the manifest are counted in skipped_gone, in the 503 body and in replication_catchup_status on /api/v1/cluster.

Both catchup_failed and catchup_dropped are surfaced in the 503 body so operators see exactly what happened. The /api/v1/cluster endpoint also exposes the cumulative failed / dropped / pulled / skipped_dup keys with their original whole-puller-lifetime semantics (preserved for dashboards landed before #392), alongside the new catchup_* keys for gate-relevant numbers. So dashboards can distinguish "the catch-up batch had a hiccup" (catchup_failed > 0) from "the puller has been having steady-state problems for hours" (failed >> catchup_failed).

Combining with `replication_catchup_enabled=false`

If you set cluster.replication_catchup_enabled=false (the emergency off-switch for pathologically large manifests), the catch-up walker never runs and the gate would never clear. Arc detects this combination at startup, logs a WARN, and auto-disables the gate so the node isn't permanently 503'd. Operators see a clear log line and can fix the configuration at their leisure. Don't enable the gate if you've also disabled the walker.

The same switch turns off the own-file re-pull and the reconciler's manifest-sweep hold: a node restored with an empty data disk does not pull back the files it originated, and its reconciliation runs are not held while it is missing them. Keep reconciliation in dry run after such a restore, or re-enable the walker first.

Endpoints affected

When the gate is enabled and the node is still catching up, these endpoints return 503:

  • POST /api/v1/query
  • POST /api/v1/query/arrow
  • POST /api/v1/query/estimate
  • GET /api/v1/query/:measurement
  • GET /api/v1/measurements

Internal endpoints (cache invalidation, cluster status, replication-control APIs) are deliberately not gated — peer nodes need them to fire during catch-up.

503 response shape

{
  "success": false,
  "error": "replication_catch_up_in_progress",
  "message": "Reader is still catching up on replicated files. Retry shortly or check /api/v1/cluster for catch-up progress.",
  "catchup_status": {
    "started_at": 1714912800,
    "completed_at": 0,
    "entries_walked": 1287,
    "enqueued": 1287,
    "catchup_inflight": 2,
    "catchup_failed": 0,
    "catchup_dropped": 0,
    "queue_depth": 7,
    "inflight_count": 2,
    "pulled": 1278,
    "tier_registered": 1278
  }
}

A Retry-After: 5 header is also set so HTTP-aware load balancers and clients can back off automatically.

completed_at = 0 means the catch-up walker is still enumerating; once it flips non-zero, watch queue_depth + inflight_count go to zero. Non-zero catchup_failed or catchup_dropped means the gate stays closed until a later pull of that path succeeds or the entry is removed from the cluster manifest; the 503 body and the reader's log name the paths, and the reconciliation sweep (POST /api/v1/reconciliation/trigger) is the supported way to remove entries whose file no node holds.

Observability

  • Cumulative gate fires: QueryHandler.QueryGate503Total() is exposed for Prometheus / metrics scrapes. Alert on a non-zero rate to detect that the gate is firing without inferring from generic HTTP error logs.

  • Sampled log line: while the gate is active, Arc emits at most one WARN log per second with the gate counter and request path. Avoids flooding under sustained catch-up while still surfacing the degraded state.

  • Live status: the /api/v1/cluster endpoint exposes replication_catchup_status with the same fields shown in the 503 body, so dashboards can show catch-up progress without waiting for a query to fail.

  • Tier metadata: tier_registered counts pulled files handed to this node's tiered storage metadata (since 26.09.3). With tiering on it tracks pulled; without tiering it stays at zero. pulled climbing while tier_registered stays flat on a tiering node means replicated files are not reaching tier metadata, which costs that node partition pruning until its next tier scan.

  • Same-size rewrites (since 27.01.1, #798): a file whose manifest content changes at the same size, or that is deleted and registered again at the same path, is refreshed rather than accepted by its size. Three counters in replication_catchup_status (and in the 503 body) track this across every forced refresh:

    • forced_refresh: forced refreshes that downloaded the file.
    • refresh_verified: refreshes that found the local copy already was the listed version, by hashing it, and downloaded nothing.
    • refresh_pending: forced refreshes that failed or were dropped and wait for their path's next arrival. It is kept in memory only; since 27.01.1 a restart no longer loses the retry, because the next manifest walk finds the copy stale by what the node recorded it holds (#1182, below). A value that stays above zero means some local copies may still be the previous version.
    • listed_unvouched: manifest-walk entries whose local copy the node could not vouch for and refreshed (hashed first when its content was unknown).
    • verify_unreadable: hashes of such copies that could not be read (a hash stalled for 30 s); the next walk retries them.
    • no_peer_exhausted: pulls that found no candidate peer in any attempt. Since 27.01.1 they count as failed, so the startup gate no longer goes green over a file the node never fetched.

    A reader that keeps a local copy because its path was registered again logs Local delete skipped: the manifest lists the path again; refreshing the local copy unless it already is the listed version. The same line at WARN means the refresh could not be queued: the pull queue was full, in which case it is retried on the path's next arrival or the periodic reconciliation, or the node was shutting down.

Syncing with the leader before the walk

Before the startup walk, the node waits until its manifest reflects everything the leader had committed at that moment, so a restarted reader does not walk a half-replayed manifest. The leader uses Raft's own barrier; a follower forwards a barrier entry through the leader and waits for it to be applied locally (since 26.09.2, #799). The gate stays closed while it waits. cluster.replication_catchup_barrier_timeout_ms (default 30000) bounds that wait; it has to outlast the leader's replication backoff toward a node that was down for a while (up to about ten seconds) and, on a node that was far behind, a snapshot install. On timeout the node logs proceeding against possibly-stale manifest with its applied, commit and last log index and walks anyway. In a mixed-version cluster an older leader rejects the barrier and the reader falls back to that same path: upgrade writers before readers.

Known limitation

There is a sub-millisecond window between the Raft FSM committing a RegisterFile entry and the puller's Enqueue callback firing. A query landing in that window can observe ReplicationReady() == true while a manifest entry from the same Raft commit is not yet in the in-flight set. Closing this gap requires a per-query Raft LastApplied() barrier on the query path, which is out of scope for this gate.

The gate's contract is "every file the puller has observed has been pulled," not "every file the manifest currently contains has been pulled." In practice this means the gate may unblock a fraction of a second before the very last files committed before the gate-clear are queryable. This is a tracked follow-up.

Pattern A vs. Pattern B

  • Shared object storage (Pattern A): the puller is disabled (replication_enabled = false), so query_gate_on_catchup is effectively a no-op — readers see the bucket directly and don't need to catch up. Safe to leave the flag at any value.
  • Local storage with peer replication (Pattern B): this is where the gate matters. Enable it on readers whose application cannot tolerate partial results during cold start or after a network partition.

Deployment example

A minimal 3-node cluster with one writer and two readers using Docker Compose:

# docker-compose.yml
version: "3.8"

services:
  # Shared storage (MinIO as S3-compatible backend)
  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin123
    ports:
      - "9001:9001"

  # Writer node (primary)
  arc-writer:
    image: basekick/arc:latest
    environment:
      ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
      ARC_STORAGE_BACKEND: minio
      ARC_STORAGE_S3_BUCKET: arc-data
      ARC_STORAGE_S3_ENDPOINT: minio:9000
      ARC_STORAGE_S3_ACCESS_KEY: minioadmin
      ARC_STORAGE_S3_SECRET_KEY: minioadmin123
      ARC_STORAGE_S3_USE_SSL: "false"
      ARC_STORAGE_S3_PATH_STYLE: "true"
      ARC_CLUSTER_ENABLED: "true"
      ARC_CLUSTER_NODE_ID: writer-01
      ARC_CLUSTER_ROLE: writer
      ARC_CLUSTER_CLUSTER_NAME: production
      ARC_CLUSTER_COORDINATOR_ADDR: ":9000"
      ARC_CLUSTER_REPLICATION_ENABLED: "true"
      ARC_AUTH_ENABLED: "true"
    ports:
      - "8000:8000"

  # Reader node 1
  arc-reader-1:
    image: basekick/arc:latest
    environment:
      ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
      ARC_STORAGE_BACKEND: minio
      ARC_STORAGE_S3_BUCKET: arc-data
      ARC_STORAGE_S3_ENDPOINT: minio:9000
      ARC_STORAGE_S3_ACCESS_KEY: minioadmin
      ARC_STORAGE_S3_SECRET_KEY: minioadmin123
      ARC_STORAGE_S3_USE_SSL: "false"
      ARC_STORAGE_S3_PATH_STYLE: "true"
      ARC_CLUSTER_ENABLED: "true"
      ARC_CLUSTER_NODE_ID: reader-01
      ARC_CLUSTER_ROLE: reader
      ARC_CLUSTER_CLUSTER_NAME: production
      ARC_CLUSTER_SEEDS: arc-writer:9000
      ARC_AUTH_ENABLED: "true"
    ports:
      - "8001:8000"

  # Reader node 2
  arc-reader-2:
    image: basekick/arc:latest
    environment:
      ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
      ARC_STORAGE_BACKEND: minio
      ARC_STORAGE_S3_BUCKET: arc-data
      ARC_STORAGE_S3_ENDPOINT: minio:9000
      ARC_STORAGE_S3_ACCESS_KEY: minioadmin
      ARC_STORAGE_S3_SECRET_KEY: minioadmin123
      ARC_STORAGE_S3_USE_SSL: "false"
      ARC_STORAGE_S3_PATH_STYLE: "true"
      ARC_CLUSTER_ENABLED: "true"
      ARC_CLUSTER_NODE_ID: reader-02
      ARC_CLUSTER_ROLE: reader
      ARC_CLUSTER_CLUSTER_NAME: production
      ARC_CLUSTER_SEEDS: arc-writer:9000
      ARC_AUTH_ENABLED: "true"
    ports:
      - "8002:8000"

High availability

Arc Enterprise's HA model depends on which deployment pattern you chose. Both deliver "writer crash recovered without operator intervention," but the mechanics are very different.

Pattern 2 — shared object storage (multi-writer)

When all nodes share an object-storage backend (S3, Azure Blob, MinIO), Arc Enterprise runs in multi-writer mode: N writer nodes accept writes concurrently behind a load balancer. There is no "primary writer" to fail over to — every writer is a peer, and the load balancer handles writer-crash recovery via its own health-check.

Key characteristics:

  • Recovery time: immediate — the next request lands on a surviving writer via the LB
  • Health-based detection: load balancer polls each writer's /ready endpoint (Traefik, nginx, HAProxy, and cloud ALBs all support this out of the box)
  • No "promotion" step: all writers are equivalent for ingestion; nothing in the cluster has to elect a new "primary"
  • Singleton background tasks (retention, continuous queries, deletes, tiered-storage migration) run on whichever node holds the cluster Raft leadership at the time. Raft re-election on leader death is sub-second and the new leader's next scheduler tick picks up the work.
  • DELETEs flush every node first (since 27.01.1): a row DELETE or a database deletion asks every other node to flush its buffered rows of the scope and waits for those files to register, so rows any writer acknowledged before it are deleted. A node that cannot answer makes it 503 unless allow_unreachable is set, and a crashed node counts until it returns or is removed: set cluster.node_id on every node. See Delete operations.

Enable by setting cluster.shared_storage_mode = true (env: ARC_CLUSTER_SHARED_STORAGE_MODE). The Helm chart sets this automatically when storage.mode=shared. Requires an Enterprise license that includes the shared_storage_multi_writer feature.

Deploy 3 writers for full HA. The load balancer routes around a failed writer, and with three of them the cluster still has a spare afterwards. 2 writers is refused by the Helm chart: the second writer is the only spare, so the first failure consumes it and no writer can be drained for a rolling upgrade. 1 writer is fine for development but has no HA, and this pattern suppresses Raft writer promotion on purpose, so losing it stops ingest until an operator restores it. Arc logs a rate-limited warning while a cluster is below three writer-role nodes.

The load balancer must be L7 (HTTP-aware) — a Kubernetes Service is not enough

A ClusterIP Service balances per TCP connection, not per HTTP request. Well-behaved clients (tsm2arc, Telegraf, most SDKs) reuse keep-alive connections, so a heavy writer — a bulk migration, a fat Telegraf aggregate — rides a handful of long-lived connections that stay pinned to whichever pods they first dialed. The symptom is one writer near its limits while the others sit idle, even though "there are 3 writers behind the Service."

Put an L7 load balancer in front of the writers — an ingress controller (nginx, Traefik), Envoy/HAProxy, or a cloud ALB — and send all write clients through it. Each request is then balanced independently and write traffic spreads evenly across writers. Use the plain Service only as the backend the L7 layer targets, never as the client-facing endpoint of a multi-writer cluster.

When a writer crashes:

  1. The writer's /ready endpoint stops responding (or returns 503).
  2. The load balancer marks the writer unhealthy and stops routing to it within one poll cycle (~5–10 s).
  3. New writes route to the surviving writer(s). No in-cluster failover action required.
  4. In-flight buffer on the crashed writer (records that arrived in memory but had not yet been flushed to S3) is lost. Records that completed the S3 PUT before the crash are durable.
  5. On writer restart, the local WAL replays any un-flushed entries into the new Arrow buffer before /ready flips back to 200 and the load balancer resumes routing.

Orphan reconciliation (v27.01.1+)

The orphan reconciler (reconciliation.enabled, off by default) runs on the active compactor. It deletes objects in the bucket that the cluster manifest does not list and that are older than reconciliation.grace_window_seconds (default 24 hours, minimum 1 hour; a smaller value is raised to 1 hour with a warning).

A writer uploads each flushed file before it registers it in the manifest. A writer that cannot reach the cluster leader keeps its files pending until it can, and the reconciler must not delete them. Each run:

  • Syncs with the leader first. Without a reachable leader, the run stops before deleting anything. The run reports abort_reason manifest_not_synced, or leader_barrier_unsupported when the leader runs an older release.
  • Asks every writer which files it has pending. If a writer does not answer, nothing is deleted from storage in that run: it reports storage_sweep_held and storage_sweep_held_reason, naming the writer, and the gauge arc_reconciliation_storage_sweep_held reads 1. A writer that is down, or one that was taken out of service without being removed from the cluster, holds the storage sweep until it is back or removed.
  • Skips files a writer published as pending. Each writer writes its pending files to _registration_pending/ in the bucket when it stops and while a file stays pending for more than a minute, and deletes the list once everything is registered. A writer that stops with pending files it could not publish does not leave the cluster, so it keeps holding the sweep.
  • Skips compaction outputs that the manifest repair will register.

The run counts these skips in skipped_pending_remote and skipped_adoptable. During a rolling upgrade the storage sweep stays held until every writer runs v27.01.1.

Pattern 1 — local storage with peer replication

When each node has its own local storage, one writer at a time takes ingest, because the data is not shared: a second active writer would hold rows no other node can see. Arc elects that writer and promotes a replacement when it fails.

Deploy three writer-role nodes. The failover pool is made of writer-role nodes, not readers. One is elected primary and takes ingest; the other two pull its files and stand by. Readers serve queries from the files they pull, but they are not promotion candidates, so a deployment with a single writer cannot fail over at all, and two absorbs exactly one failure before it is back to a single writer with nothing left to promote. Arc logs a rate-limited warning while a cluster is below three writer-role nodes.

A local-storage cluster with Raft always elects one primary writer, which runs retention, continuous queries, deletes and tiered-storage migration (arc#872). cluster.failover_enabled (with the writer_failover license feature) decides whether a lost primary is replaced automatically. Without it, nothing promotes a replacement, and Arc warns about that shape specifically. Enable failover before adding writers to a local-storage cluster.

Key characteristics:

  • Recovery time: less than 30 seconds
  • Health-based detection: continuous health monitoring with configurable thresholds
  • Automatic promotion: a standby writer is promoted via Raft consensus (CommandPromoteWriter FSM apply)
  • Cooldown protection: after a completed failover, the next one waits cluster.failover_cooldown (60 s). A failed attempt does not start it: the next health check tries again (v27.01.1+)

Enable by setting cluster.failover_enabled = true (env: ARC_CLUSTER_FAILOVER_ENABLED). Requires the writer_failover license feature. By default, standby writers do not receive the primary's WAL. Rows the primary accepted but had not yet flushed when it failed are on no other node; they come back when the failed node restarts and replays its own WAL. From v27.01.1 the files that replay writes are announced to the cluster like any other (arc#1206). With stream takeover on, the new primary publishes those rows itself, from its copy of the primary's stream (arc#1207).

When the primary writer fails:

  1. Health checks detect the failure. The Raft leader judges the writer the cluster designates as primary, so a leader that joined after the last promotion decides the same way (arc#1267).
  2. The Raft leader selects a healthy standby writer, lowest node id first. A writer the cluster state refuses to promote is skipped for a minute.
  3. The selected node is promoted via CommandPromoteWriter Raft apply.
  4. Write traffic re-routes to the new primary (clients reconnect, or the load balancer picks up its /ready/write 200).

Sending writes to any writer (v27.01.1+). A standby writer that receives a write forwards it to the primary when it can trust its view of the cluster: it is in steady contact with the Raft leader, no stream-takeover state holds it, and the primary is healthy. Otherwise it refuses with 503 as described below (arc#1270). A forwarded write makes one hop at most, so a Service or load balancer that spreads writes over all writers works. Readers and compactors forward writes to the designated primary too.

A forwarded write is sent once, never retried once sent:

  • 503 with Retry-After and X-Arc-Ingest-Fenced: the primary could not be reached, and nothing was written. Retry.
  • 504: the write was sent but no answer came back within cluster.write_forward_timeout_ms (default 60000). It may have been written, so a retry may duplicate rows.

arc_cluster_writes_forwarded_total{result} counts forwards by result: ok, refused (the primary's own 503), error, unavailable, ambiguous.

Imports (/api/v1/import/*, v27.01.1+). A reader, a compactor or a standby writer forwards an import to a writer the same way: once, with the upload as received (arc#1070). Before, a reader or a compactor imported the file into its own buffer. cluster.write_forward_timeout_ms does not apply to an import, so a large one is not cut off while the writer is still importing it. It is bound by cluster.import_forward_timeout_ms (default 3600000, 1 hour; 0 = no bound), a backstop for a writer that never answers: the forward is not cancelled when the client gives up, and until then the forwarding node holds the upload. A 504 on an import means the backstop ran out or the connection to the writer broke after the upload was sent; the import may have finished. An upload over the 500 MB import limit is refused with 413 before it is forwarded. A writer writes up to ingest.import_write_concurrency imports at once, so a forwarded import can wait for a slot; that wait counts against the backstop too, and past ingest.import_queue_timeout_ms (default 30 s) the writer refuses it with 503 and nothing written, which the forwarding node returns as is.

MQTT and continuous queries on a standby still refuse with 503.

Which writers accept writes (v27.01.1+)

From v27.01.1 a Pattern 1 writer refuses writes with 503 and Retry-After: 5 when it may not be the primary (arc#1207):

  • Another writer is the designated primary. This applies to standby writers, and to a former primary after failover or a manual hand-over.
  • It lost contact with the cluster. It is not the Raft leader and has not heard from the leader for longer than half of cluster.health_check_interval × cluster.unhealthy_threshold (7.5 s by default, at least 3 s). This rule needs automatic failover and at least three writers. Failover itself takes about 30 s at defaults to replace a lost primary. Once contact returns, writes resume after it has held for 2 s and the writer has caught up with the cluster log. cluster.ingest_fence_after_ms overrides the delay; it is capped at the product above.

Nothing is written before the refusal, the WAL included, so a retry is safe. /ready/write reports not-ready meanwhile. arc_cluster_ingest_fenced is 1 while a writer has lost contact (a standby refusing writes is normal and does not set it). arc_cluster_ingest_fenced_writes_total counts refusals by reason: contact (lost contact), standby (another writer is the primary), claimed, announcing, not_designated and delete_ack (stream takeover), and other. Sum by reason for the total. A refusal is counted per attempt: an MQTT message retried while refused counts once per retry.

A refused write is answered with 503, Retry-After: 5 and X-Arc-Ingest-Fenced: 1. It is not counted in arc_ingest_errors_total and is not logged per request. Each node logs Writes refused on this node (retryable) at WARN at most once per reason every 30 seconds, with the number of refusals since the previous line (suppressed). A reader or compactor that forwarded the write logs it the same way, with reason forwarded (arc#1268).

MQTT subscriptions on a writer that refuses writes are suspended without unsubscribing. With clean_session=false the broker queues QoS 1 and 2 messages until writes resume. Run MQTT subscriptions on the primary.

A network failure that blocks only the Raft port, and not the cluster heartbeat port, fences the primary while failover still considers it healthy. Writes stop until Raft connectivity returns.

WAL coverage (experimental, v27.01.1+)

cluster.wal_coverage = true (environment ARC_CLUSTER_WAL_COVERAGE) is the first step toward letting a standby writer take over rows that a failed primary acknowledged but had not flushed. It is off by default. It applies to a Pattern 1 writer with the WAL enabled and is ignored elsewhere, with a warning in the log.

With it on:

  • Every ingest batch gets a coverage id: unique per writer process and measurement, assigned in commit order. The id is written into the batch's WAL entry and, with every file the batch is flushed into, into the registration journal.
  • WAL entries stay until their files are registered in the cluster manifest, not just until they are flushed. During a manifest outage (lost Raft quorum, for example) the WAL grows with the unregistered data. The hold survives restarts. WAL recovery keeps a file whose batches still wait to register, and every file after it, until the registrations land. A clean shutdown keeps them instead of purging the WAL. If the cluster does not come up at all, the node keeps them until a later start registers them and logs an Error. Files registered with coverage stay held even after the setting is turned off.
  • A write the buffer refuses (schema churn, shutdown, a cancelled request) leaves nothing in the WAL.
MetricMeaning
arc_wal_registered_floor_lagBatches committed but not yet in a registered file. Returns to 0 once every flushed file is registered; a lost batch stays in it until the process restarts.
arc_wal_coverage_lost_totalBatches whose file settled without being registered (gone or rewritten first), or whose flush failed with no complete WAL copy. Alert on any increase.
arc_wal_held_filesWAL files kept because batches in them wait for their files to register. Later files are kept too, so watch arc_wal_dir_bytes and arc_cluster_file_registration_oldest_pending_seconds while it is above 0. A Warn repeats every 10 minutes while files are held.
Standby writers keep a copy

With cluster.wal_coverage on, the designated primary also streams every committed batch to the standby writers over the cluster port. Each standby keeps the batches on disk under <wal.directory>/received/ and never applies them. A standby discards a batch once two things hold: the batch is in a registered file, and the standby holds every current file of that measurement itself. This copy is what stream takeover publishes a failed primary's acknowledged rows from.

The stream needs cluster.shared_secret. Batches are authenticated but are not encrypted unless cluster.tls_enabled is set. Every writer in the cluster must run v27.01.1 or later before you turn the setting on.

SettingDefaultMeaning
cluster.standby_store_max_bytes0 (a quarter of the WAL volume, at least 1 GiB, at most 32 GiB)Caps a standby's copy; negative means no cap. Past it, batches from a primary process that has since restarted are evicted first. If that is not enough, nothing new is kept, and the missed batches are recorded as gaps until trimming frees room. On a volume that does not enforce its size (k3s local-path, hostPath, Docker named volumes) the volume is the host's disk; set the cap explicitly there.
cluster.origin_stream_queue_bytes64 MiBCaps what the primary holds in memory for one standby that is not keeping up (about 1.8 s of stream at a million rows a second). Past it, that standby misses batches; the other standbys do not. Raising it costs primary memory, once per standby.
MetricWhereMeaning
arc_origin_stream_sessionsprimaryStandby writers being streamed to.
arc_origin_stream_dropped_totalprimaryBatches a slow standby missed. Until their files are registered they are on the primary only: a primary lost with its volume in that window loses those acknowledged rows. Alert on increase(arc_origin_stream_dropped_total[5m]) > 0 (v27.01.1 keeps up at about 1.2M rows/s on a 4-core QA node).
arc_standby_stream_connectedstandby1 while receiving from the primary.
arc_standby_store_bytesstandbySize of the copy.
arc_standby_store_incomplete_keysstandbyMeasurements missing batches above their registered floor.
arc_standby_store_blocked_keysstandbyMeasurements with registered batches this standby may not trim yet: a file holding their rows is not here (a file the puller cannot fetch keeps those batches), this node's manifest is behind the primary's, a claim is open, or the primary is not heard from. Expect it to be nonzero for every measurement of a primary that just failed, until the takeover finishes.
arc_standby_store_lost_recordsstandbyBatches the primary reported lost. The standby keeps them: they may be the only copy.
arc_standby_store_dropped_total, arc_standby_store_evicted_bytes_totalstandbyBatches not kept (at the cap or below the free-space reserve), and bytes evicted to stay under it. Alert on increase(arc_standby_store_dropped_total[5m]) > 0: a takeover cannot publish those batches.
arc_standby_store_capped, arc_standby_store_low_space, arc_standby_store_max_bytesstandby1 while the store refuses batches at its cap; 1 while its volume is below the free-space reserve; the effective cap (v27.01.1+).

The copy is trimmed every 5 seconds, and at once when it passes half its cap (arc#1266). It holds each batch until the batch is in a registered file and that file is on the standby, so size the cap for the ingest rate times the replication lag. On a QA cluster at about 1.2 million rows a second the copy stayed under 900 MB with a 1 GiB cap, including right after a standby restart. A batch goes once the files holding its rows (by their coverage) are on the standby; an unpulled file of other batches, or an unpulled compaction output, does not hold it back. With stream takeover off, nothing claims from the copy, and a batch goes once it is registered. The automatic cap stops at 32 GiB because a full store is read whole at startup; set a larger one explicitly. The store keeps no new batches while its volume has less than 10 % (at least 2 GiB) free, and resumes 1 GiB above that, so the WAL and the Parquet files on the same volume come first. On a volume of 2 GiB or less the store keeps nothing; Arc warns about this at startup.

The received/ directory shares the WAL volume but is not counted in arc_wal_dir_bytes.

Stream takeover (v27.01.1+)

Stream takeover lets the new primary publish the rows a failed primary acknowledged but never registered, exactly once (arc#1207). It is off by default.

Requirements. Set cluster.stream_takeover = true (environment ARC_CLUSTER_STREAM_TAKEOVER) on every node, readers included. Writers also need cluster.wal_coverage = true and an explicit cluster.node_id. Each node advertises support when it joins; the cluster turns takeover on by itself once every node does, and from then on refuses a node that joins without it. The switch is one-way.

What happens when the primary dies.

  1. Failover designates another writer, which stops receiving the old primary's stream.
  2. The new primary waits cluster.stream_takeover_grace (default 60s, at least 10s) from the moment it last heard from the old primary. A primary that restarts with its volumes within the grace, or that left gracefully and comes back, publishes its own rows and nothing is taken over.
  3. It claims, in the cluster manifest, every batch of the old primary it holds that is not in a registered file, replays them to Parquet and registers them. The manifest refuses any row registered twice, so a batch the old primary registered in part is completed, not duplicated.
  4. Before it finishes, the new primary makes sure it holds every file of the old primary whose rows its standby copy holds. A file every node says it does not have (or that no node could answer for during twice the grace, counted for every such file from the start of the claim, v27.01.1+) is removed from the manifest, and its rows are published from the standby copy too: without this, rows the old primary registered but nobody pulled before its volume was lost would be gone. A node that later comes back with a copy of such a file removes that copy. A file the standby copy could not publish again is never removed.
  5. Once every claimed batch is in a registered file, the claim finishes. Any other writer that holds batches of that process the new primary lacked then claims those too (each writer once), as long as the old primary has not come back with a new process and is not the designated primary.

When the old primary comes back with its volume. Rows it acknowledged that no standby held (the replication lag), and rows the new primary could not publish, are still in its WAL. It reads them back and publishes them, once; its copies of rows the takeover already published are refused and removed. This holds even when the cluster had given that writer up (it was removed, or its departure expired): such processes are kept as retired, with what was registered of them (the last 256), so that a writer coming back late still publishes exactly the rows nobody registered.

A claimed writer exits with code 3. If the old primary was only cut off, it refuses writes the moment it sees the claim, flushes, and exits with status 3 so the restart policy starts a new process. Its own copies of the claimed batches are parked under <storage>/_refused/, not registered. Run writers under a restart policy that restarts on a non-zero exit (systemd Restart=on-failure, Kubernetes, Docker restart: on-failure or unless-stopped). A writer that is also the Raft leader steps down as it exits.

Writes go to the primary only. With takeover on, a standby writer refuses writes with a retryable 503, checked on every write.

A DELETE stays deleted through a takeover. With takeover on, a row DELETE (POST /api/v1/delete) or a database deletion (DELETE /api/v1/databases/{name}) is recorded in the cluster before it runs. Rows acknowledged before it that are not in a registered file yet (a standby's copy a takeover replays, an old primary's WAL, a writer's pending files) are rewritten without the rows it removes before they register, with the DELETE's own SQL; the cluster manifest refuses any such file that was not. Rows acknowledged after the DELETE's own flush are outside the record (a write concurrent with the DELETE may or may not be deleted).

  • Per-statement time functions in the WHERE clause (now(), current_timestamp, current_date, current_time, localtimestamp, today()) are fixed to the time the DELETE ran, in UTC, so a later replay matches the same rows. Functions no replay can reproduce (random(), uuid(), gen_random_uuid(), nextval(), setseed()) are refused with 400.
  • The DELETE rewrites the files the primary has, so it first waits, up to 30 seconds, for every file the manifest lists of the measurement to reach the primary. A file that does not arrive answers 503 with unreachable_files. Send "allow_unreachable": true to run it anyway: the response lists the files it skipped, and their rows are not deleted.
  • Each writer acknowledges every DELETE with the batches it had accepted, and refuses writes (retryable 503) for the moment that takes (about 100 ms).
  • For a few seconds, a node other than the primary can still show deleted rows in a file it has just written and not registered yet. The primary never does once the DELETE has answered.
  • The cluster keeps up to 256 DELETEs until every writer process they name has settled the batches before them. Past that, the oldest one that only retired writer processes still hold is dropped, and those processes are removed (their copies are parked if they come back); otherwise a DELETE answers 503 until older ones settle.
SettingDefaultMeaning
cluster.stream_takeoverfalseOpt this node in. Needed on every node.
cluster.stream_takeover_grace60sHow long a primary that stopped answering may take to come back before its batches are taken over. Also how long a claimant may be gone before the leader releases its unfinished claim.
cluster.stream_takeover_departed_ttl24hHow long a writer that left gracefully keeps its takeover state. After it, its processes may be taken over by any writer holding them.
MetricMeaning
arc_cluster_stream_claims_enabled, arc_cluster_stream_claimsWhether takeover is on, and the claims the manifest holds.
arc_stream_takeover_oldest_claim_age_secondsHow long the oldest unfinished claim has been open. A claim whose replay keeps failing (a full disk) is never given up on by itself: alert on it.
arc_stream_takeover_departed_writersWriters that left gracefully and have not re-joined.
arc_stream_takeover_unpulled_filesFiles of the old primary the new primary does not hold yet; its claim finishes only once it holds them, or they are found on no node.
arc_wal_parked_files, arc_wal_parked_bytesFiles a writer could not register because their batches were claimed or already registered.

Operator API (admin):

  • GET /api/v1/cluster/stream-claims lists writer processes, claims, departed writers, and the DELETEs the cluster keeps (delete_records, with the writer processes that have not reported their batches yet as unacknowledged).
  • POST /api/v1/cluster/stream-claims/{instance}/release with {"epoch": n} releases an unfinished claim: any writer holding the batches may claim them again, its claimant included. The leader does this by itself when a claimant has been gone for the grace.
  • The same call with {"epoch": n, "abandon": true} finishes the claim instead. It cannot be undone. The batches it has not published are left to their origin, which publishes them if it comes back with its volume; they are lost when the origin is gone for good. The claimant drops its copy and does not claim the process again (another writer still may, from its own copy). The claim then reports every batch it held as unreplayable, published or not. Use it only for a claim that cannot finish.

Both POST forms run on the Raft leader (409 elsewhere) and answer 404 for no such unfinished claim. instance is the 16-digit hex id the GET lists.

What takeover does not cover.

  • Rows the primary acknowledged but had not streamed to any standby yet when its volume was lost for good: the replication lag, usually milliseconds.
  • A DELETE run with allow_unreachable leaves the rows of the files it skipped.

Rolling back with WAL coverage on

WAL entries written with cluster.wal_coverage on use a format that releases before v27.01.1 cannot read. On a rollback, an older release quarantines them and loses any rows not yet flushed. Before rolling a node back, either turn the setting off and restart once, or stop the node cleanly so it flushes and purges its WAL. The registration journal stays readable by older releases.

Choosing between the patterns

  • Cloud-native deployments (EKS/GKE/AKS, anywhere managed S3 is available) → Pattern 2 multi-writer. Simpler operationally, scales writes horizontally, the LB does failover.
  • Bare metal, on-prem, edge without easy access to S3-compatible storage → Pattern 1 with writer failover. One writer takes ingest at a time, so throughput has a single-node ceiling; HA comes from promoting a standby writer.

Both patterns want three writer-role nodes. In shared storage all three take traffic; in local storage two of them stand by. Either way, three is what lets the cluster lose one node and still elect a leader.

See Deployment Patterns for the full trade-off comparison.

Compaction commit (v27.01.1+)

Compaction commit makes a row DELETE and a compaction of the same files safe to run together, and on local storage keeps a compaction's rows on at least two nodes at every step. A DELETE's manifest update applies only while the file is still registered with the content the DELETE read. If a compaction replaced it, the DELETE scans again and deletes from the compaction's output (see Delete operations).

On local storage, a compaction is staged before it commits:

  1. The compactor writes its output under _compaction_staged/ in its storage, where no query reads it, and leaves its inputs alone.
  2. Every other node that keeps a local copy of the data pulls the staged output and reports holding it.
  3. Once another node holds it, the cluster registers the output and deletes the inputs in one step, only if every input is still registered exactly as the compactor read it. Otherwise the job is dropped (its inputs are untouched) and compacted again later.

While the compactor is the only node keeping a local copy of the data up, a staged job waits and is dropped after 10 minutes (longer for a large output). A staged output takes space on each node that holds it until the commit.

On object storage (S3, Azure), a compaction is staged too:

  1. The compactor uploads its output under _compaction_staged/ in the bucket, recording its checksum on the object, and leaves its inputs alone.
  2. It commits at once (every node reads the same bucket): the cluster registers the output and unregisters the inputs, only if every input is still registered exactly as the compactor read it.
  3. A finisher copies the staged object to the output's path inside the store (S3 CopyObject or a multipart copy above 5 GiB; Azure Put Blob From URL or blocks from URL above 5000 MiB), deletes the input objects and ends the job. If Azure refuses the server-side copy (some Entra or managed-identity setups), the finisher copies the bytes through the node.

The compactor runs the finisher at once; if it is gone, the leader takes the job over after 5 minutes. Until the copy, reads see the inputs' rows; a DELETE on that measurement waits for the output (up to 30 seconds per scan) and otherwise answers 503. If the output leaves the manifest before the finisher runs, a DELETE, retention, a database deletion, tiering or a restore removed its rows with it, and the inputs are deleted too; any other removal registers the inputs again. If that is not possible either (an input object is gone), the job is listed in compaction_commit.wedged_jobs and waits for an operator: check the data, then end it with POST /api/v1/cluster/compaction-jobs/{job}/abort (admin). A restore's compaction pause waits for committed jobs to finish first. The leader removes staged objects no job owns after 6 hours; on S3, keep a lifecycle rule that aborts incomplete multipart uploads.

There is no setting. The leader turns it on once every node in the cluster runs a release that supports it. GET /api/v1/cluster reports it:

"compaction_commit": {"enabled": true, "staged": 0, "finalizing": 0}

finalizing counts committed object-store jobs not yet finished, and wedged_jobs lists those waiting for an operator. The metrics arc_compaction_finished_total, arc_compaction_wedged_total and arc_compaction_copy_fallback_total count them over time. While it is off, the leader also lists the nodes keeping it off in blocking_nodes. Once it is on, it stays on, and a node running an earlier release cannot join.

On a shared bucket, releases before 2027.01 did not record compaction in the cluster manifest. Compaction outputs stayed unregistered, and the inputs they replaced stayed listed. Queries were not affected, because they read the bucket. After the upgrade, the leader repairs the manifest 2 minutes after it becomes leader, then every hour. A repair waits while a compaction pause is in force. On any cluster it moves entries that older releases registered with a compaction tier (hourly, daily) to hot. On a shared bucket with stream claims off, it also:

  • drops entries whose object is gone;
  • registers compaction outputs (*_compacted.parquet in an hour directory, *_daily.parquet in a day directory) the manifest does not list, once they are an hour old and unless tiering records them as cold. Each output is read for its checksum, at most 1000 objects or 10 GiB per run, so on a large bucket the first repairs take several runs.

Other objects the manifest does not list are left to the reconciler. Until the repair has run, a DELETE still removes every matching row a query reads: it skips a listed file whose object is gone, and it rewrites an object the manifest does not list, registering the rewrite. On the leader, compaction_commit.manifest_repair in GET /api/v1/cluster shows the last run. The metrics arc_manifest_repair_dropped_total and arc_manifest_repair_adopted_total count the entries dropped and the outputs registered; the gauge arc_manifest_repair_pending counts outputs left for a later run.

Rolling a node back

Once compaction commit is on, do not restart a node on an earlier release with its existing data. The earlier release skips the new cluster log commands, so its view of the cluster manifest silently diverges from the other nodes. To run an earlier release on a node, remove the node from the cluster, wipe its data directory, and join it fresh. To roll back the whole cluster, restore a backup taken before the upgrade.

Copies a node missed deleting (v27.01.1+)

On local storage, each node removes its own copy of a file when the cluster manifest drops it (a DELETE, retention, compaction, tiering). A node misses that removal when it is down while the file is deleted and rejoins from the leader's snapshot, or when it crashes with the removal still queued. Before v27.01.1 such a copy stayed on that node's disk, and its queries kept returning the deleted rows (or duplicates of compacted ones) until the opt-in orphan sweep removed it (arc#1071).

Each node now records the local files it has seen registered, in seen-local.log in cluster.raft_data_dir. Every 5 minutes it syncs with the leader and removes a recorded file the manifest no longer lists, when all of these hold:

  • the node's manifest has applied past the file's registration;
  • the file has been unlisted for at least 10 minutes, over two passes that ran (a pass skipped for a compaction pause or a failed sync starts the wait again);
  • it is not waiting to register on this node, not held by an edge-sync delivery, not part of a compaction in progress, and not in the cold tier on this node (a tier record that cannot be read counts as cold);
  • it is still the same file the node recorded (size, modification time and inode). The copy is moved under _stale_local/ in the storage directory and checked again there before it is deleted, so a file written to the path meanwhile is put back.

A file the node never saw registered is never removed by this sweep: files a writer has flushed but not registered yet, a restore in progress, and files left over from before the upgrade (those remain for the reconciler's orphan sweep, reconciliation.enabled).

A pass that would remove more than cluster.stale_local_sweep_max_fraction of the node's tracked files (default 0.5), and at least 1000, removes nothing: a manifest rolled back by an unsafe Raft recovery looks like that, and the copies are then the recovery material. The node logs an error and arc_stale_local_blocked reads 1. If the deletes are real (a long outage with heavy retention), raise the setting and restart the node. The setting must be a number above 0; above 1 the sweep never refuses a pass.

The sweep runs on local storage with cluster.replication_enabled and cluster.replication_catchup_enabled; cluster.stale_local_sweep = false turns the removals off, and the status then reads removals_off: true. The tracking keeps running, because the manifest walks judge local copies by it (below). It costs about 150–200 bytes of memory per local file, and a journal of up to 1.5 times that on the Raft volume. If the journal cannot be opened or written (a full Raft volume), the node stops tracking and removing until it restarts. GET /api/v1/cluster reports the last pass under stale_local (last_run, tracked, unlinked, waiting, blocked, removals_off, error). Metrics: arc_stale_local_unlinked_total, arc_stale_local_tracked, arc_stale_local_blocked, arc_stale_local_tracking_failed.

What a local copy holds

Since 27.01.1 (#1182) the same record also says what each local copy holds: the checksum the node pulled, hashed, or wrote itself (a flush, a compaction output). It stays while the file is the same one, so a registration of new content at the path does not change it. The startup catch-up and every periodic replication walk compare it with the manifest:

  • a copy that holds the listed checksum is trusted, as before;
  • a copy known to hold other bytes is pulled again;
  • a copy whose content the node does not know is hashed first, and pulled only if it differs.

This covers a content change whose refresh failed and was then lost to a restart, one the node learned of from a Raft snapshot (which fires no callback), and a file replaced on disk after the record was taken. These refreshes count in the startup batch, so with cluster.query_gate_on_catchup the gate stays red until each is settled, and a node that cannot refresh one reports it in catchup_failed. A copy is never removed by this: when no peer serves the listed bytes, the node keeps its copy and the next walk retries.

A node upgraded from an earlier release has no record of its copies, so its first start hashes each local file once, on the pull workers, in the background; pulls of new files queue behind it. GET /api/v1/cluster reports local_copy_trust: recorded, or size_only when the record cannot be kept (the journal failed or is not open yet), in which case copies are judged by their size alone.

Membership after a restart

Every node keeps two views of the cluster: the Raft-committed node table, written by the authenticated join and restored with each Raft snapshot, and an in-memory registry that the health checker, the heartbeat fan-out, the node listings and the file puller read. On a restart the node table comes back with the snapshot; the registry is rebuilt from it as the snapshot is restored and from any node-added entries replayed after it, so a restarted leader authorises forwarded writes from its followers, heartbeats them and lists them without any of them re-joining (since 26.09.2, #807). A follower whose cluster.seeds is empty, or whose discovery ran after Raft already knew the leader, therefore no longer needs a re-join either: the leader answers its forwarded writes from the node table, and the follower resolves the leader's address from its own copy of it.

Before 26.09.2 the leader consulted only its registry, which a snapshot restore did not refill, so after a whole-cluster restart followers that had not re-joined got unknown node on every forwarded write. A write on such a follower still returned success, because ingestion is local, but the file never reached the manifest. If a cluster on an older release shows ForwardApply rejected: node not found in registry in the leader's log after a restart, restarting the affected follower with cluster.seeds set makes it re-join.

Cluster-wide policies

Since Arc 27.01.1, these policies live in the cluster state instead of each node's own database: a policy written on any node applies on every node and survives a failover. Once every node runs 27.01.1, each kind turns on by itself, and the cluster adopts the policies the nodes already hold.

KindPoliciesFlag in cluster_policiesAdopt API kind
tieringTiering policiesenabled(merged by itself)
retentionRetention policiesretention_enabledretention
cqContinuous queriescq_enabledcq
governanceQuery governance policiesgovernance_enabledgovernance

GET /api/v1/cluster reports them under cluster_policies:

  • the flags above. Until a flag is on, the Raft leader lists the nodes still on an earlier release in blocking_nodes, retention_blocking_nodes, cq_blocking_nodes or governance_blocking_nodes;
  • kinds.<kind>: seeded once the cluster holds that kind's policies; otherwise missing_nodes, last_error, failed_attempts, and an admin's pending chosen_node.

When the nodes' own policies cannot be adopted by themselves (see each kind's page), an admin chooses one node's policies on the Raft leader:

# Each kind's state and, until the cluster holds it, every node's own policies
# with a hash of each node's set. ?kind=retention, cq or governance narrows it to one kind.
curl -H "Authorization: Bearer $ARC_ADMIN_TOKEN" \
  "http://leader:8000/api/v1/cluster/policies?kind=governance"

# Adopt one node's policies of that kind, naming the hash GET returned for it
curl -X POST -H "Authorization: Bearer $ARC_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"kind": "governance", "node": "arc-reader-0", "hash": "<hash>"}' \
  http://leader:8000/api/v1/cluster/policies/adopt
  • Both endpoints are admin-only and run on the Raft leader; any other node answers 409 with the leader's ID in leader.
  • kind is retention, cq or governance; anything else answers 400. node and hash are required.
  • POST answers 202. cluster_policies.kinds.<kind> in GET /api/v1/cluster shows the outcome. It answers 409 when that kind is not on yet, or the cluster already holds its policies.
  • The node's policies are adopted only if it still holds exactly the ones the hash names; every other node's own policies of that kind stop applying and are kept as dormant rows in that node's database (for governance, governance_policies_dormant). Take a backup first.

No rolling back a single node

Once a cluster policy flag is on, a node running an earlier release can no longer join. Roll back the whole cluster from a backup, never one node in place.

API reference

All cluster endpoints require admin authentication.

Get cluster status

curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/cluster

Response (abridged; from a Pattern 1 cluster on v27.01.1):

{
  "cluster_name": "production",
  "enabled": true,
  "mode": "cluster",
  "running": true,
  "local_node_id": "writer-02",
  "local_role": "writer",
  "node_count": 5,
  "healthy_count": 5,
  "writers": 3,
  "readers": 1,
  "compactors": 1,
  "primary_writer_id": "writer-01",
  "writer_failover": {
    "consecutive_fails": 0,
    "cooldown_period": "1m0s",
    "failover_in_progress": false,
    "last_failover_at": "2026-10-14T17:49:58Z",
    "primary_id": "writer-01",
    "running": true
  },
  "nodes": [
    {
      "id": "writer-01",
      "role": "writer",
      "state": "healthy",
      "writer_state": "primary",
      "address": "10.0.1.10:9100",
      "api_address": ":8000",
      "version": "27.01.1",
      "is_active_compactor": false
    }
  ]
}
  • primary_writer_id is the writer the cluster designates as primary (v27.01.1+).
  • writer_failover is this node's failover manager. Only the Raft leader acts on failures, so read it there. After a failed attempt it also reports failed_attempts, last_failure_reason and last_failure_at.
  • nodes[].writer_state is this node's view: primary for the designated writer, empty for the others.

List cluster nodes

# All nodes
curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/cluster/nodes

# Filter by role
curl -H "Authorization: Bearer $ARC_TOKEN" \
  "http://localhost:8000/api/v1/cluster/nodes?role=reader"

# Filter by state
curl -H "Authorization: Bearer $ARC_TOKEN" \
  "http://localhost:8000/api/v1/cluster/nodes?state=healthy"

Response:

{
  "success": true,
  "data": [
    {
      "id": "writer-01",
      "role": "writer",
      "state": "healthy",
      "address": "10.0.1.10:9000",
      "last_heartbeat": "2026-02-13T10:30:00Z"
    },
    {
      "id": "reader-01",
      "role": "reader",
      "state": "healthy",
      "address": "10.0.1.11:9000",
      "last_heartbeat": "2026-02-13T10:30:01Z"
    }
  ]
}

Get specific node

curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/cluster/nodes/writer-01

Get local node info

curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/cluster/local

Cluster health check

curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/cluster/health

Response:

{
  "success": true,
  "data": {
    "status": "healthy",
    "node_id": "writer-01",
    "role": "writer",
    "cluster_name": "production"
  }
}

Best practices

  1. Pick a deployment pattern — Use shared object storage (S3, MinIO, Azure) for cloud-native deployments, or local storage with peer replication for bare metal, VMs, and edge. Don't mix the two in the same cluster.

  2. Size for HA based on pattern:

    • Pattern 2 (shared storage): 3 writers behind a load balancer. The LB routes around a failed writer and a spare remains. Single writer is fine for dev; 2 is refused by the chart, because the first failure consumes the only spare.
    • Pattern 1 (local storage): 3 writer-role nodes with cluster.replication_enabled=true, one elected primary and two standing by, plus as many readers as your query load needs. Readers are never promoted, so they do not count toward writer redundancy.
  3. Scale readers independently — Add reader nodes to handle increased query load without affecting write performance.

  4. Use one dedicated compactor — Run compaction on a single dedicated node to avoid duplicate outputs. Enable ARC_CLUSTER_FAILOVER_ENABLED=true for automatic compactor failover.

  5. Configure seed nodes — Reader and compactor nodes should list writer nodes as seeds for cluster discovery.

  6. Always set a shared secret — ARC_CLUSTER_SHARED_SECRET is required for peer authentication. Arc refuses to start replication without it.

  7. Monitor cluster health — Use the /api/v1/cluster/health endpoint with your monitoring system (Prometheus, Grafana) to detect issues early.

Next steps

  • RBAC — Secure your cluster with role-based access control
  • Tiered Storage — Optimize storage costs with hot/cold tiering
  • Audit Logging — Track all operations for compliance

On this page