Write-Ahead Log (WAL)
How the Arc Enterprise WAL bounds data loss to the sync interval on a writer node, the durability and throughput tradeoff per sync mode, and recovery behaviour after failover.
Arc's Write-Ahead Log (WAL) provides zero data loss guarantees on system crashes.
Important Clarification
There are two different WAL features in Arc:
-
SQLite WAL mode (always enabled) - Internal mode for Arc's metadata database (
arc.db). This enables concurrent access to connection settings, export jobs, and compaction locks. You'll see the log message"SQLite WAL mode enabled for concurrent access"on startup - this is expected and not related to data ingestion. -
Arc's WAL feature (disabled by default) - Optional durability feature for data ingestion that provides zero data loss guarantees. This page documents the Arc WAL feature, controlled by the
WAL_ENABLEDenvironment variable.
TL;DR: The startup log "SQLite WAL mode enabled" is normal and does NOT mean Arc's data ingestion WAL is enabled.
Overview
WAL is an optional durability feature that persists all incoming data to disk before acknowledging writes. When enabled, Arc guarantees that data can be recovered even if the instance crashes.
WAL is disabled by default to maximize ingest throughput. Enable it when zero data loss is required.
When to enable WAL
Enable WAL if you need:
- Zero data loss on system crashes
- Guaranteed durability for regulatory compliance (finance, healthcare)
- Recovery from unexpected failures (power loss, OOM kills)
Keep WAL disabled if you:
- Prioritize maximum ingest throughput
- Can tolerate 0-5 seconds data loss on rare crashes
- Have client-side retry logic or message queue upstream
Performance vs durability tradeoff
| Configuration | Throughput | Data Loss Risk |
|---|---|---|
| No WAL (default) | Highest | 0-5 seconds |
WAL enabled (default ack_mode, durable) | Lower; each write waits for a sync | None of the acknowledged writes |
WAL + ack_mode = "queued", async | Slightly reduced | <1 second of synced data, plus entries still queued |
WAL + ack_mode = "queued", fdatasync or fsync | Slightly reduced | Entries still queued or not yet synced at the crash |
The loss figures for queued follow from acknowledging a write before it is
synced; see Acknowledgement mode.
With cluster.wal_coverage = true the default resolves to queued.
Tradeoff: throughput for no loss of acknowledged writes. The cost is in the acknowledgement mode (each write waits for a sync), not in the sync mode.
How syncs work
Arc does not sync on every write. A background writer appends entries to the
WAL file, and a separate sync runs beside it: one is requested once
sync_bytes of written data is unsynced (default 1 MiB), and every 100 ms
while anything is unsynced. Each sync covers everything written before it
started, and the writer keeps appending while it runs, so WAL throughput
follows the disk's bandwidth rather than its round-trip latency (since
27.01.1, #1246).
Before 27.01.1 the writer waited for each sync itself, which capped WAL-on
ingest at roughly 170 MB/s of WAL on AWS gp3 storage (about 3.5M MessagePack
points per second), whatever the CPU count. If the WAL cannot keep up, its
queue fills and entries are dropped while the write is still acknowledged:
watch arc_wal_dropped_entries_total.
A buffer flush writes a checkpoint to the WAL, and that checkpoint is always
synced before the flush completes, in every sync mode, async included. A
sync that fails is counted in arc_wal_failed_syncs_total; the checkpoints in
that WAL file then fail too, and Arc moves to a new WAL file, because a later
sync of the same file could succeed over data the disk already reported lost.
Architecture
Data flow with WAL
┌──────────────────────────────────────────────────────────┐
│ HTTP Request (MessagePack or Line Protocol) │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 1. WAL.append(records) │
│ - Serialize to MessagePack binary │
│ - Calculate CRC32 checksum │
│ - Write to disk │
│ - fdatasync() ← Force physical disk sync │
└──────────────────┬───────────────────────────────────────┘
│
▼ Data is DURABLE (on disk)
┌──────────────────────────────────────────────────────────┐
│ 2. HTTP 202 Accepted ← Response to client │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 3. Buffer.write(records) │
│ - Add to in-memory buffer │
│ - Flush when 50K records or 5 seconds │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 4. Parquet Writer │
│ - Convert to Arrow columnar format │
│ - Write Parquet file │
│ - Upload to S3/MinIO │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 5. WAL.mark_completed() ← Can now delete WAL entry │
└──────────────────────────────────────────────────────────┘Key Insight
Once WAL confirms the write (step 1), the data is guaranteed durable even if Arc crashes before step 4 completes.
WAL files
Arc uses a single WAL writer with goroutines for concurrent access:
./data/wal/
├── arc-20251008_140530.wal
└── arc-20251008_150530.walBenefits:
- Simple implementation
- Automatic rotation
- Parallel recovery on startup
Configuration
Enable WAL
Edit arc.toml:
[wal]
enabled = true
sync_mode = "fdatasync" # Recommended for production
directory = "./data/wal"
max_size_mb = 500 # Rotate at 500MB
max_age_seconds = 3600 # Rotate after 1 hour
sync_bytes = 1048576 # Request a sync once this many bytes are unsynced (default 1 MiB)
ack_mode = "auto" # durable with the WAL on (default, v27.01.1+); or queued, durable
ack_timeout_seconds = 30 # a durable write waits at most this long for its sync (v27.01.1+); 0 = no boundOr via environment variables:
ARC_WAL_ENABLED=true
ARC_WAL_DIRECTORY=./data/wal
ARC_WAL_SYNC_MODE=fdatasync
ARC_WAL_MAX_SIZE_MB=500
ARC_WAL_MAX_AGE_SECONDS=3600
ARC_WAL_SYNC_BYTES=1048576
ARC_WAL_ACK_MODE=auto
ARC_WAL_ACK_TIMEOUT_SECONDS=30Sync modes
Arc supports three sync modes with different durability/performance tradeoffs:
fdatasync (recommended)
[wal]
sync_mode = "fdatasync"How it works:
- Syncs data to disk (file contents)
- Skips metadata sync (file size, modified time)
- A little less work than
fsync(no timestamp metadata), same data durability
Guarantees:
- Synced data is on physical disk and survives a crash
- With the default
ack_mode = "auto"(durable with the WAL on), a write is acknowledged only once a sync covering its entry has completed, so a crash loses no acknowledged write. Withack_mode = "queued"a write is acknowledged before its entry is synced, and entries still queued or written but unsynced at the crash are lost; see Acknowledgement mode - File metadata may be stale (not critical)
Use case: Production deployments (recommended)
fsync (maximum safety)
[wal]
sync_mode = "fsync"How it works:
- Syncs both data AND metadata to disk
- Slowest; the strongest guarantee for synced data. Acknowledged writes are
covered too with the default
ack_mode(durable)
Use when:
- Regulatory compliance requires it
- Zero tolerance for any data loss (with the default
ack_mode, durable) - Performance is secondary
async (performance-first)
[wal]
sync_mode = "async"How it works:
- Writes to OS buffer cache
- No explicit sync of data (OS flushes periodically); flush checkpoints are still synced
- Very fast, but small risk window
Use when:
- Need 90% of original throughput
- Can tolerate ~1 second data loss
- Have upstream retry mechanisms
Acknowledgement mode (planned for v27.01.1)
Planned for v27.01.1
wal.ack_mode is merged for the planned v27.01.1 release, with auto
(durable with the WAL on) as its default. Earlier versions always behave as
queued.
ack_mode decides when Arc answers a write with the WAL enabled.
ack_mode | A 2xx means | Lost on a crash | Full WAL queue |
|---|---|---|---|
durable (what auto, the default, is with the WAL on) | A sync covering the entry has completed | None of the acknowledged writes | The request waits (backpressure) instead of dropping |
queued | The entry is queued for the WAL writer | Entries still queued, or written but not yet synced | Entries are dropped while the write is still acknowledged (arc_wal_dropped_entries_total) |
auto (default) | durable with the WAL on; queued with the WAL off or with cluster.wal_coverage = true, which durable does not support (logged as a config warning and in the WAL enabled startup line) |
A write Arc refuses (schema churn on its measurement, a request its client
cancelled, a node shutting down) leaves nothing to replay from the WAL
(v27.01.1+). With queued, its entry is only queued once nothing can refuse
the write any more; with durable, the entry is written first and a refusal
retracts it with a checkpoint. Before v27.01.1, without WAL coverage, such an
entry could be replayed after a crash or restart, storing a write the client
was told failed.
[wal]
enabled = true
sync_mode = "fdatasync"
ack_mode = "queued" # opt out of durable acknowledgementHow durable works: each request waits for the sync that covers its
entry. Syncs are shared: every request whose entry was written before a sync
started is answered by that one sync, so concurrent writes do not each pay a
sync. A request that spans several measurements waits once per measurement.
With sync_mode = "async", durable still syncs: a waiting write forces it.
Failures: if an entry cannot be made durable (a write or sync failure, a
payload the WAL cannot hold, or a shutdown), the write is refused with 503, a
Retry-After header and X-Arc-Ingest-Refused: wal_not_durable, counted in
arc_wal_not_durable_rejections_total, and nothing of the request is
buffered. A request that spans several measurements is
written one measurement at a time: when the failure follows measurements
already buffered, the answer is instead a 500 naming what was written (see
a write that stops part-way,
v27.01.1+). Arc retracts the refused entry with a checkpoint, so recovery does
not replay it (v27.01.1+). If the checkpoint cannot be written either (on a
failing WAL disk it often cannot), the entry may still reach the WAL file and
replay after a restart, so a retry can duplicate rows (at-least-once). Such
retractions are counted in arc_wal_retractions_failed_total and logged at
Error, at most once every 10 seconds.
MQTT: MQTT ingestion follows the same rules: each message waits for its
sync. While the WAL refuses writes as not durable (a timed-out or failed sync:
the device is stalled or failing), subscriptions are suspended, as they are
for a fence, WAL disk pressure and a full buffer, and resume once a sync
succeeds (v27.01.1+); the writer syncs the WAL file once a second meanwhile,
so an idle node recovers too. With clean_session=false and QoS 1
or 2 the broker holds the messages. The message in flight when the refusal
starts is still briefly retried and then dropped, as for the other reasons,
and a merely slow device alternates between suspended and resumed at about
the ack timeout. arc_wal_durable_ack_failing is 1 while the state holds. In
a cluster with the origin stream, the entry is streamed to standbys before the
local sync, so a standby can hold a write the client was refused.
Cost: each request waits for the sync that covers its entry, about one
fsync, shared between the requests in flight. Sequential small writers feel
it most. Measured on an M3 Max (8.7M rows of MessagePack columnar writes,
fsync about 7 ms): 50,000-row requests from one client 4.7M rows/s with
queued, 2.9M with durable; 5,000-row requests from one client 2.8M and
0.66M; 5,000-row requests from eight clients 9.4M and 2.4M. Under sustained
load the backpressure keeps memory bounded. Throughput-first benchmarks
should set queued to compare with numbers published before v27.01.1. A WAL
device that stalls refuses the writes waiting on it after
ack_timeout_seconds (default 30; 0 = no bound, v27.01.1+): each is
answered 503 with X-Arc-Ingest-Refused: wal_not_durable and its WAL entry
is retracted without waiting on the device, where queued would drop entries
and keep acknowledging. Watch arc_wal_durable_ack_timeouts_total (a stalled
device), arc_wal_failed_syncs_total (a failing one) and
arc_wal_not_durable_rejections_total.
Not supported with cluster.wal_coverage = true. auto resolves to
queued there and logs a config warning; an explicit durable is refused at
startup. The covered append commits under the ingest lock, where waiting for a
sync would stall other writers.
Rotation settings
Control when WAL files rotate:
[wal]
max_size_mb = 100 # Rotate when file reaches 100MB
max_age_seconds = 3600 # Rotate after 1 hour (even if file is small)Why rotation matters:
- Prevents unbounded growth
- Faster recovery (smaller files)
- Automatic cleanup of old WALs
Operations
When WAL files are deleted (v27.01.1+)
A WAL file is deleted once every entry in it has reached Parquet in storage
(and, in a cluster with WAL coverage, once its batches are registered). Arc
checks this as soon as a file rotates or a flush completes, at most once per
second, so the WAL directory holds roughly ingest rate × buffer age
(ingest.max_buffer_age_ms, default 5 s) plus the file being written.
Before v27.01.1 the check ran only every wal.recovery_interval_seconds
(default 300 s), so a node ingesting 1M rows/s kept about 7 GB of already
flushed WAL between checks. The periodic pass still runs as a backstop.
Files whose data has not reached storage are kept: if storage is unreachable, the WAL grows until it is back, then shrinks within seconds.
When the WAL volume runs low (v27.01.1+)
Arc refuses live writes before the WAL volume fills, instead of accepting writes it cannot log. It reads the free space on the WAL volume every second, and at once after a rotation or a failed write. Between readings it counts every byte admitted to the WAL, and each request reserves its size before anything is written.
When free space falls below wal.disk_min_free_mb:
- Writes are refused with 503 Service Unavailable,
Retry-After: 5and the headerX-Arc-Ingest-Refused: wal_disk.- Nothing from a refused request is written, so retrying it is safe.
- A request is accepted or refused whole: a line-protocol write that spans several measurements is never half written.
- MQTT subscriptions disconnect, then reconnect when writes are accepted again.
Subscriptions with
clean_session=falseand QoS 1 or 2 find their messages queued at the broker. Subscriptions withclean_session=truelose what is published in the meantime. - A scheduled continuous query skips its run. The next run covers the window.
- WAL replay, recovery and replication are never refused, so the WAL can always
drain.
/healthand/readyare unaffected.
Writes are accepted again once free space is back above disk_min_free_mb plus
two WAL files (max_size_mb), and no sooner than 5 s after they were refused.
Starting on a full volume. Arc keeps a 64 MiB reserve file,
.arc-wal-reserve, in the WAL directory. A node that starts with less than
32 MiB free deletes it before anything else writes, logs an error and starts.
It then refuses writes until there is room again. The reserve is created again
once free space allows.
| Setting | Default | Meaning |
|---|---|---|
wal.disk_pressure_enabled | true | Turns the check and the reserve file on or off. |
wal.disk_min_free_mb | 0 | Free space below which writes are refused. 0 means max(512, 2 × max_size_mb). It is capped at a fifth of the volume, with a warning. |
wal.max_directory_mb | 0 | Cap on the WAL directory's size (0 = none). Use it where free space does not show the real limit, such as a Kubernetes emptyDir. A cap below four WAL files is raised to that, with a warning. Quarantined *.failed files do not count toward it. |
Each setting has an ARC_WAL_* environment variable, for example
ARC_WAL_DISK_MIN_FREE_MB. The "WAL enabled" startup line logs the effective
reserve, the level at which writes resume, and the cap.
Recovery on startup
Arc automatically recovers from WAL files on startup:
2025-10-08 14:30:00 [INFO] WAL recovery started: 4 files
2025-10-08 14:30:01 [INFO] Recovering WAL: worker-1-20251008_143000.wal
2025-10-08 14:30:01 [INFO] WAL read complete: 1000 entries, 5242880 bytes, 0 corrupted
2025-10-08 14:30:02 [INFO] Recovering WAL: worker-2-20251008_143000.wal
...
2025-10-08 14:30:05 [INFO] WAL recovery complete: 4000 batches, 200000 entries, 0 corrupted
2025-10-08 14:30:05 [INFO] WAL archived: worker-1-20251008_143000.wal.recoveredProcess:
- Find all
*.walfiles inWAL_DIR - Read and validate each entry (checksum verification)
- Replay records into buffer system
- Archive recovered WAL as
*.wal.recovered - Continue normal operations
Recovery time:
- ~5 seconds per 100MB WAL file
- Parallel recovery across workers
- Corrupted entries are skipped (logged)
Replay skips entries that already reached Parquet
Since v26.09.3, recovery replays only entries whose batch had not yet been written to Parquet.
Every WAL entry carries a tracked identity, and a checkpoint recording flushed identities is appended to the WAL — and durably synced — only after the Parquet write for that batch succeeds. Recovery gathers those checkpoints across all WAL files, including recently rotated ones and the active file, and skips every entry they cover. Measured on a hard kill -9 against a tagless measurement with 310 acknowledged records of which 300 had flushed: recovery replays 10 entries and the post-restart row count is exactly 310.
The ordering is deliberate and load-bearing: a checkpoint is written only after the flush is confirmed, never before, because advancing it early would convert duplicate rows into lost rows. One window remains — a crash in the gap between the Parquet write completing and the checkpoint becoming durable replays that batch. So replay is still at-least-once in the strict sense, but the window is now a single in-flight batch rather than a whole WAL file.
Before v26.09.3 recovery replayed every entry it found, including already-durable ones:
- Tagged measurements: duplicates were exact copies and were removed the next time compaction merged the partition (dedup keys on tags + time), so queries read high between recovery and that pass.
- Measurements without tags: compaction deliberately does not dedup tagless data (two tagless rows with one timestamp can be two legitimate events), so those duplicates persisted permanently. If you run tagless ingest with the WAL enabled and have restarted after a hard crash on an earlier version, historical over-counts from that replay are still in your data; they are not repaired retroactively.
The replay window is bounded by rotation (max_size_mb, max_age_seconds). Graceful shutdown is unaffected: it flushes buffers and purges the WAL cleanly, so replay only comes into play after a hard crash (power loss, OOM kill, kill -9).
Monitoring
Arc has no WAL-specific API endpoint. Monitor the WAL through the arc_wal_*
counters on the Prometheus endpoint, GET /metrics, which needs no token:
curl -s http://localhost:8000/metrics | grep '^arc_wal_'On Arc 26.09.4 this prints six counters:
arc_wal_records_preserved_total 0
arc_wal_recovery_total 0
arc_wal_recovery_records_total 0
arc_wal_dropped_entries_total 0
arc_wal_failed_writes_total 0
arc_wal_oversized_payloads_total 0| Metric | Meaning |
|---|---|
arc_wal_records_preserved_total | Records left to WAL recovery because their flush or shutdown write failed |
arc_wal_recovery_total | Successful WAL recovery runs |
arc_wal_recovery_records_total | Records replayed from the WAL |
arc_wal_dropped_entries_total | Entries dropped because the WAL buffer was full; a rising value means data can be lost |
arc_wal_failed_writes_total | WAL write failures |
arc_wal_oversized_payloads_total | Payloads refused for exceeding the single-entry cap even after chunking |
The same six counters appear as wal_* keys in the JSON from
GET /api/v1/metrics. Arc 27.01.1 adds more arc_wal_* metrics; the
monitoring playbook lists them, marking those
that need 27.01.1, and the alerts worth paging on.
To see the WAL files themselves, list the WAL directory (wal.directory,
default ./data/wal) on the node.
Troubleshooting
WAL recovery taking too long
Symptoms:
2025-10-08 14:30:00 [INFO] WAL recovery started: 50 files
... (minutes pass) ...Solutions:
-
Adjust rotation settings:
[wal] max_size_mb = 50 # Smaller files, faster recovery max_age_seconds = 1800 # Rotate more frequently -
Use faster disks for WAL:
[wal] directory = "/mnt/nvme/arc-wal" # NVMe SSD -
Use faster storage:
- NVMe SSD for WAL directory
- Separate disk from data storage
WAL disk space growing
Symptoms:
$ du -sh ./data/wal
5.2G ./data/walSolutions:
-
Check that flushes reach storage. WAL files are only deleted once their data is in storage. Look for flush or upload errors in the logs; when storage is back, the directory shrinks within seconds.
-
Remove leftover recovered files. Files ending in
.wal.recoveredare kept after a recovery and can be deleted:rm -f ./data/wal/*.wal.recovered -
Size the volume for outages. While storage is unreachable the WAL grows at your ingest rate. Leave room for the longest storage outage you want to ride through.
WAL write failures
Since v27.01.1, when a WAL write fails, Arc cuts the partial entry off the
end of the file before writing again, so later entries stay readable. On a
full disk (ENOSPC, EDQUOT) it does not create a new WAL file for each
failed write. A failed write that a retry on a new file recovers is counted
in arc_wal_recovered_writes_total; an entry that could not be written is
counted in arc_wal_failed_writes_total (and, when the disk was full, in
arc_wal_disk_full_writes_total). Its rows reach storage only if their
buffer's flush succeeds.
Symptoms:
2025-10-08 14:30:00 [ERROR] WAL append failed: [Errno 28] No space left on deviceSolutions:
-
Check disk space:
df -h /path/to/WAL_DIR -
Check permissions:
ls -ld ./data/wal chmod 755 ./data/wal -
Move WAL to larger disk:
[wal] directory = "/mnt/large-disk/arc-wal"
Performance degradation with WAL
Symptoms:
- Ingest throughput dropped sharply after enabling WAL
- High CPU usage from fsync calls
Solutions:
-
Check whether the WAL keeps up. A rising
arc_wal_dropped_entries_totalmeans writes arrive faster than the WAL disk can take them: those writes are acknowledged without a WAL copy. Before 27.01.1 this happened at about 170 MB/s of WAL on network-attached storage regardless of CPU (see How syncs work); upgrade, or move the WAL to faster storage. Switching sync modes to chase throughput will not help. -
Check disk I/O wait:
iostat -x 1 # Look for %iowait > 50% -
Move WAL to faster disk:
[wal] directory = "/mnt/nvme/arc-wal" -
Consider disabling WAL if durability isn't critical:
[wal] enabled = false
Best practices
Production deployment
Recommended configuration:
[wal]
enabled = true
sync_mode = "fdatasync"
directory = "/mnt/fast-ssd/arc-wal"
max_size_mb = 100
max_age_seconds = 3600Monitoring setup:
- Monitor WAL disk usage
- Alert on write failures
- Track recovery time during restarts
- Log rotation metrics
Backup strategy:
- WAL files are ephemeral (deleted after recovery)
- Don't backup WAL files directly
- Backup final Parquet files in S3/MinIO instead
Development/testing
Recommended configuration:
[wal]
enabled = false # WAL disabled for maximum speedOr if testing WAL:
[wal]
enabled = true
sync_mode = "async"
max_size_mb = 10 # Small files for testingSummary
Enable WAL if:
- Zero data loss is required
- Regulated industry (finance, healthcare)
- Can accept 19% throughput reduction
Disable WAL if:
- Maximum throughput is priority
- Can tolerate 0-5s data loss risk
- Have upstream retry/queue mechanisms
Recommended settings:
[wal]
enabled = true
sync_mode = "fdatasync" # Best balance
directory = "/mnt/nvme/arc-wal" # Fast diskNext steps
- Configure Compaction - Optimize query performance
- Monitor Arc - Set up health checks
- Performance Tuning - Maximize throughput