BasekickLabs

Write-Ahead Log (WAL)

How the Arc Enterprise WAL bounds data loss to the sync interval on a writer node, the durability and throughput tradeoff per sync mode, and recovery behaviour after failover.

Arc's Write-Ahead Log (WAL) provides zero data loss guarantees on system crashes.

Important Clarification

There are two different WAL features in Arc:

  1. SQLite WAL mode (always enabled) - Internal mode for Arc's metadata database (arc.db). This enables concurrent access to connection settings, export jobs, and compaction locks. You'll see the log message "SQLite WAL mode enabled for concurrent access" on startup - this is expected and not related to data ingestion.

  2. Arc's WAL feature (disabled by default) - Optional durability feature for data ingestion that provides zero data loss guarantees. This page documents the Arc WAL feature, controlled by the WAL_ENABLED environment variable.

TL;DR: The startup log "SQLite WAL mode enabled" is normal and does NOT mean Arc's data ingestion WAL is enabled.

Overview

WAL is an optional durability feature that persists all incoming data to disk before acknowledging writes. When enabled, Arc guarantees that data can be recovered even if the instance crashes.

WAL is disabled by default to maximize ingest throughput. Enable it when zero data loss is required.

When to enable WAL

Enable WAL if you need:

  • Zero data loss on system crashes
  • Guaranteed durability for regulatory compliance (finance, healthcare)
  • Recovery from unexpected failures (power loss, OOM kills)

Keep WAL disabled if you:

  • Prioritize maximum ingest throughput
  • Can tolerate 0-5 seconds data loss on rare crashes
  • Have client-side retry logic or message queue upstream

Performance vs durability tradeoff

ConfigurationThroughputData Loss Risk
No WAL (default)Highest0-5 seconds
WAL enabled (default ack_mode, durable)Lower; each write waits for a syncNone of the acknowledged writes
WAL + ack_mode = "queued", asyncSlightly reduced<1 second of synced data, plus entries still queued
WAL + ack_mode = "queued", fdatasync or fsyncSlightly reducedEntries still queued or not yet synced at the crash

The loss figures for queued follow from acknowledging a write before it is synced; see Acknowledgement mode. With cluster.wal_coverage = true the default resolves to queued.

Tradeoff: throughput for no loss of acknowledged writes. The cost is in the acknowledgement mode (each write waits for a sync), not in the sync mode.

How syncs work

Arc does not sync on every write. A background writer appends entries to the WAL file, and a separate sync runs beside it: one is requested once sync_bytes of written data is unsynced (default 1 MiB), and every 100 ms while anything is unsynced. Each sync covers everything written before it started, and the writer keeps appending while it runs, so WAL throughput follows the disk's bandwidth rather than its round-trip latency (since 27.01.1, #1246).

Before 27.01.1 the writer waited for each sync itself, which capped WAL-on ingest at roughly 170 MB/s of WAL on AWS gp3 storage (about 3.5M MessagePack points per second), whatever the CPU count. If the WAL cannot keep up, its queue fills and entries are dropped while the write is still acknowledged: watch arc_wal_dropped_entries_total.

A buffer flush writes a checkpoint to the WAL, and that checkpoint is always synced before the flush completes, in every sync mode, async included. A sync that fails is counted in arc_wal_failed_syncs_total; the checkpoints in that WAL file then fail too, and Arc moves to a new WAL file, because a later sync of the same file could succeed over data the disk already reported lost.

Architecture

Data flow with WAL

┌──────────────────────────────────────────────────────────┐
│  HTTP Request (MessagePack or Line Protocol)             │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  1. WAL.append(records)                                  │
│     - Serialize to MessagePack binary                    │
│     - Calculate CRC32 checksum                           │
│     - Write to disk                                      │
│     - fdatasync() ← Force physical disk sync             │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼ Data is DURABLE (on disk)
┌──────────────────────────────────────────────────────────┐
│  2. HTTP 202 Accepted ← Response to client               │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  3. Buffer.write(records)                                │
│     - Add to in-memory buffer                            │
│     - Flush when 50K records or 5 seconds               │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  4. Parquet Writer                                       │
│     - Convert to Arrow columnar format                   │
│     - Write Parquet file                                 │
│     - Upload to S3/MinIO                                 │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  5. WAL.mark_completed() ← Can now delete WAL entry      │
└──────────────────────────────────────────────────────────┘

Key Insight

Once WAL confirms the write (step 1), the data is guaranteed durable even if Arc crashes before step 4 completes.

WAL files

Arc uses a single WAL writer with goroutines for concurrent access:

./data/wal/
├── arc-20251008_140530.wal
└── arc-20251008_150530.wal

Benefits:

  • Simple implementation
  • Automatic rotation
  • Parallel recovery on startup

Configuration

Enable WAL

Edit arc.toml:

[wal]
enabled = true
sync_mode = "fdatasync"    # Recommended for production
directory = "./data/wal"
max_size_mb = 500          # Rotate at 500MB
max_age_seconds = 3600     # Rotate after 1 hour
sync_bytes = 1048576       # Request a sync once this many bytes are unsynced (default 1 MiB)
ack_mode = "auto"          # durable with the WAL on (default, v27.01.1+); or queued, durable
ack_timeout_seconds = 30   # a durable write waits at most this long for its sync (v27.01.1+); 0 = no bound

Or via environment variables:

ARC_WAL_ENABLED=true
ARC_WAL_DIRECTORY=./data/wal
ARC_WAL_SYNC_MODE=fdatasync
ARC_WAL_MAX_SIZE_MB=500
ARC_WAL_MAX_AGE_SECONDS=3600
ARC_WAL_SYNC_BYTES=1048576
ARC_WAL_ACK_MODE=auto
ARC_WAL_ACK_TIMEOUT_SECONDS=30

Sync modes

Arc supports three sync modes with different durability/performance tradeoffs:

[wal]
sync_mode = "fdatasync"

How it works:

  • Syncs data to disk (file contents)
  • Skips metadata sync (file size, modified time)
  • A little less work than fsync (no timestamp metadata), same data durability

Guarantees:

  • Synced data is on physical disk and survives a crash
  • With the default ack_mode = "auto" (durable with the WAL on), a write is acknowledged only once a sync covering its entry has completed, so a crash loses no acknowledged write. With ack_mode = "queued" a write is acknowledged before its entry is synced, and entries still queued or written but unsynced at the crash are lost; see Acknowledgement mode
  • File metadata may be stale (not critical)

Use case: Production deployments (recommended)

fsync (maximum safety)

[wal]
sync_mode = "fsync"

How it works:

  • Syncs both data AND metadata to disk
  • Slowest; the strongest guarantee for synced data. Acknowledged writes are covered too with the default ack_mode (durable)

Use when:

  • Regulatory compliance requires it
  • Zero tolerance for any data loss (with the default ack_mode, durable)
  • Performance is secondary

async (performance-first)

[wal]
sync_mode = "async"

How it works:

  • Writes to OS buffer cache
  • No explicit sync of data (OS flushes periodically); flush checkpoints are still synced
  • Very fast, but small risk window

Use when:

  • Need 90% of original throughput
  • Can tolerate ~1 second data loss
  • Have upstream retry mechanisms

Acknowledgement mode (planned for v27.01.1)

Planned for v27.01.1

wal.ack_mode is merged for the planned v27.01.1 release, with auto (durable with the WAL on) as its default. Earlier versions always behave as queued.

ack_mode decides when Arc answers a write with the WAL enabled.

ack_modeA 2xx meansLost on a crashFull WAL queue
durable (what auto, the default, is with the WAL on)A sync covering the entry has completedNone of the acknowledged writesThe request waits (backpressure) instead of dropping
queuedThe entry is queued for the WAL writerEntries still queued, or written but not yet syncedEntries are dropped while the write is still acknowledged (arc_wal_dropped_entries_total)
auto (default)durable with the WAL on; queued with the WAL off or with cluster.wal_coverage = true, which durable does not support (logged as a config warning and in the WAL enabled startup line)

A write Arc refuses (schema churn on its measurement, a request its client cancelled, a node shutting down) leaves nothing to replay from the WAL (v27.01.1+). With queued, its entry is only queued once nothing can refuse the write any more; with durable, the entry is written first and a refusal retracts it with a checkpoint. Before v27.01.1, without WAL coverage, such an entry could be replayed after a crash or restart, storing a write the client was told failed.

[wal]
enabled = true
sync_mode = "fdatasync"
ack_mode = "queued"        # opt out of durable acknowledgement

How durable works: each request waits for the sync that covers its entry. Syncs are shared: every request whose entry was written before a sync started is answered by that one sync, so concurrent writes do not each pay a sync. A request that spans several measurements waits once per measurement. With sync_mode = "async", durable still syncs: a waiting write forces it.

Failures: if an entry cannot be made durable (a write or sync failure, a payload the WAL cannot hold, or a shutdown), the write is refused with 503, a Retry-After header and X-Arc-Ingest-Refused: wal_not_durable, counted in arc_wal_not_durable_rejections_total, and nothing of the request is buffered. A request that spans several measurements is written one measurement at a time: when the failure follows measurements already buffered, the answer is instead a 500 naming what was written (see a write that stops part-way, v27.01.1+). Arc retracts the refused entry with a checkpoint, so recovery does not replay it (v27.01.1+). If the checkpoint cannot be written either (on a failing WAL disk it often cannot), the entry may still reach the WAL file and replay after a restart, so a retry can duplicate rows (at-least-once). Such retractions are counted in arc_wal_retractions_failed_total and logged at Error, at most once every 10 seconds.

MQTT: MQTT ingestion follows the same rules: each message waits for its sync. While the WAL refuses writes as not durable (a timed-out or failed sync: the device is stalled or failing), subscriptions are suspended, as they are for a fence, WAL disk pressure and a full buffer, and resume once a sync succeeds (v27.01.1+); the writer syncs the WAL file once a second meanwhile, so an idle node recovers too. With clean_session=false and QoS 1 or 2 the broker holds the messages. The message in flight when the refusal starts is still briefly retried and then dropped, as for the other reasons, and a merely slow device alternates between suspended and resumed at about the ack timeout. arc_wal_durable_ack_failing is 1 while the state holds. In a cluster with the origin stream, the entry is streamed to standbys before the local sync, so a standby can hold a write the client was refused.

Cost: each request waits for the sync that covers its entry, about one fsync, shared between the requests in flight. Sequential small writers feel it most. Measured on an M3 Max (8.7M rows of MessagePack columnar writes, fsync about 7 ms): 50,000-row requests from one client 4.7M rows/s with queued, 2.9M with durable; 5,000-row requests from one client 2.8M and 0.66M; 5,000-row requests from eight clients 9.4M and 2.4M. Under sustained load the backpressure keeps memory bounded. Throughput-first benchmarks should set queued to compare with numbers published before v27.01.1. A WAL device that stalls refuses the writes waiting on it after ack_timeout_seconds (default 30; 0 = no bound, v27.01.1+): each is answered 503 with X-Arc-Ingest-Refused: wal_not_durable and its WAL entry is retracted without waiting on the device, where queued would drop entries and keep acknowledging. Watch arc_wal_durable_ack_timeouts_total (a stalled device), arc_wal_failed_syncs_total (a failing one) and arc_wal_not_durable_rejections_total.

Not supported with cluster.wal_coverage = true. auto resolves to queued there and logs a config warning; an explicit durable is refused at startup. The covered append commits under the ingest lock, where waiting for a sync would stall other writers.

Rotation settings

Control when WAL files rotate:

[wal]
max_size_mb = 100           # Rotate when file reaches 100MB
max_age_seconds = 3600      # Rotate after 1 hour (even if file is small)

Why rotation matters:

  • Prevents unbounded growth
  • Faster recovery (smaller files)
  • Automatic cleanup of old WALs

Operations

When WAL files are deleted (v27.01.1+)

A WAL file is deleted once every entry in it has reached Parquet in storage (and, in a cluster with WAL coverage, once its batches are registered). Arc checks this as soon as a file rotates or a flush completes, at most once per second, so the WAL directory holds roughly ingest rate × buffer age (ingest.max_buffer_age_ms, default 5 s) plus the file being written.

Before v27.01.1 the check ran only every wal.recovery_interval_seconds (default 300 s), so a node ingesting 1M rows/s kept about 7 GB of already flushed WAL between checks. The periodic pass still runs as a backstop.

Files whose data has not reached storage are kept: if storage is unreachable, the WAL grows until it is back, then shrinks within seconds.

When the WAL volume runs low (v27.01.1+)

Arc refuses live writes before the WAL volume fills, instead of accepting writes it cannot log. It reads the free space on the WAL volume every second, and at once after a rotation or a failed write. Between readings it counts every byte admitted to the WAL, and each request reserves its size before anything is written.

When free space falls below wal.disk_min_free_mb:

  • Writes are refused with 503 Service Unavailable, Retry-After: 5 and the header X-Arc-Ingest-Refused: wal_disk.
    • Nothing from a refused request is written, so retrying it is safe.
    • A request is accepted or refused whole: a line-protocol write that spans several measurements is never half written.
  • MQTT subscriptions disconnect, then reconnect when writes are accepted again. Subscriptions with clean_session=false and QoS 1 or 2 find their messages queued at the broker. Subscriptions with clean_session=true lose what is published in the meantime.
  • A scheduled continuous query skips its run. The next run covers the window.
  • WAL replay, recovery and replication are never refused, so the WAL can always drain. /health and /ready are unaffected.

Writes are accepted again once free space is back above disk_min_free_mb plus two WAL files (max_size_mb), and no sooner than 5 s after they were refused.

Starting on a full volume. Arc keeps a 64 MiB reserve file, .arc-wal-reserve, in the WAL directory. A node that starts with less than 32 MiB free deletes it before anything else writes, logs an error and starts. It then refuses writes until there is room again. The reserve is created again once free space allows.

SettingDefaultMeaning
wal.disk_pressure_enabledtrueTurns the check and the reserve file on or off.
wal.disk_min_free_mb0Free space below which writes are refused. 0 means max(512, 2 × max_size_mb). It is capped at a fifth of the volume, with a warning.
wal.max_directory_mb0Cap on the WAL directory's size (0 = none). Use it where free space does not show the real limit, such as a Kubernetes emptyDir. A cap below four WAL files is raised to that, with a warning. Quarantined *.failed files do not count toward it.

Each setting has an ARC_WAL_* environment variable, for example ARC_WAL_DISK_MIN_FREE_MB. The "WAL enabled" startup line logs the effective reserve, the level at which writes resume, and the cap.

Recovery on startup

Arc automatically recovers from WAL files on startup:

2025-10-08 14:30:00 [INFO] WAL recovery started: 4 files
2025-10-08 14:30:01 [INFO] Recovering WAL: worker-1-20251008_143000.wal
2025-10-08 14:30:01 [INFO] WAL read complete: 1000 entries, 5242880 bytes, 0 corrupted
2025-10-08 14:30:02 [INFO] Recovering WAL: worker-2-20251008_143000.wal
...
2025-10-08 14:30:05 [INFO] WAL recovery complete: 4000 batches, 200000 entries, 0 corrupted
2025-10-08 14:30:05 [INFO] WAL archived: worker-1-20251008_143000.wal.recovered

Process:

  1. Find all *.wal files in WAL_DIR
  2. Read and validate each entry (checksum verification)
  3. Replay records into buffer system
  4. Archive recovered WAL as *.wal.recovered
  5. Continue normal operations

Recovery time:

  • ~5 seconds per 100MB WAL file
  • Parallel recovery across workers
  • Corrupted entries are skipped (logged)

Replay skips entries that already reached Parquet

Since v26.09.3, recovery replays only entries whose batch had not yet been written to Parquet.

Every WAL entry carries a tracked identity, and a checkpoint recording flushed identities is appended to the WAL — and durably synced — only after the Parquet write for that batch succeeds. Recovery gathers those checkpoints across all WAL files, including recently rotated ones and the active file, and skips every entry they cover. Measured on a hard kill -9 against a tagless measurement with 310 acknowledged records of which 300 had flushed: recovery replays 10 entries and the post-restart row count is exactly 310.

The ordering is deliberate and load-bearing: a checkpoint is written only after the flush is confirmed, never before, because advancing it early would convert duplicate rows into lost rows. One window remains — a crash in the gap between the Parquet write completing and the checkpoint becoming durable replays that batch. So replay is still at-least-once in the strict sense, but the window is now a single in-flight batch rather than a whole WAL file.

Before v26.09.3 recovery replayed every entry it found, including already-durable ones:

  • Tagged measurements: duplicates were exact copies and were removed the next time compaction merged the partition (dedup keys on tags + time), so queries read high between recovery and that pass.
  • Measurements without tags: compaction deliberately does not dedup tagless data (two tagless rows with one timestamp can be two legitimate events), so those duplicates persisted permanently. If you run tagless ingest with the WAL enabled and have restarted after a hard crash on an earlier version, historical over-counts from that replay are still in your data; they are not repaired retroactively.

The replay window is bounded by rotation (max_size_mb, max_age_seconds). Graceful shutdown is unaffected: it flushes buffers and purges the WAL cleanly, so replay only comes into play after a hard crash (power loss, OOM kill, kill -9).

Monitoring

Arc has no WAL-specific API endpoint. Monitor the WAL through the arc_wal_* counters on the Prometheus endpoint, GET /metrics, which needs no token:

curl -s http://localhost:8000/metrics | grep '^arc_wal_'

On Arc 26.09.4 this prints six counters:

arc_wal_records_preserved_total 0
arc_wal_recovery_total 0
arc_wal_recovery_records_total 0
arc_wal_dropped_entries_total 0
arc_wal_failed_writes_total 0
arc_wal_oversized_payloads_total 0
MetricMeaning
arc_wal_records_preserved_totalRecords left to WAL recovery because their flush or shutdown write failed
arc_wal_recovery_totalSuccessful WAL recovery runs
arc_wal_recovery_records_totalRecords replayed from the WAL
arc_wal_dropped_entries_totalEntries dropped because the WAL buffer was full; a rising value means data can be lost
arc_wal_failed_writes_totalWAL write failures
arc_wal_oversized_payloads_totalPayloads refused for exceeding the single-entry cap even after chunking

The same six counters appear as wal_* keys in the JSON from GET /api/v1/metrics. Arc 27.01.1 adds more arc_wal_* metrics; the monitoring playbook lists them, marking those that need 27.01.1, and the alerts worth paging on.

To see the WAL files themselves, list the WAL directory (wal.directory, default ./data/wal) on the node.

Troubleshooting

WAL recovery taking too long

Symptoms:

2025-10-08 14:30:00 [INFO] WAL recovery started: 50 files
... (minutes pass) ...

Solutions:

  1. Adjust rotation settings:

    [wal]
    max_size_mb = 50          # Smaller files, faster recovery
    max_age_seconds = 1800    # Rotate more frequently
  2. Use faster disks for WAL:

    [wal]
    directory = "/mnt/nvme/arc-wal"   # NVMe SSD
  3. Use faster storage:

    • NVMe SSD for WAL directory
    • Separate disk from data storage

WAL disk space growing

Symptoms:

$ du -sh ./data/wal
5.2G    ./data/wal

Solutions:

  1. Check that flushes reach storage. WAL files are only deleted once their data is in storage. Look for flush or upload errors in the logs; when storage is back, the directory shrinks within seconds.

  2. Remove leftover recovered files. Files ending in .wal.recovered are kept after a recovery and can be deleted:

    rm -f ./data/wal/*.wal.recovered
  3. Size the volume for outages. While storage is unreachable the WAL grows at your ingest rate. Leave room for the longest storage outage you want to ride through.

WAL write failures

Since v27.01.1, when a WAL write fails, Arc cuts the partial entry off the end of the file before writing again, so later entries stay readable. On a full disk (ENOSPC, EDQUOT) it does not create a new WAL file for each failed write. A failed write that a retry on a new file recovers is counted in arc_wal_recovered_writes_total; an entry that could not be written is counted in arc_wal_failed_writes_total (and, when the disk was full, in arc_wal_disk_full_writes_total). Its rows reach storage only if their buffer's flush succeeds.

Symptoms:

2025-10-08 14:30:00 [ERROR] WAL append failed: [Errno 28] No space left on device

Solutions:

  1. Check disk space:

    df -h /path/to/WAL_DIR
  2. Check permissions:

    ls -ld ./data/wal
    chmod 755 ./data/wal
  3. Move WAL to larger disk:

    [wal]
    directory = "/mnt/large-disk/arc-wal"

Performance degradation with WAL

Symptoms:

  • Ingest throughput dropped sharply after enabling WAL
  • High CPU usage from fsync calls

Solutions:

  1. Check whether the WAL keeps up. A rising arc_wal_dropped_entries_total means writes arrive faster than the WAL disk can take them: those writes are acknowledged without a WAL copy. Before 27.01.1 this happened at about 170 MB/s of WAL on network-attached storage regardless of CPU (see How syncs work); upgrade, or move the WAL to faster storage. Switching sync modes to chase throughput will not help.

  2. Check disk I/O wait:

    iostat -x 1
    # Look for %iowait > 50%
  3. Move WAL to faster disk:

    [wal]
    directory = "/mnt/nvme/arc-wal"
  4. Consider disabling WAL if durability isn't critical:

    [wal]
    enabled = false

Best practices

Production deployment

Recommended configuration:

[wal]
enabled = true
sync_mode = "fdatasync"
directory = "/mnt/fast-ssd/arc-wal"
max_size_mb = 100
max_age_seconds = 3600

Monitoring setup:

  1. Monitor WAL disk usage
  2. Alert on write failures
  3. Track recovery time during restarts
  4. Log rotation metrics

Backup strategy:

  • WAL files are ephemeral (deleted after recovery)
  • Don't backup WAL files directly
  • Backup final Parquet files in S3/MinIO instead

Development/testing

Recommended configuration:

[wal]
enabled = false  # WAL disabled for maximum speed

Or if testing WAL:

[wal]
enabled = true
sync_mode = "async"
max_size_mb = 10  # Small files for testing

Summary

Enable WAL if:

  • Zero data loss is required
  • Regulated industry (finance, healthcare)
  • Can accept 19% throughput reduction

Disable WAL if:

  • Maximum throughput is priority
  • Can tolerate 0-5s data loss risk
  • Have upstream retry/queue mechanisms

Recommended settings:

[wal]
enabled = true
sync_mode = "fdatasync"     # Best balance
directory = "/mnt/nvme/arc-wal"   # Fast disk

Next steps

On this page