Write-Ahead Log (WAL)

Arc's Write-Ahead Log (WAL) provides zero data loss guarantees on system crashes.

Overview

WAL is an optional durability feature that persists all incoming data to disk before acknowledging writes. When enabled, Arc guarantees that data can be recovered even if the instance crashes.

When to enable WAL

Enable WAL if you need:

• Zero data loss on system crashes
• Guaranteed durability for regulatory compliance (finance, healthcare)
• Recovery from unexpected failures (power loss, OOM kills)

Keep WAL disabled if you:

• Prioritize maximum throughput
• Can tolerate 0-5 seconds data loss on rare crashes
• Have client-side retry logic or message queue upstream

Performance vs durability tradeoff

Tradeoff: a modest throughput reduction for near-zero data loss.

The cost is in enabling the WAL at all, not in the sync mode: choose the sync mode for the durability semantics you need, not for speed.

How syncs work

Arc does not sync on every write. A background writer appends entries to the WAL file, and a separate sync runs beside it: one is requested once sync_bytes of written data is unsynced (default 1 MiB), and every 100 ms while anything is unsynced. Each sync covers everything written before it started, and the writer keeps appending while it runs, so WAL throughput follows the disk's bandwidth rather than its round-trip latency (since 27.01.1, #1246).

Before 27.01.1 the writer waited for each sync itself, which capped WAL-on ingest at roughly 170 MB/s of WAL on AWS gp3 storage (about 3.5M MessagePack points per second), whatever the CPU count. If the WAL cannot keep up, its queue fills and entries are dropped while the write is still acknowledged: watch arc_wal_dropped_entries_total.

A buffer flush writes a checkpoint to the WAL, and that checkpoint is always synced before the flush completes, in every sync mode, async included. A sync that fails is counted in arc_wal_failed_syncs_total; the checkpoints in that WAL file then fail too, and Arc moves to a new WAL file, because a later sync of the same file could succeed over data the disk already reported lost.

Architecture

Data flow with WAL

┌──────────────────────────────────────────────────────────┐
│  HTTP Request (MessagePack or Line Protocol)             │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  1. WAL.append(records)                                  │
│     - Serialize to MessagePack binary                    │
│     - Calculate CRC32 checksum                           │
│     - Write to disk                                      │
│     - fdatasync() ← Force physical disk sync             │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼ Data is DURABLE (on disk)
┌──────────────────────────────────────────────────────────┐
│  2. HTTP 202 Accepted ← Response to client               │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  3. Buffer.write(records)                                │
│     - Add to in-memory buffer                            │
│     - Flush when 50K records or 5 seconds               │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  4. Parquet Writer                                       │
│     - Convert to Arrow columnar format                   │
│     - Write Parquet file                                 │
│     - Upload to S3/MinIO                                 │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  5. WAL.mark_completed() ← Can now delete WAL entry      │
└──────────────────────────────────────────────────────────┘

WAL files

Arc uses a single WAL writer with goroutines for concurrent access:

./data/wal/
├── arc-20251008_140530.wal
└── arc-20251008_150530.wal

Benefits:

• Simple implementation
• Automatic rotation
• Parallel recovery on startup

Configuration

Enable WAL

Edit arc.toml:

[wal]
enabled = true
sync_mode = "fdatasync"    # Recommended for production
directory = "./data/wal"
max_size_mb = 500          # Rotate at 500MB
max_age_seconds = 3600     # Rotate after 1 hour
sync_bytes = 1048576       # Request a sync once this many bytes are unsynced (default 1 MiB)

Or via environment variables:

ARC_WAL_ENABLED=true
ARC_WAL_DIRECTORY=./data/wal
ARC_WAL_SYNC_MODE=fdatasync
ARC_WAL_MAX_SIZE_MB=500
ARC_WAL_MAX_AGE_SECONDS=3600
ARC_WAL_SYNC_BYTES=1048576

Sync modes

Arc supports three sync modes with different durability/performance tradeoffs:

fdatasync (recommended)

[wal]
sync_mode = "fdatasync"

How it works:

• Syncs data to disk (file contents)
• Skips the metadata-only journal flush (timestamps)
• Marginally cheaper than fsync at equal durability for Arc's access pattern

Guarantees:

• Data is on physical disk
• Can recover all data on crash

Platform support: fdatasync(2) is used on Linux. macOS and Windows do not expose it, so Arc falls back to a full fsync there and logs this once at startup:

fdatasync is unavailable on this platform; using full fsync instead

The WAL startup log also reports fdatasync_supported, so the effective behavior is visible without guessing.

Do not expect a large throughput change from this setting. Syncs run beside the WAL writer (see How syncs work), so the sync mode determines durability semantics far more than throughput. On Linux the saving over fsync is the inode's timestamp metadata; because the WAL file grows on every append, the changed file size must still be persisted, so the practical difference is small — larger on rotational or network-backed storage than on NVMe.

Use case: Production deployments (recommended)

fsync (maximum safety)

[wal]
sync_mode = "fsync"

How it works:

• Syncs both data AND metadata to disk
• Slowest, but absolute guarantee

Use when:

• Regulatory compliance requires it
• Zero tolerance for any data loss
• Performance is secondary

async (performance-first)

[wal]
sync_mode = "async"

How it works:

• Writes to OS buffer cache
• No explicit sync of data (OS flushes periodically); flush checkpoints are still synced
• Very fast, but small risk window

Use when:

• Need 90% of original throughput
• Can tolerate ~1 second data loss
• Have upstream retry mechanisms

Rotation settings

Control when WAL files rotate:

[wal]
max_size_mb = 100           # Rotate when file reaches 100MB
max_age_seconds = 3600      # Rotate after 1 hour (even if file is small)

Why rotation matters:

• Prevents unbounded growth
• Faster recovery (smaller files)
• Automatic cleanup of old WALs

Operations

When WAL files are deleted (v27.01.1+)

A WAL file is deleted once every entry in it has reached Parquet in storage (and, in a cluster with WAL coverage, once its batches are registered). Arc checks this as soon as a file rotates or a flush completes, at most once per second, so the WAL directory holds roughly ingest rate × buffer age (ingest.max_buffer_age_ms, default 5 s) plus the file being written.

Before v27.01.1 the check ran only every wal.recovery_interval_seconds (default 300 s), so a node ingesting 1M rows/s kept about 7 GB of already flushed WAL between checks. The periodic pass still runs as a backstop.

Files whose data has not reached storage are kept: if storage is unreachable, the WAL grows until it is back, then shrinks within seconds.

When the WAL volume runs low (v27.01.1+)

Arc refuses live writes before the WAL volume fills, instead of accepting writes it cannot log. It reads the free space on the WAL volume every second, and at once after a rotation or a failed write. Between readings it counts every byte admitted to the WAL, and each request reserves its size before anything is written.

When free space falls below wal.disk_min_free_mb:

• Writes are refused with 503 Service Unavailable, Retry-After: 5 and the header X-Arc-Ingest-Refused: wal_disk.
  • Nothing from a refused request is written, so retrying it is safe.
  • A request is accepted or refused whole: a line-protocol write that spans several measurements is never half written.
• MQTT subscriptions disconnect, then reconnect when writes are accepted again. Subscriptions with clean_session=false and QoS 1 or 2 find their messages queued at the broker. Subscriptions with clean_session=true lose what is published in the meantime.
• A scheduled continuous query skips its run. The next run covers the window.
• WAL replay, recovery and replication are never refused, so the WAL can always drain. /health and /ready are unaffected.

Writes are accepted again once free space is back above disk_min_free_mb plus two WAL files (max_size_mb), and no sooner than 5 s after they were refused.

Starting on a full volume. Arc keeps a 64 MiB reserve file, .arc-wal-reserve, in the WAL directory. A node that starts with less than 32 MiB free deletes it before anything else writes, logs an error and starts. It then refuses writes until there is room again. The reserve is created again once free space allows.

Each setting has an ARC_WAL_* environment variable, for example ARC_WAL_DISK_MIN_FREE_MB. The "WAL enabled" startup line logs the effective reserve, the level at which writes resume, and the cap.

Recovery on startup

Arc automatically recovers from WAL files on startup:

2025-10-08 14:30:00 [INFO] WAL recovery started: 4 files
2025-10-08 14:30:01 [INFO] Recovering WAL: worker-1-20251008_143000.wal
2025-10-08 14:30:01 [INFO] WAL read complete: 1000 entries, 5242880 bytes, 0 corrupted
2025-10-08 14:30:02 [INFO] Recovering WAL: worker-2-20251008_143000.wal
...
2025-10-08 14:30:05 [INFO] WAL recovery complete: 4000 batches, 200000 entries, 0 corrupted
2025-10-08 14:30:05 [INFO] WAL archived: worker-1-20251008_143000.wal.recovered

Process:

1. Find all *.wal files in WAL_DIR
2. Read and validate each entry (checksum verification)
3. Replay the entries that were not yet flushed into the buffers
4. Once the replayed rows are flushed, delete the recovered WAL file
5. Continue normal operations

Recovery time:

• ~5 seconds per 100MB WAL file
• An entry that fails its checksum or cannot be replayed is logged, and its file is kept for the next pass (see below)

Files that keep failing are quarantined

A WAL file that holds an entry recovery cannot replay is kept and retried on the next pass. After three failed passes it is renamed to *.wal.failed (quarantined), so the files after it can be recovered. Arc never deletes a quarantined file: it holds the entries that could not be replayed, and it can hold flush checkpoints that other WAL files depend on. Keep it until you have dealt with it. Do not simply rename it back to .wal: recovery would replay every entry in it, including entries already in Parquet whose flush checkpoint has since been deleted, and duplicate those rows. Check which of its rows are missing from storage first.

An intact entry whose flush checkpoint shows it is already in Parquet is skipped without being decoded, so it never fails its file (since v27.01.1). Before v27.01.1, a columnar MessagePack write with a numeric measurement ("m": 5, stored as measurement_5) failed its WAL file this way, even after it was flushed.

Recovery also runs while the node is up, after a flush fails. When it quarantines a file the node itself wrote, the node stops waiting for that file's unflushed entries, so it can keep deleting WAL files whose data is flushed (since v27.01.1; before, WAL files piled up until a restart).

Replay skips entries that already reached Parquet

Since v26.09.3, recovery replays only entries whose batch had not yet been written to Parquet.

Every WAL entry carries a tracked identity, and a checkpoint recording flushed identities is appended to the WAL — and durably synced — only after the Parquet write for that batch succeeds. Recovery gathers those checkpoints across all WAL files, including recently rotated ones and the active file, and skips every entry they cover. Measured on a hard kill -9 against a tagless measurement with 310 acknowledged records of which 300 had flushed: recovery replays 10 entries and the post-restart row count is exactly 310.

The ordering is deliberate and load-bearing: a checkpoint is written only after the flush is confirmed, never before, because advancing it early would convert duplicate rows into lost rows. One window remains — a crash in the gap between the Parquet write completing and the checkpoint becoming durable replays that batch. So replay is still at-least-once in the strict sense, but the window is now a single in-flight batch rather than a whole WAL file.

Before v26.09.3 recovery replayed every entry it found, including already-durable ones:

• Tagged measurements: duplicates were exact copies and were removed the next time compaction merged the partition (dedup keys on tags + time), so queries read high between recovery and that pass.
• Measurements without tags: compaction deliberately does not dedup tagless data (two tagless rows with one timestamp can be two legitimate events), so those duplicates persisted permanently. If you run tagless ingest with the WAL enabled and have restarted after a hard crash on an earlier version, historical over-counts from that replay are still in your data; they are not repaired retroactively.

The replay window is bounded by rotation (max_size_mb, max_age_seconds). Graceful shutdown is unaffected: it flushes buffers and purges the WAL cleanly, so replay only comes into play after a hard crash (power loss, OOM kill, kill -9).

Monitoring

WAL status

curl http://localhost:8000/api/wal/status \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "enabled": true,
  "configuration": {
    "sync_mode": "fdatasync",
    "worker_id": 1,
    "current_file": "./data/wal/worker-1-20251008_143000.wal"
  },
  "stats": {
    "current_size_mb": 45.2,
    "current_age_seconds": 1850,
    "total_entries": 5000,
    "total_bytes": 47382528,
    "total_syncs": 5000,
    "total_rotations": 2
  }
}

WAL files

curl http://localhost:8000/api/wal/files \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "active": [
    {
      "name": "worker-1-20251008_143000.wal",
      "size_mb": 45.2,
      "modified": 1696775400
    }
  ],
  "recovered": [
    {
      "name": "worker-1-20251008_120000.wal.recovered",
      "size_mb": 98.5,
      "modified": 1696768800
    }
  ],
  "total_size_mb": 143.7
}

Health check

curl http://localhost:8000/api/wal/health \
  -H "Authorization: Bearer $ARC_TOKEN"

Troubleshooting

WAL recovery taking too long

Symptoms:

2025-10-08 14:30:00 [INFO] WAL recovery started: 50 files
... (minutes pass) ...

Solutions:

1. Adjust rotation settings:

   [wal]
   max_size_mb = 50          # Smaller files, faster recovery
   max_age_seconds = 1800    # Rotate more frequently

2. Use faster disks for WAL:

   [wal]
   directory = "/mnt/nvme/arc-wal"   # NVMe SSD

3. Use faster storage:

   • NVMe SSD for WAL directory
   • Separate disk from data storage

WAL disk space growing

Symptoms:

$ du -sh ./data/wal
5.2G    ./data/wal

Solutions:

1. Check that flushes reach storage. WAL files are only deleted once their data is in storage. Look for flush or upload errors in the logs; when storage is back, the directory shrinks within seconds.

2. Remove leftover recovered files. Files ending in .wal.recovered are kept after a recovery and can be deleted:

   rm -f ./data/wal/*.wal.recovered

3. Size the volume for outages. While storage is unreachable the WAL grows at your ingest rate. Leave room for the longest storage outage you want to ride through.

WAL write failures

Since v27.01.1, when a WAL write fails, Arc cuts the partial entry off the end of the file before writing again, so later entries stay readable. On a full disk (ENOSPC, EDQUOT) it does not create a new WAL file for each failed write. A failed write that a retry on a new file recovers is counted in arc_wal_recovered_writes_total; an entry that could not be written is counted in arc_wal_failed_writes_total (and, when the disk was full, in arc_wal_disk_full_writes_total). Its rows reach storage only if their buffer's flush succeeds.

Symptoms:

2025-10-08 14:30:00 [ERROR] WAL append failed: [Errno 28] No space left on device

Solutions:

1. Check disk space:

   df -h /path/to/WAL_DIR

2. Check permissions:

   ls -ld ./data/wal
   chmod 755 ./data/wal

3. Move WAL to larger disk:

   [wal]
   directory = "/mnt/large-disk/arc-wal"

Performance degradation with WAL

Symptoms:

• Throughput dropped sharply after enabling WAL
• High CPU usage from fsync calls

Solutions:

1. Check whether the WAL keeps up. A rising arc_wal_dropped_entries_total means writes arrive faster than the WAL disk can take them: those writes are acknowledged without a WAL copy. Before 27.01.1 this happened at about 170 MB/s of WAL on network-attached storage regardless of CPU (see How syncs work); upgrade, or move the WAL to faster storage. Switching sync modes to chase throughput will not help.

2. Check disk I/O wait:

   iostat -x 1
   # Look for %iowait > 50%

3. Move WAL to faster disk:

   [wal]
   directory = "/mnt/nvme/arc-wal"

4. Consider disabling WAL if durability isn't critical:

   [wal]
   enabled = false

Best practices

Production deployment

Recommended configuration:

[wal]
enabled = true
sync_mode = "fdatasync"
directory = "/mnt/fast-ssd/arc-wal"
max_size_mb = 100
max_age_seconds = 3600

Monitoring setup:

1. Monitor WAL disk usage
2. Alert on write failures
3. Track recovery time during restarts
4. Log rotation metrics

Backup strategy:

• WAL files are ephemeral (deleted after recovery)
• Don't backup WAL files directly
• Backup final Parquet files in S3/MinIO instead

Development/testing

Recommended configuration:

[wal]
enabled = false  # WAL disabled for maximum speed

Or if testing WAL:

[wal]
enabled = true
sync_mode = "async"
max_size_mb = 10  # Small files for testing

Summary

Enable WAL if:

• Zero data loss is required
• Regulated industry (finance, healthcare)
• Can accept 19% throughput reduction

Disable WAL if:

• Maximum throughput is priority
• Can tolerate 0-5s data loss risk
• Have upstream retry/queue mechanisms

Recommended settings:

[wal]
enabled = true
sync_mode = "fdatasync"     # Best balance
directory = "/mnt/nvme/arc-wal"   # Fast disk

Next steps

• Configure Compaction - Optimize query performance
• Monitor Arc - Set up health checks
• Configuration Reference - Maximize throughput