Write-Ahead Log (WAL)

Arc's Write-Ahead Log (WAL) provides zero data loss guarantees on system crashes.

Overview

WAL is an optional durability feature that persists all incoming data to disk before acknowledging writes. When enabled, Arc guarantees that data can be recovered even if the instance crashes.

When to enable WAL

Enable WAL if you need:

• Zero data loss on system crashes
• Guaranteed durability for regulatory compliance (finance, healthcare)
• Recovery from unexpected failures (power loss, OOM kills)

Keep WAL disabled if you:

• Prioritize maximum throughput
• Can tolerate 0-5 seconds data loss on rare crashes
• Have client-side retry logic or message queue upstream

Performance vs durability tradeoff

Tradeoff: a modest throughput reduction for near-zero data loss.

The cost is in enabling the WAL at all, not in the sync mode: choose the sync mode for the durability semantics you need, not for speed.

How syncs work

Arc does not sync on every write. A background writer appends entries to the WAL file, and a separate sync runs beside it: one is requested once sync_bytes of written data is unsynced (default 1 MiB), and every 100 ms while anything is unsynced. Each sync covers everything written before it started, and the writer keeps appending while it runs, so WAL throughput follows the disk's bandwidth rather than its round-trip latency (since 27.01.1, #1246).

Before 27.01.1 the writer waited for each sync itself, which capped WAL-on ingest at roughly 170 MB/s of WAL on AWS gp3 storage (about 3.5M MessagePack points per second), whatever the CPU count. If the WAL cannot keep up, its queue fills and entries are dropped while the write is still acknowledged: watch arc_wal_dropped_entries_total.

A buffer flush writes a checkpoint to the WAL, and that checkpoint is always synced before the flush completes, in every sync mode, async included. A sync that fails is counted in arc_wal_failed_syncs_total; the checkpoints in that WAL file then fail too, and Arc moves to a new WAL file, because a later sync of the same file could succeed over data the disk already reported lost.

Architecture

Data flow with WAL

┌──────────────────────────────────────────────────────────┐
│  HTTP Request (MessagePack or Line Protocol)             │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  1. WAL.append(records)                                  │
│     - Serialize to MessagePack binary                    │
│     - Calculate CRC32 checksum                           │
│     - Write to disk                                      │
│     - fdatasync() ← Force physical disk sync             │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼ Data is DURABLE (on disk)
┌──────────────────────────────────────────────────────────┐
│  2. HTTP 202 Accepted ← Response to client               │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  3. Buffer.write(records)                                │
│     - Add to in-memory buffer                            │
│     - Flush when 50K records or 5 seconds               │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  4. Parquet Writer                                       │
│     - Convert to Arrow columnar format                   │
│     - Write Parquet file                                 │
│     - Upload to S3/MinIO                                 │
└──────────────────┬───────────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────────────────┐
│  5. WAL.mark_completed() ← Can now delete WAL entry      │
└──────────────────────────────────────────────────────────┘

WAL files

Arc uses a single WAL writer with goroutines for concurrent access:

./data/wal/
├── arc-20251008_140530.wal
└── arc-20251008_150530.wal

Benefits:

• Simple implementation
• Automatic rotation
• Parallel recovery on startup

Configuration

Enable WAL

Edit arc.toml:

[wal]
enabled = true
sync_mode = "fdatasync"    # Recommended for production
directory = "./data/wal"
max_size_mb = 500          # Rotate at 500MB
max_age_seconds = 3600     # Rotate after 1 hour
sync_bytes = 1048576       # Request a sync once this many bytes are unsynced (default 1 MiB)

Or via environment variables:

ARC_WAL_ENABLED=true
ARC_WAL_DIRECTORY=./data/wal
ARC_WAL_SYNC_MODE=fdatasync
ARC_WAL_MAX_SIZE_MB=500
ARC_WAL_MAX_AGE_SECONDS=3600
ARC_WAL_SYNC_BYTES=1048576

Sync modes

Arc supports three sync modes with different durability/performance tradeoffs:

fdatasync (recommended)

[wal]
sync_mode = "fdatasync"

How it works:

• Syncs data to disk (file contents)
• Skips the metadata-only journal flush (timestamps)
• Marginally cheaper than fsync at equal durability for Arc's access pattern

Guarantees:

• Data is on physical disk
• Can recover all data on crash

Platform support: fdatasync(2) is used on Linux. macOS and Windows do not expose it, so Arc falls back to a full fsync there and logs this once at startup:

fdatasync is unavailable on this platform; using full fsync instead

The WAL startup log also reports fdatasync_supported, so the effective behavior is visible without guessing.

Do not expect a large throughput change from this setting. Syncs run beside the WAL writer (see How syncs work), so the sync mode determines durability semantics far more than throughput. On Linux the saving over fsync is the inode's timestamp metadata; because the WAL file grows on every append, the changed file size must still be persisted, so the practical difference is small — larger on rotational or network-backed storage than on NVMe.

Use case: Production deployments (recommended)

fsync (maximum safety)

[wal]
sync_mode = "fsync"

How it works:

• Syncs both data AND metadata to disk
• Slowest, but absolute guarantee

Use when:

• Regulatory compliance requires it
• Zero tolerance for any data loss
• Performance is secondary

async (performance-first)

[wal]
sync_mode = "async"

How it works:

• Writes to OS buffer cache
• No explicit sync of data (OS flushes periodically); flush checkpoints are still synced
• Very fast, but small risk window

Use when:

• Need 90% of original throughput
• Can tolerate ~1 second data loss
• Have upstream retry mechanisms

Rotation settings

Control when WAL files rotate:

[wal]
max_size_mb = 100           # Rotate when file reaches 100MB
max_age_seconds = 3600      # Rotate after 1 hour (even if file is small)

Why rotation matters:

• Prevents unbounded growth
• Faster recovery (smaller files)
• Automatic cleanup of old WALs

Operations

Recovery on startup

Arc automatically recovers from WAL files on startup:

2025-10-08 14:30:00 [INFO] WAL recovery started: 4 files
2025-10-08 14:30:01 [INFO] Recovering WAL: worker-1-20251008_143000.wal
2025-10-08 14:30:01 [INFO] WAL read complete: 1000 entries, 5242880 bytes, 0 corrupted
2025-10-08 14:30:02 [INFO] Recovering WAL: worker-2-20251008_143000.wal
...
2025-10-08 14:30:05 [INFO] WAL recovery complete: 4000 batches, 200000 entries, 0 corrupted
2025-10-08 14:30:05 [INFO] WAL archived: worker-1-20251008_143000.wal.recovered

Process:

1. Find all *.wal files in WAL_DIR
2. Read and validate each entry (checksum verification)
3. Replay the entries that were not yet flushed into the buffers
4. Once the replayed rows are flushed, delete the recovered WAL file
5. Continue normal operations

Recovery time:

• ~5 seconds per 100MB WAL file
• An entry that fails its checksum or cannot be replayed is logged, and its file is kept for the next pass (see below)

Files that keep failing are quarantined

A WAL file that holds an entry recovery cannot replay is kept and retried on the next pass. After three failed passes it is renamed to *.wal.failed (quarantined), so the files after it can be recovered. Arc never deletes a quarantined file: it holds the entries that could not be replayed, and it can hold flush checkpoints that other WAL files depend on. Keep it until you have dealt with it. Do not simply rename it back to .wal: recovery would replay every entry in it, including entries already in Parquet whose flush checkpoint has since been deleted, and duplicate those rows. Check which of its rows are missing from storage first.

An intact entry whose flush checkpoint shows it is already in Parquet is skipped without being decoded, so it never fails its file (since v27.01.1). Before v27.01.1, a columnar MessagePack write with a numeric measurement ("m": 5, stored as measurement_5) failed its WAL file this way, even after it was flushed.

Recovery also runs while the node is up, after a flush fails. When it quarantines a file the node itself wrote, the node stops waiting for that file's unflushed entries, so it can keep deleting WAL files whose data is flushed (since v27.01.1; before, WAL files piled up until a restart).

Replay skips entries that already reached Parquet

Since v26.09.3, recovery replays only entries whose batch had not yet been written to Parquet.

Every WAL entry carries a tracked identity, and a checkpoint recording flushed identities is appended to the WAL — and durably synced — only after the Parquet write for that batch succeeds. Recovery gathers those checkpoints across all WAL files, including recently rotated ones and the active file, and skips every entry they cover. Measured on a hard kill -9 against a tagless measurement with 310 acknowledged records of which 300 had flushed: recovery replays 10 entries and the post-restart row count is exactly 310.

The ordering is deliberate and load-bearing: a checkpoint is written only after the flush is confirmed, never before, because advancing it early would convert duplicate rows into lost rows. One window remains — a crash in the gap between the Parquet write completing and the checkpoint becoming durable replays that batch. So replay is still at-least-once in the strict sense, but the window is now a single in-flight batch rather than a whole WAL file.

Before v26.09.3 recovery replayed every entry it found, including already-durable ones:

• Tagged measurements: duplicates were exact copies and were removed the next time compaction merged the partition (dedup keys on tags + time), so queries read high between recovery and that pass.
• Measurements without tags: compaction deliberately does not dedup tagless data (two tagless rows with one timestamp can be two legitimate events), so those duplicates persisted permanently. If you run tagless ingest with the WAL enabled and have restarted after a hard crash on an earlier version, historical over-counts from that replay are still in your data; they are not repaired retroactively.

The replay window is bounded by rotation (max_size_mb, max_age_seconds). Graceful shutdown is unaffected: it flushes buffers and purges the WAL cleanly, so replay only comes into play after a hard crash (power loss, OOM kill, kill -9).

Monitoring

WAL status

curl http://localhost:8000/api/wal/status \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "enabled": true,
  "configuration": {
    "sync_mode": "fdatasync",
    "worker_id": 1,
    "current_file": "./data/wal/worker-1-20251008_143000.wal"
  },
  "stats": {
    "current_size_mb": 45.2,
    "current_age_seconds": 1850,
    "total_entries": 5000,
    "total_bytes": 47382528,
    "total_syncs": 5000,
    "total_rotations": 2
  }
}

WAL files

curl http://localhost:8000/api/wal/files \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "active": [
    {
      "name": "worker-1-20251008_143000.wal",
      "size_mb": 45.2,
      "modified": 1696775400
    }
  ],
  "recovered": [
    {
      "name": "worker-1-20251008_120000.wal.recovered",
      "size_mb": 98.5,
      "modified": 1696768800
    }
  ],
  "total_size_mb": 143.7
}

Health check

curl http://localhost:8000/api/wal/health \
  -H "Authorization: Bearer $ARC_TOKEN"

Cleanup old WAL files

# Cleanup files older than 24 hours (default)
curl -X POST http://localhost:8000/api/wal/cleanup \
  -H "Authorization: Bearer $ARC_TOKEN"

# Custom age (in hours)
curl -X POST "http://localhost:8000/api/wal/cleanup?max_age_hours=48" \
  -H "Authorization: Bearer $ARC_TOKEN"

Troubleshooting

WAL recovery taking too long

Symptoms:

2025-10-08 14:30:00 [INFO] WAL recovery started: 50 files
... (minutes pass) ...

Solutions:

1. Adjust rotation settings:

   [wal]
   max_size_mb = 50          # Smaller files, faster recovery
   max_age_seconds = 1800    # Rotate more frequently

2. Use faster disks for WAL:

   [wal]
   directory = "/mnt/nvme/arc-wal"   # NVMe SSD

3. Use faster storage:

   • NVMe SSD for WAL directory
   • Separate disk from data storage

WAL disk space growing

Symptoms:

$ du -sh ./data/wal
5.2G    ./data/wal

Solutions:

1. Manual cleanup:

   rm -f ./data/wal/*.wal.recovered

2. Reduce retention:

   [wal]
   max_size_mb = 50          # Rotate sooner
   max_age_seconds = 1800    # 30 minutes

3. Add cron job for cleanup:

   # Cleanup recovered WALs older than 24 hours
   0 2 * * * find /path/to/data/wal -name "*.wal.recovered" -mtime +1 -delete

WAL write failures

Symptoms:

2025-10-08 14:30:00 [ERROR] WAL append failed: [Errno 28] No space left on device

Solutions:

1. Check disk space:

   df -h /path/to/WAL_DIR

2. Check permissions:

   ls -ld ./data/wal
   chmod 755 ./data/wal

3. Move WAL to larger disk:

   [wal]
   directory = "/mnt/large-disk/arc-wal"

Performance degradation with WAL

Symptoms:

• Throughput dropped sharply after enabling WAL
• High CPU usage from fsync calls

Solutions:

1. Check whether the WAL keeps up. A rising arc_wal_dropped_entries_total means writes arrive faster than the WAL disk can take them: those writes are acknowledged without a WAL copy. Before 27.01.1 this happened at about 170 MB/s of WAL on network-attached storage regardless of CPU (see How syncs work); upgrade, or move the WAL to faster storage. Switching sync modes to chase throughput will not help.

2. Check disk I/O wait:

   iostat -x 1
   # Look for %iowait > 50%

3. Move WAL to faster disk:

   [wal]
   directory = "/mnt/nvme/arc-wal"

4. Consider disabling WAL if durability isn't critical:

   [wal]
   enabled = false

Best practices

Production deployment

Recommended configuration:

[wal]
enabled = true
sync_mode = "fdatasync"
directory = "/mnt/fast-ssd/arc-wal"
max_size_mb = 100
max_age_seconds = 3600

Monitoring setup:

1. Monitor WAL disk usage
2. Alert on write failures
3. Track recovery time during restarts
4. Log rotation metrics

Backup strategy:

• WAL files are ephemeral (deleted after recovery)
• Don't backup WAL files directly
• Backup final Parquet files in S3/MinIO instead

Development/testing

Recommended configuration:

[wal]
enabled = false  # WAL disabled for maximum speed

Or if testing WAL:

[wal]
enabled = true
sync_mode = "async"
max_size_mb = 10  # Small files for testing

Summary

Enable WAL if:

• Zero data loss is required
• Regulated industry (finance, healthcare)
• Can accept 19% throughput reduction

Disable WAL if:

• Maximum throughput is priority
• Can tolerate 0-5s data loss risk
• Have upstream retry/queue mechanisms

Recommended settings:

[wal]
enabled = true
sync_mode = "fdatasync"     # Best balance
directory = "/mnt/nvme/arc-wal"   # Fast disk

Next steps

• Configure Compaction - Optimize query performance
• Monitor Arc - Set up health checks
• Configuration Reference - Maximize throughput