File Compaction

How Arc merges small Parquet files into larger ones on an hourly and daily schedule, why the small-file problem slows queries, and which compaction keys to tune.

Arc's automatic compaction system merges small Parquet files into larger, optimized files for dramatically faster queries.

Overview

Compaction is Arc's file optimization system that merges small files into larger ones, substantially improving query performance.

Key Features:

Automatic - Runs on schedule (default: hourly at :05)
Safe - Locked partitions prevent concurrent compaction
Efficient - Parallel, sorted merging by the query engine
Non-blocking - Queries work during compaction
Enabled by default - Essential for production

Why compaction matters

The small file problem

Arc's high-throughput ingestion creates many small files:

At high ingest rates with a 5-second flush interval:
→ 12 files per minute per measurement
→ 720 files per hour per measurement
→ 17,280 files per day per measurement

Impact on Queries:

Slow queries - The query engine must open/scan hundreds of files
High costs - More S3/MinIO API calls
Poor compression - Small files compress less efficiently
Reduced pruning - Less effective partition elimination

After compaction

Real Production Test Results:

Before: 2,704 small files (Snappy) = 3.7 GB
After:  3 compacted files (ZSTD)   = 724 MB

Compression: 80.4% space savings
File reduction: 901x fewer files (2,704 → 3)
Compaction time: 5 seconds

Per-Measurement Breakdown:

mem: 888 files → 1 file, 1,213 MB → 239 MB (80.3% compression)
disk: 906 files → 1 file, 1,237 MB → 242 MB (80.4% compression)
cpu: 910 files → 1 file, 1,246 MB → 243 MB (80.5% compression)

Query Performance:

Far fewer file opens - Single file scan vs hundreds
99% fewer API calls - Massive cost reduction (2,704 → 3 LIST operations)
80.4% compression - ZSTD compaction vs Snappy writes
Effective pruning - The query engine can skip entire files

How it works

Compaction flow

1. Scheduler wakes up (cron: "5 * * * *")
   ↓
2. Scan storage for eligible partitions
   ↓
3. For each partition:
   - Check age (&gt;1 hour old?)
   - Check file count (≥10 files?)
   - Check if already compacted?
   ↓
4. Acquire partition lock (SQLite)
   ↓
5. Download small files to temp directory
   ↓
6. Compact via the query engine (parallel, sorted)
   ↓
7. Upload compacted file to storage
   ↓
8. Delete old small files
   ↓
9. Release lock & cleanup temp files
   ↓
10. Repeat for next partition

Partition structure

Data is organized by hour:

arc/                              # Bucket
├── default/                      # Database
│   └── cpu/                      # Measurement
│       └── 2025/10/08/           # Date
│           ├── 14/               # Hour (2 PM) - Eligible for compaction
│           │   ├── file1.parquet (50 MB)
│           │   ├── file2.parquet (48 MB)
│           │   └── ...
│           ├── 15/               # Hour (3 PM) - Eligible for compaction
│           └── 16/               # Hour (4 PM) - CURRENT, skip!

Compaction merges all files in a partition (e.g., 2025/10/08/14/) into one optimized file.

Configuration

Default configuration

Compaction is enabled by default in arc.toml:

[compaction]
enabled = true

# Hourly tier
hourly_enabled = true
hourly_schedule = "5 * * * *"   # Cron schedule: every hour at :05
hourly_min_age_hours = 1        # Wait 1 hour before compacting (let the hour complete)
hourly_min_files = 10           # Only compact if >=10 files exist

# Daily tier
daily_enabled = true
daily_schedule = "0 3 * * *"    # Cron schedule: 3 AM daily
daily_min_age_hours = 24        # Wait 24 hours
daily_min_files = 12            # Only compact if >=12 files exist

max_concurrent = 2              # Run 2 compactions in parallel

Configuration options

Schedule

[compaction]
hourly_schedule = "5 * * * *"     # Every hour at :05 (default)
daily_schedule = "0 3 * * *"      # 3 AM daily (default)
# hourly_schedule = "0 */2 * * *"   # Every 2 hours at :00

Cron format: minute hour day month weekday

Minimum age

[compaction]
hourly_min_age_hours = 1    # Don't compact the current hour (default)
daily_min_age_hours = 24    # Daily tier waits a full day (default)
# hourly_min_age_hours = 2    # Wait 2 hours (more conservative)
# hourly_min_age_hours = 0    # Compact immediately (aggressive)

Minimum files

[compaction]
hourly_min_files = 10    # Only compact if >=10 files (default)
daily_min_files = 12     # Daily tier threshold (default)
# hourly_min_files = 50    # Only compact with many files
# hourly_min_files = 5     # Compact more aggressively

Concurrent jobs

[compaction]
max_concurrent = 2    # Run 2 compactions in parallel (default)
# max_concurrent = 4    # More parallelism (uses more CPU/memory)
# max_concurrent = 1    # Sequential (lower resource usage)

Memory limit and threads (per subprocess)

Each compaction job runs in an isolated subprocess with its own query engine instance. These keys bound that instance's resources:

[compaction]
memory_limit = ""    # Per-subprocess engine memory limit; "" (default) = auto
threads = 0          # Per-subprocess engine threads; 0 (default) = auto
# memory_limit = "2GB"   # Explicit cap
# threads = 4            # Explicit thread count

Env vars: ARC_COMPACTION_MEMORY_LIMIT, ARC_COMPACTION_THREADS.

Auto behavior:

memory_limit derives as database.memory_limit / max_concurrent, so all concurrent compaction jobs together stay within roughly one database.memory_limit. With database.memory_limit = "8GB" and the default concurrency of 2, each subprocess gets 4GB.
threads defaults to half the CPU cores (minimum 1), so the default two concurrent jobs together use about one machine's worth of cores, leaving headroom for ingest and queries.

Accepted memory_limit forms are absolute sizes with a unit: "8GB", "512MB", "0.5GB". Percent and unit-less forms are rejected at startup (DuckDB's SET memory_limit does not support them), as are other invalid values. The effective values appear in the startup log (subprocess_memory_limit, subprocess_threads).

When a job exceeds its memory limit, DuckDB spills to a duckdb-spill/ directory inside the job's temp directory (under compaction.temp_directory) — size that volume for your largest partitions. Spill files are removed by normal job cleanup and by the crash sweeps on startup.

Lower these when compaction competes with ingest for RAM during backfill catch-up (many partitions become candidates at once); raise them to make individual large compactions faster on dedicated compactor nodes.

Files per batch

A partition with more files than this is split into several batches, each compacted as an independent job producing its own output file.

[compaction]
max_files_per_batch = 30   # Files per compaction job (default)
# max_files_per_batch = 5    # Smaller outputs, more jobs per partition
# max_files_per_batch = 60   # Fewer, larger outputs

Valid range is 2–500. Values outside it fall back to the default with a startup warning; 1 is rejected because compaction's adaptive retry cannot process a single-file batch.

This bounds the file count per job, not the output size in bytes — compacted file size tracks input file size, which follows your ingest buffer settings. The main reason to lower it is transferring compacted files over a constrained or intermittent link (edge deployments), where smaller, independently-transferable files resume better after an interruption. The trade-off is more compaction jobs per partition, and in cluster mode proportionally more Raft manifest entries.

The upper bound exists because a single read_parquet() call spanning too many files can abort.

Compression

Compaction always writes its output with ZSTD, which is why compacted files are substantially smaller than the freshly-ingested files they replace. This is not configurable per tier.

The compression used for incoming writes is separate, and is set by ingest.compression (default snappy) — see the configuration overview.

Disable compaction

[compaction]
enabled = false

When to disable:

Testing ingestion performance
Very low write volume (<10 files/hour)
Debugging compaction issues

Monitoring

Check compaction status

curl http://localhost:8000/api/compaction/status \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "enabled": true,
  "running": false,
  "last_run": "2025-10-08T14:05:00Z",
  "next_run": "2025-10-08T15:05:00Z",
  "stats": {
    "total_jobs": 42,
    "successful_jobs": 40,
    "failed_jobs": 2,
    "total_files_compacted": 12580,
    "total_bytes_saved": 8589934592
  }
}

Get detailed statistics

curl http://localhost:8000/api/compaction/stats \
  -H "Authorization: Bearer $ARC_TOKEN"

List eligible partitions

curl http://localhost:8000/api/compaction/candidates \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "candidates": [
    {
      "partition": "default/cpu/2025/10/08/14",
      "file_count": 150,
      "total_size_mb": 7500,
      "age_hours": 2.5,
      "eligible": true
    },
    {
      "partition": "default/mem/2025/10/08/14",
      "file_count": 120,
      "total_size_mb": 6000,
      "age_hours": 2.5,
      "eligible": true
    }
  ],
  "total_candidates": 2
}

Manually trigger compaction

curl -X POST http://localhost:8000/api/compaction/trigger \
  -H "Authorization: Bearer $ARC_TOKEN"

View active jobs

curl http://localhost:8000/api/compaction/jobs \
  -H "Authorization: Bearer $ARC_TOKEN"

View job history

curl http://localhost:8000/api/compaction/history \
  -H "Authorization: Bearer $ARC_TOKEN"

Performance impact

Compaction performance

Test Environment: Apple M3 Max (14 cores, 36GB RAM)

Files	Size	Compaction Time	Final Size	Compression
888	1.2 GB	2.1s	239 MB	80.3%
906	1.2 GB	2.2s	242 MB	80.4%
910	1.2 GB	2.3s	243 MB	80.5%

Total: 2,704 files (3.7 GB) → 3 files (724 MB) in 6.6 seconds

Query performance

Before Compaction:

SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL 1 HOUR;
-- 5.2 seconds (scan 720 files)

After Compaction:

SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL 1 HOUR;
-- 0.05 seconds (scan 1 file)

Storage savings

Original files (Snappy):  3.7 GB
Compacted files (ZSTD):   724 MB
Space saved:              80.4%

Best practices

1. Let compaction run automatically

The default schedule (hourly) works well for most use cases:

[compaction]
enabled = true
hourly_schedule = "5 * * * *"

2. Monitor compaction jobs

Set up alerts for:

Failed compaction jobs
Partitions with >1000 files
Compaction taking >10 minutes

3. Adjust based on write volume

High volume (>10M records/sec):

[compaction]
hourly_min_files = 100   # Wait for more files
max_concurrent = 4       # More parallelism

Low volume (<100K records/sec):

[compaction]
hourly_min_files = 5              # Compact with fewer files
hourly_schedule = "0 */6 * * *"   # Every 6 hours

4. Tune files per batch

[compaction]
max_files_per_batch = 30     # Files per compaction job (default)
# max_files_per_batch = 60     # Fewer, larger outputs
# max_files_per_batch = 5      # Smaller outputs, more jobs per partition

5. Reduce file generation at source

Best practice: Increase buffer sizes to generate fewer files:

[ingest]
max_buffer_size = 200000        # Up from 50,000 (4x fewer files)
max_buffer_age_ms = 10000       # Up from 5000 (2x fewer files)

Impact:

Files generated: 2,000/hour → 250/hour (8x reduction)
Compaction time: substantially reduced
Memory usage: +300MB per worker

This is the most effective optimization - fewer files means faster compaction AND faster queries.

Troubleshooting

Compaction not running

Check status:

curl http://localhost:8000/api/compaction/status

Verify configuration:

# Check if enabled
grep "enabled" arc.toml

# Check schedule
grep "schedule" arc.toml

Check logs:

# Docker
docker logs arc | grep compaction

# Native
sudo journalctl -u arc | grep compaction

Partition skipped: No `time` column

Symptoms: A warning in the logs:

Skipping compaction: no 'time' column in any input file (data was not written by Arc ingest); leaving source files in place

Cause: Compaction requires a time column — it normalizes the column's type and sorts output by it. Arc's ingest path always writes one, so this only happens for Parquet files placed into the storage directory by external tools (custom loaders, bulk copies from other systems).

What happens: The partition is left untouched and the job counts as completed, not failed. The warning repeats each cycle as long as the partition stays above the compaction file-count threshold.

Solutions:

Rewrite the data through Arc ingest so it carries a proper time column, or
Rewrite the files in place with the timestamp column renamed/cast to time (type TIMESTAMP WITH TIME ZONE), or
Leave it as-is — the data stays queryable; it just won't be compacted.

Partitions where only some files lack time are not skipped: they compact normally, and rows from files without the column get NULL time values.

Compaction taking too long

Symptoms: Compaction jobs running for >30 minutes

Solutions:

Reduce files per batch:

[compaction]
max_files_per_batch = 10  # Smaller compaction jobs

Increase parallelism:
```
[compaction]
max_concurrent = 4
```
Reduce files at source:
```
[ingest]
max_buffer_size = 200000
```

Out of disk space during compaction

Symptoms: Compaction fails with disk space errors

Solutions:

Use temp directory on larger disk:
```
export TMPDIR=/mnt/large-disk/tmp
```
Reduce concurrent jobs:
```
[compaction]
max_concurrent = 1
```

Clean up old compacted files manually:

# Remove small files that were already compacted
find ./data -name "*.parquet" -size -10M -delete

Compaction locks not releasing

Symptoms: Partitions stuck in "locked" state

Check locks:

# View active locks
sqlite3 ./data/arc.db "SELECT * FROM compaction_locks;"

Clear stale locks:

# Locks expire automatically after 2 hours
# Or manually clear:
sqlite3 ./data/arc.db "DELETE FROM compaction_locks WHERE expires_at < datetime('now');"

API reference

GET /api/v1/compaction/status

Get current compaction status.

Response:

{
  "enabled": true,
  "running": false,
  "last_run": "2025-10-08T14:05:00Z",
  "next_run": "2025-10-08T15:05:00Z"
}

{
  "message": "Compaction triggered",
  "job_id": "comp_1696775400"
}

Faster queries
80% storage savings
99% fewer API calls
Automatic and safe

Default configuration works for most cases:

[compaction]
enabled = true
hourly_schedule = "5 * * * *"
hourly_min_age_hours = 1
hourly_min_files = 10

Monitor regularly:

Check /api/v1/compaction/status
Alert on failed jobs
Watch for partitions with >1000 files

Next steps

Monitor Compaction - Set up health checks
Configure WAL - Add durability guarantees
Configuration Reference - Tune settings for your workload

On this page