File Compaction

Arc's automatic compaction system merges small Parquet files into larger, optimized files for dramatically faster queries.

Overview

Compaction is Arc's file optimization system that merges small files into larger ones, substantially improving query performance.

Key Features:

• Automatic - Runs on schedule (default: hourly at :05)
• Safe - Locked partitions prevent concurrent compaction
• Efficient - Parallel, sorted merging by the query engine
• Non-blocking - Queries work during compaction
• Enabled by default - Essential for production

Why compaction matters

The small file problem

Arc's high-throughput ingestion creates many small files:

At high ingest rates with a 5-second flush interval:
→ 12 files per minute per measurement
→ 720 files per hour per measurement
→ 17,280 files per day per measurement

A backfill of history in random order through a write endpoint makes many more: each flush writes a small file into every hour it holds. Import endpoints group rows by hour before writing; see backfills.

Impact on Queries:

• Slow queries - The query engine must open/scan hundreds of files
• High costs - More S3/MinIO API calls
• Poor compression - Small files compress less efficiently
• Reduced pruning - Less effective partition elimination

After compaction

Real Production Test Results:

Before: 2,704 small files (Snappy) = 3.7 GB
After:  3 compacted files (ZSTD)   = 724 MB

Compression: 80.4% space savings
File reduction: 901x fewer files (2,704 → 3)
Compaction time: 5 seconds

Per-Measurement Breakdown:

• mem: 888 files → 1 file, 1,213 MB → 239 MB (80.3% compression)
• disk: 906 files → 1 file, 1,237 MB → 242 MB (80.4% compression)
• cpu: 910 files → 1 file, 1,246 MB → 243 MB (80.5% compression)

Query Performance:

• Far fewer file opens - Single file scan vs hundreds
• 99% fewer API calls - Massive cost reduction (2,704 → 3 LIST operations)
• 80.4% compression - ZSTD compaction vs Snappy writes
• Effective pruning - The query engine can skip entire files

How it works

Compaction flow

1. Scheduler wakes up (cron: "5 * * * *")
   ↓
2. Scan storage for eligible partitions
   ↓
3. For each partition:
   - Check age (>1 hour old?)
   - Check file count (≥10 files?)
   - Check if already compacted?
   ↓
4. Acquire partition lock (SQLite)
   ↓
5. Download small files to temp directory
   ↓
6. Compact via the query engine (parallel, sorted)
   ↓
7. Upload compacted file to storage
   ↓
8. Delete old small files
   ↓
9. Release lock & cleanup temp files
   ↓
10. Repeat for next partition

Partition structure

Data is organized by hour:

arc/                              # Bucket
├── default/                      # Database
│   └── cpu/                      # Measurement
│       └── 2025/10/08/           # Date
│           ├── 14/               # Hour (2 PM) - Eligible for compaction
│           │   ├── file1.parquet (50 MB)
│           │   ├── file2.parquet (48 MB)
│           │   └── ...
│           ├── 15/               # Hour (3 PM) - Eligible for compaction
│           └── 16/               # Hour (4 PM) - CURRENT, skip!

Compaction merges all files in a partition (e.g., 2025/10/08/14/) into one optimized file.

Configuration

Default configuration

Compaction is enabled by default in arc.toml:

[compaction]
enabled = true

# Hourly tier
hourly_enabled = true
hourly_schedule = "5 * * * *"   # Cron schedule: every hour at :05
hourly_min_age_hours = 1        # Wait 1 hour before compacting (let the hour complete)
hourly_min_files = 10           # Only compact if >=10 files exist

# Daily tier
daily_enabled = true
daily_schedule = "0 3 * * *"    # Cron schedule: 3 AM daily
daily_min_age_hours = 24        # Wait 24 hours
daily_min_files = 12            # Only compact if >=12 files exist

max_concurrent = 2              # Run 2 compactions in parallel

Configuration options

Schedule

[compaction]
hourly_schedule = "5 * * * *"     # Every hour at :05 (default)
daily_schedule = "0 3 * * *"      # 3 AM daily (default)
# hourly_schedule = "0 */2 * * *"   # Every 2 hours at :00

Cron format: minute hour day month weekday

Minimum age

[compaction]
hourly_min_age_hours = 1    # Don't compact the current hour (default)
daily_min_age_hours = 24    # Daily tier waits a full day (default)
# hourly_min_age_hours = 2    # Wait 2 hours (more conservative)
# hourly_min_age_hours = 0    # Compact immediately (aggressive)

Minimum files

[compaction]
hourly_min_files = 10    # Only compact if >=10 files (default)
daily_min_files = 12     # Daily tier threshold (default)
# hourly_min_files = 50    # Only compact with many files
# hourly_min_files = 5     # Compact more aggressively

Concurrent jobs

[compaction]
max_concurrent = 2    # Run 2 compactions in parallel (default)
# max_concurrent = 4    # More parallelism (uses more CPU/memory)
# max_concurrent = 1    # Sequential (lower resource usage)

This knob also divides the per-subprocess memory and thread budgets below, so raising it does not multiply Arc's total compaction resource use — it splits it further. That is deliberate: a raised max_concurrent gets you more jobs in flight, not more CPU and RAM.

Memory limit and threads (per subprocess)

Each compaction job runs in an isolated subprocess with its own query engine instance. These keys bound that instance's resources:

[compaction]
memory_limit = ""    # Per-subprocess engine memory limit; "" (default) = auto
threads = 0          # Per-subprocess engine threads; 0 (default) = auto
# memory_limit = "2GB"   # Explicit cap
# threads = 4            # Explicit thread count

Env vars: ARC_COMPACTION_MEMORY_LIMIT, ARC_COMPACTION_THREADS.

Auto behavior:

• memory_limit depends on whether database.memory_limit is set.

  • Set explicitly: it derives as database.memory_limit / max_concurrent, so all concurrent compaction jobs together stay within roughly one database.memory_limit. With database.memory_limit = "8GB" and the default concurrency of 2, each subprocess gets 4GB.
  • Unset (the default since v26.09.3): database.memory_limit is left to the query engine, which uses 80% of the container's cgroup limit — so there is no string to divide. Arc detects the limit itself for this one purpose and each subprocess gets detected x 0.8 / (max_concurrent + 1). The + 1 is because the subprocesses and the main Arc process are separate processes sharing one cgroup, not independent budgets.

  Worth knowing what that bounds: it caps the subprocesses' combined budget at roughly one share, but the main process still takes the engine's own 80% of the whole cgroup, so the worst-case total is around 133% of the container at the default concurrency. These are mostly spill thresholds rather than reservations, so that is tolerable — but if you are tight on memory, set database.memory_limit explicitly and size it with compaction in mind.

  One sharp edge at small sizes: below roughly 85 MB the engine stops spilling and raises an out-of-memory error instead. A 256 MiB container at the default max_concurrent = 2 derives about 68 MiB, which is inside that range. Compaction treats the failure as recoverable and halves its batch with a warning rather than crashing, so it is visible in the log — but such a container will make no compaction progress. Raise the container's memory or lower max_concurrent.

• threads defaults to the CPUs Arc may actually use divided by max(2, max_concurrent), minimum 1. "May actually use" means the container's CPU quota where there is one and the machine's core count where there is not — before v26.09.3 it was always the host's core count, so a 2-CPU pod on a 64-core node gave every job 32 threads (arc#1030). The derivation reads one number and does not distinguish a quota from a cpuset or an explicitly set GOMAXPROCS.

  The divisor tracks max_concurrent only from v27.01.1. Before that it was always 2, which hardcoded the default concurrency: raising max_concurrent gave you that many subprocesses each still claiming half the cores, so max_concurrent = 4 on 16 CPUs ran four jobs at 8 threads each (arc#1037). At the default max_concurrent = 2 the value is unchanged — the divisor is still 2 — so upgrading costs nothing unless you had raised concurrency, in which case each subprocess gets proportionally fewer threads. The floor of 2 means lowering max_concurrent to 1 does not give a single job all the cores; that headroom is left for ingest and queries.

  An explicit value is not rewritten by the automatic default. It is not exempt from the licence cap, though — see below.

• Licence cap (v27.01.1+, core-limited licences only). Whatever threads ends up as — automatic or explicit, since the automatic value is resolved first — is lowered to the smaller of the licensed core count and the CPUs Arc may use (arc#1036). The cap is per subprocess: it does not divide the licensed cores across the main process and all subprocesses, so the aggregate can still exceed the licence. A licence with no core limit changes nothing.

Accepted memory_limit forms are absolute sizes with a unit: "8GB", "512MB", "0.5GB". Percent and unit-less forms are rejected at startup (DuckDB's SET memory_limit does not support them), as are other invalid values. The effective values appear in the startup log (subprocess_memory_limit, subprocess_threads).

Thread count and memory budget interact, and the default trades one for the other. Measured under a 2-CPU quota on 351 files: with the ~4.47 GB per-subprocess budget a container with no memory limit derives, 32 threads finishes faster than 1 (9.8 s against 13.0 s) — but at 1 GB and at 512 MB the 32-thread run fails with Out of Memory Error while 1 thread completes, because sort and scan buffers scale with the thread count. A 2-CPU / 2 Gi pod on a 64-core node derives about 546 MB per subprocess, which is in the failing range. If your container limits CPU but not memory, set threads explicitly — the quota-derived default is deliberately conservative because the alternative fails rather than merely slows.

When a job exceeds its memory limit, DuckDB spills to a duckdb-spill/ directory inside the job's temp directory (under compaction.temp_directory) — size that volume for your largest partitions. Spill files are removed by normal job cleanup and by the crash sweeps on startup.

Temporary directory

compaction.temp_directory (default ./data/compaction) is the scratch space for one job: whatever DuckDB spills, plus the compacted output before it is published. Two things about where you point it, and both of them matter on a local backend.

Put it on the same filesystem as storage.local_path, and the output is moved rather than copied. From v27.01.1 Arc renames the finished file into the storage root instead of streaming it through a copy, so publishing an output costs a rename rather than a full read plus a full write of it (arc#969). The defaults (./data/arc and ./data/compaction) are siblings, so this applies without any configuration. On a different filesystem the rename is impossible and Arc copies exactly as it did before, logging one warning per job that names both paths — if you see that line and want the saving, co-locate the two paths.

On S3 and Azure nothing changes: publishing an output is a network transfer either way.

Do not put it inside storage.local_path. Everything under the storage root is treated as storage, so the job directory is listed as a database by SHOW DATABASES, and a backup of the instance sweeps it — a half-written compaction output can end up in a backup. A sibling directory on the same filesystem gives you the move without this. Ordinary queries are unaffected either way, because they glob <database>/<measurement>/**/*.parquet and a job directory at another top level does not match.

Sizing. Whatever DuckDB spills, plus the whole output. Inputs are not staged here on a local backend — from v27.01.1 compaction reads them from their own paths rather than copying them in first. On a different filesystem from the storage root, the storage root additionally needs room for a second copy of the output while it is being published, and a volume too tight for that does not fail cleanly: the adaptive splitter halves the batch and reports success, leaving several partly compacted files where one belonged.

Lower these when compaction competes with ingest for RAM during backfill catch-up (many partitions become candidates at once); raise them to make individual large compactions faster on dedicated compactor nodes.

Files per batch

A partition with more files than this is split into several batches, each compacted as an independent job producing its own output file.

[compaction]
max_files_per_batch = 30   # Files per compaction job (default)
# max_files_per_batch = 5    # Smaller outputs, more jobs per partition
# max_files_per_batch = 60   # Fewer, larger outputs

Valid range is 2–500. Values outside it fall back to the default with a startup warning; 1 is rejected because compaction's adaptive retry cannot process a single-file batch.

This bounds the file count per job, not the output size in bytes — compacted file size tracks input file size, which follows your ingest buffer settings. The main reason to lower it is transferring compacted files over a constrained or intermittent link (edge deployments), where smaller, independently-transferable files resume better after an interruption. The trade-off is more compaction jobs per partition, and in cluster mode proportionally more Raft manifest entries.

The upper bound exists because a single read_parquet() call spanning too many files can abort.

Cycle timeout

One scheduled or manual compaction cycle runs under a wall-clock budget (v26.09.2+). A cycle that reaches it stops launching work, waits for the active jobs, and records the batches it completed, failed, interrupted and never started, so a long backfill is not reported as job failures.

[compaction]
cycle_timeout = "30m"    # Positive Go duration: 90s, 30m, 2h

A longer budget does not reduce peak memory demand or guarantee that every candidate completes; it only lets one cycle process more of them. A scheduled tick that arrives while a cycle is still running is skipped, and a manual trigger during a running cycle returns 409.

Exclude databases

Scheduled cycles — and manual triggers that don't name a database — skip the databases in this list during candidate discovery (v26.09.2+):

[compaction]
exclude_databases = ["staging", "imports_backlog"]

Names match exactly and case-sensitively: no globs, no prefixes, and no splitting on separators, so excluding wh never touches wh-other. A trigger that names a database bypasses the list (see Manually trigger compaction) — exclude what should wait, then trigger scoped cycles in the order you want.

On an edge-sync hub, received spoke data is excluded either as one pseudo-database (spoke1/telemetry) or as the whole namespace (spoke1); excluding the namespace also skips listing its children. While excluded, a received namespace accumulates raw files until it is un-excluded or triggered manually — at spoke granularity, since database= takes plain database names.

The list gates compaction only: manifest recovery still completes interrupted compactions (exclusion stops new work, never the completion of started work), and retention, tiering, and Iceberg export ignore it. The candidates endpoint applies the same filter, /api/v1/compaction/stats reports the active list, and Arc warns at startup about entries whose shape can never match a discovered database.

The environment override is whitespace-separated (ARC_COMPACTION_EXCLUDE_DATABASES="staging imports_backlog"); a name containing spaces needs the TOML array.

Compression

Compaction always writes its output with ZSTD, which is why compacted files are substantially smaller than the freshly-ingested files they replace. This is not configurable per tier.

The compression used for incoming writes is separate, and is set by ingest.compression (default snappy) — see the configuration overview.

Disable compaction

[compaction]
enabled = false

When to disable:

• Testing ingestion performance
• Very low write volume (<10 files/hour)
• Debugging compaction issues

Monitoring

Check compaction status

curl http://localhost:8000/api/v1/compaction/status \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "manager": {
    "active_jobs": 2,
    "total_completed": 1894,
    "total_failed": 3
  },
  "schedulers": {
    "hourly": {
      "enabled": true,
      "running": true,
      "schedule": "5 * * * *",
      "next_run": "2027-01-14T15:05:00Z"
    },
    "daily": {
      "enabled": false,
      "running": false,
      "schedule": "0 3 * * *"
    }
  }
}

The three manager counters are attempts, not batches or partitions. next_run appears only while that scheduler is running; in cluster mode each scheduler also carries role_gated and gate_role, which is how a node that is not compacting because of its role is told apart from one whose schedule has not fired.

Get detailed statistics

curl http://localhost:8000/api/v1/compaction/stats \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "total_jobs_completed": 1894,
  "total_jobs_failed": 3,
  "total_jobs_interrupted": 0,
  "total_files_compacted": 12580,
  "total_bytes_saved": 8589934592,
  "total_bytes_saved_mb": 8192,
  "total_manifests_recover": 0,
  "active_jobs": 2,
  "cycle_running": true,
  "current_cycle_id": 42,
  "paused": false,
  "exclude_databases": [],
  "last_cycle": {
    "cycle_id": 41,
    "status": "completed",
    "discovered_batches": 86,
    "started_batches": 86,
    "succeeded_batches": 84,
    "failed_batches": 2,
    "interrupted_batches": 0,
    "unstarted_batches": 0,
    "discovery_errors": 0
  },
  "recent_jobs": []
}

last_cycle is a single global record — the most recently finished cycle on this node. It is not a substitute for /cycles/{id}: if two cycles ran, the one you triggered may not be the one described here. Resolve the cycle_id the trigger returned instead.

List eligible partitions

curl http://localhost:8000/api/v1/compaction/candidates \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{
  "count": 2,
  "candidates": [
    {
      "database": "default",
      "measurement": "cpu",
      "partition_path": "default/cpu/2027/01/14/14",
      "file_count": 150,
      "tier": "hourly"
    },
    {
      "database": "default",
      "measurement": "mem",
      "partition_path": "default/mem/2027/01/14/14",
      "file_count": 120,
      "tier": "hourly"
    }
  ]
}

Only partitions that already pass the age and min_files thresholds are listed, so every entry is eligible by construction — there is no per-entry eligibility flag, and no size or age field.

Manually trigger compaction

curl -X POST http://localhost:8000/api/v1/compaction/trigger \
  -H "Authorization: Bearer $ARC_TOKEN"

The manual cycle uses the configured cycle_timeout. Narrow it with query parameters: tier (comma-separated, defaults to every enabled tier), database, and measurement (v26.09.2+, requires database). The database and measurement filters also scope manifest recovery, so a targeted run touches nothing else. A trigger that names a database also bypasses exclude_databases (v26.09.2+); an unscoped trigger honors the list and echoes it in the response.

curl -X POST "http://localhost:8000/api/v1/compaction/trigger?database=prod&measurement=cpu&tier=hourly" \
  -H "Authorization: Bearer $ARC_TOKEN"

View work in flight

curl http://localhost:8000/api/v1/compaction/jobs \
  -H "Authorization: Bearer $ARC_TOKEN"

{ "active_jobs": 2, "unit": "attempts", "jobs": [] }

active_jobs counts attempts, the same unit as the history records and as total_completed on /status. One batch can account for several attempts in sequence, because a batch too large to compact in one pass is split and retried at smaller sizes. The jobs array is always empty: Arc tracks how many attempts are running, not which ones.

Resolve a cycle you triggered

A trigger hands back a cycle_id. Ask what that cycle did:

curl http://localhost:8000/api/v1/compaction/cycles/42 \
  -H "Authorization: Bearer $ARC_TOKEN"

{
  "cycle_id": 42,
  "status": "completed",
  "source": "api",
  "databases": ["prod"],
  "measurement": "cpu",
  "tiers": ["hourly"],
  "started_at": "2027-01-14T14:05:00Z",
  "finished_at": "2027-01-14T14:07:12Z",
  "duration_seconds": 132,
  "discovered_batches": 86,
  "started_batches": 86,
  "succeeded_batches": 84,
  "failed_batches": 2,
  "interrupted_batches": 0,
  "unstarted_batches": 0,
  "discovery_errors": 0,
  "failed_partitions": {
    "prod/cpu/2027/01/13/22": 1,
    "prod/cpu/2027/01/13/23": 1
  },
  "failed_partition_count": 2
}

failed_partitions is the retry list, not a sample. It is keyed by partition path and deduplicated, so one entry covers a partition however many batches it was split into, and sum(failed_partitions) equals failed_batches unless failed_partitions_truncated is present. That flag means more distinct partitions failed than the cycle record retains (200), so the list is incomplete — never read a capped list as the full answer.

status is running, completed, failed, or interrupted. A cycle that has been claimed but has not yet written its record answers claimed, which is what you get if you resolve an id in the same millisecond you triggered it.

source says who started it: api for a manual trigger, scheduler for a cron tick, unspecified otherwise.

A 404 means this node retains no record of that id — and that is not the same as "the cycle did nothing". Cycle ids restart at 1 when the process restarts, and cycles run on whichever node holds the compactor lease, so the response names the retained window (oldest_retained, newest_retained) to tell a trimmed id apart from one that was never here.

List recent cycles

curl "http://localhost:8000/api/v1/compaction/cycles?limit=20" \
  -H "Authorization: Bearer $ARC_TOKEN"

{
  "cycles": [ { "cycle_id": 42, "status": "completed" } ],
  "retained": 137,
  "oldest_retained": 1,
  "newest_retained": 42,
  "running_cycle_id": 42
}

limit defaults to 10 and is capped at the 500 retained cycles.

What 500 cycles buys you in time depends entirely on your schedule, and it is worth working out before you rely on this history. The record is in memory only — it does not survive a restart — and a cycle is consumed whether or not it finds anything to compact. At the default schedules (hourly plus daily) that is 25 ids a day, so 500 cycles is roughly 20 days. At hourly_schedule = "*/5 * * * *" it is 289 a day, so the same 500 cycles is about 41 hours — an idle node burns through the window just as fast as a busy one. If you need a cycle to still be resolvable tomorrow, check that arithmetic against your own cron. running_cycle_id is present only while a cycle is running. The list omits the failed_partitions map — 500 cycles of up to 200 partitions each would be megabytes in one response — but keeps failed_partition_count and the truncation flag, so you can spot the cycle you care about and fetch its full map from /cycles/{id}.

View job history

curl "http://localhost:8000/api/v1/compaction/history?limit=50" \
  -H "Authorization: Bearer $ARC_TOKEN"

Narrow it to one cycle:

curl "http://localhost:8000/api/v1/compaction/history?cycle_id=42" \
  -H "Authorization: Bearer $ARC_TOKEN"

{
  "total_jobs": 1894,
  "recent_jobs": [
    {
      "database": "prod",
      "measurement": "cpu",
      "partition_path": "prod/cpu/2027/01/13/22",
      "success": false,
      "cycle_id": 42,
      "attempt_depth": 0
    }
  ],
  "matched": 4,
  "page_size": 4,
  "retained": 100,
  "capacity": 100,
  "truncated": false,
  "unit": "attempts",
  "cycle_id": 42,
  "cycle_status": "completed",
  "cycle_failed_batches": 2,
  "cycle_started_batches": 86
}

limit defaults to 10 and is capped at the 100 retained records. Before v27.01.1 it was ignored and you always got 10.

Each record is one attempt. attempt_depth is how deep in the adaptive splitter the attempt sat: depth 0 is the batch as dispatched, and deeper attempts are the halves it was split into. So a single failing batch can leave several records with the same partition_path.

Do not read depth-0 failures as the batch failures. A batch that fails at depth 0 and is then rescued by splitting counts as succeeded and contributes no failed partition, yet still leaves a depth-0 record with success: false. The cycle record is the ground truth, which is why the cycle_* fields are echoed here when you filter by cycle_id.

truncated is true when the ring cannot prove it is showing you everything that matched. may_have_evicted_earlier_attempts is the hedged half of it: the ring is full and the filtered cycle's earliest attempts may already have been overwritten. It is present only when that is possible — attempts of one cycle are contiguous in the ring, so a retained cycle older than the one you asked about proves nothing of yours was evicted, and the flag is then absent. cycle_retained: false means the cycle itself has aged out of the far longer cycle history, so an empty page says nothing about what failed.

Performance impact

Compaction performance

Test Environment: Apple M3 Max (14 cores, 36GB RAM)

Total: 2,704 files (3.7 GB) → 3 files (724 MB) in 6.6 seconds

Query performance

Before Compaction:

SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL 1 HOUR;
-- 5.2 seconds (scan 720 files)

After Compaction:

SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL 1 HOUR;
-- 0.05 seconds (scan 1 file)

Storage savings

Original files (Snappy):  3.7 GB
Compacted files (ZSTD):   724 MB
Space saved:              80.4%

Best practices

1. Let compaction run automatically

The default schedule (hourly) works well for most use cases:

[compaction]
enabled = true
hourly_schedule = "5 * * * *"

2. Monitor compaction jobs

Set up alerts for:

• Failed compaction jobs
• Partitions with >1000 files
• Compaction taking >10 minutes

3. Adjust based on write volume

High volume (>10M records/sec):

[compaction]
hourly_min_files = 100   # Wait for more files
max_concurrent = 4       # More parallelism
threads = 8              # Optional: hold the per-job thread count (see below)

From v27.01.1 the automatic threads value divides by max_concurrent, so raising concurrency alone gives you more jobs in flight at fewer threads each rather than more total compaction CPU. Set threads explicitly if you want both.

Low volume (<100K records/sec):

[compaction]
hourly_min_files = 5              # Compact with fewer files
hourly_schedule = "0 */6 * * *"   # Every 6 hours

4. Tune files per batch

[compaction]
max_files_per_batch = 30     # Files per compaction job (default)
# max_files_per_batch = 60     # Fewer, larger outputs
# max_files_per_batch = 5      # Smaller outputs, more jobs per partition

5. Reduce file generation at source

Best practice: Increase buffer sizes to generate fewer files:

[ingest]
max_buffer_size = 200000        # Up from 50,000 (4x fewer files)
max_buffer_age_ms = 10000       # Up from 5000 (2x fewer files)

Impact:

• Files generated: 2,000/hour → 250/hour (8x reduction)
• Compaction time: substantially reduced
• Memory usage: +300MB per worker

This is the most effective optimization - fewer files means faster compaction AND faster queries.

Troubleshooting

Compaction not running

Check status:

curl http://localhost:8000/api/v1/compaction/status

Verify configuration:

# Check if enabled
grep "enabled" arc.toml

# Check schedule
grep "schedule" arc.toml

Check logs:

# Docker
docker logs arc | grep compaction

# Native
sudo journalctl -u arc | grep compaction

Partition skipped: No time column

Symptoms: A warning in the logs:

Skipping compaction: no 'time' column in any input file (data was not written by Arc ingest); leaving source files in place

Cause: Compaction requires a time column — it normalizes the column's type and sorts output by it. Arc's ingest path always writes one, so this only happens for Parquet files placed into the storage directory by external tools (custom loaders, bulk copies from other systems).

What happens: The partition is left untouched and the job counts as completed, not failed. The warning repeats each cycle as long as the partition stays above the compaction file-count threshold.

Solutions:

1. Rewrite the data through Arc ingest so it carries a proper time column, or
2. Rewrite the files in place with the timestamp column renamed/cast to time (type TIMESTAMP WITH TIME ZONE), or
3. Leave it as-is — the data stays queryable; it just won't be compacted.

Partitions where only some files lack time are not skipped: they compact normally, and rows from files without the column get NULL time values.

Compaction taking too long

Symptoms: Compaction jobs running for >30 minutes

Solutions:

1. Reduce files per batch:

   [compaction]
   max_files_per_batch = 10  # Smaller compaction jobs

2. Increase parallelism:

   [compaction]
   max_concurrent = 4

   On v27.01.1+ this splits the automatic per-subprocess thread and memory budgets four ways instead of two, so it raises how many partitions progress at once without raising total CPU. If compaction is CPU-bound rather than queued behind concurrency, set threads explicitly as well, and confirm the startup log shows the values you expect (subprocess_threads, subprocess_memory_limit).

3. Reduce files at source:

   [ingest]
   max_buffer_size = 200000

Out of disk space during compaction

Symptoms: Compaction fails with disk space errors

Solutions:

1. Use temp directory on larger disk:

   export TMPDIR=/mnt/large-disk/tmp

2. Reduce concurrent jobs:

   [compaction]
   max_concurrent = 1

3. Clean up old compacted files manually:

   # Remove small files that were already compacted
   find ./data -name "*.parquet" -size -10M -delete

Compaction locks not releasing

Symptoms: Partitions stuck in "locked" state

Check locks:

# View active locks
sqlite3 ./data/arc.db "SELECT * FROM compaction_locks;"

Clear stale locks:

# Locks expire automatically after 2 hours
# Or manually clear:
sqlite3 ./data/arc.db "DELETE FROM compaction_locks WHERE expires_at < datetime('now');"

API reference

GET /api/v1/compaction/status

Get current compaction status. The manager block counts attempts — one compaction subprocess invocation — not batches or partitions.

Response:

{
  "manager": {
    "active_jobs": 2,
    "total_completed": 1894,
    "total_failed": 3
  },
  "schedulers": {
    "hourly": {
      "enabled": true,
      "running": true,
      "schedule": "5 * * * *",
      "next_run": "2027-01-14T15:05:00Z"
    },
    "daily": {
      "enabled": false,
      "running": false,
      "schedule": "0 3 * * *"
    }
  }
}

next_run appears only while a scheduler is running. In cluster mode each scheduler also carries role_gated and gate_role, which is how you see that a node is not compacting because of its role rather than its schedule.

active_jobs is the number of attempts in flight. It reports null before v27.01.1, because nothing produced the value.

GET /api/v1/compaction/stats

Get detailed compaction statistics.

GET /api/v1/compaction/candidates

List partitions eligible for compaction.

POST /api/v1/compaction/trigger

Manually trigger a compaction cycle. Query parameters: tier (comma-separated, defaults to every enabled tier), database, and measurement (v26.09.2+, requires database).

The returned cycle_id is the id the cycle actually claimed, so you can resolve it on /cycles/{id}. Before v27.01.1 it was computed as "current id plus one" before the claim existed, and under concurrent triggers it could name another caller's cycle.

Returns 409 while a cycle is already running, with the running cycle_id and is_running in the body, and 409 when compaction is paused cluster-wide. In cluster mode a node that does not hold the compaction lease returns 503 with role, can_compact, and lease_holder, naming the node to send the trigger to.

Response:

{
  "message": "Compaction triggered",
  "status": "running",
  "tiers": ["hourly"],
  "cycle_id": 42,
  "database": "prod",
  "measurement": "cpu"
}

GET /api/v1/compaction/jobs

Attempts currently in flight. Returns active_jobs, unit: "attempts", and an always-empty jobs array — only the count is tracked, not the identity of each attempt. Reports 0 unconditionally before v27.01.1.

GET /api/v1/compaction/history

Compaction job history, one record per attempt. Query parameters: limit (default 10, capped at the 100 retained records; ignored before v27.01.1) and cycle_id, which filters to one cycle and echoes that cycle's authoritative counters alongside the page. Records carry cycle_id and attempt_depth.

GET /api/v1/compaction/cycles

The retained cycle history, newest first. Query parameter: limit (default 10, capped at the 500 retained cycles). Returns retained, the oldest_retained/newest_retained window, and running_cycle_id while a cycle is running. The failed_partitions map is omitted here; fetch it per cycle.

GET /api/v1/compaction/cycles/{id}

One cycle by the cycle_id a trigger returned: status, source, scope, timings, per-batch counters, and failed_partitions — a deduplicated map of partition path to failed-batch count, which is the list of partitions to retry.

A 404 names the retained window rather than claiming the cycle never ran: ids restart at 1 on process restart, and cycles run on the node holding the compactor lease.

Summary

Compaction is essential for production deployments:

Benefits:

• Faster queries
• 80% storage savings
• 99% fewer API calls
• Automatic and safe

Default configuration works for most cases:

[compaction]
enabled = true
hourly_schedule = "5 * * * *"
hourly_min_age_hours = 1
hourly_min_files = 10

Monitor regularly:

• Check /api/v1/compaction/status
• Alert on failed jobs
• Watch for partitions with >1000 files

Next steps

• Monitor Compaction - Set up health checks
• Configure WAL - Add durability guarantees
• Configuration Reference - Tune settings for your workload