Configuration Overview

Every arc.toml section for Arc Enterprise, including the license key, cluster, tiered_storage, audit_log, and governance blocks, plus the environment variables that override them.

Arc uses a TOML configuration file (arc.toml) with environment variable overrides for flexibility.

Enterprise license

Configure your Arc Enterprise license key to enable enterprise features:

[license]
key = "ARC-XXXX-XXXX-XXXX-XXXX"

Environment variable:

ARC_LICENSE_KEY="ARC-XXXX-XXXX-XXXX-XXXX"

On startup, Arc validates your license and enables the features included in your plan. See Arc Enterprise Overview for the full feature list and Licensing for how to get a key, core and machine limits, and expiry behavior.

As of 26.09.1, a transient license-server failure no longer crash-loops Enterprise pods: startup retries briefly and then falls back to the last signature-verified license cached on disk, honored until that license's own expiry. A definitive server rejection (revoked, expired, unknown key) still disables enterprise features immediately.

Air-gapped: Offline license file (26.09.1+)

For environments with no route to enterprise.basekick.net, download an offline license file from the activation server admin (an explicit site license: unbound, valid on any machine until expiry, audit-logged at mint) and point Arc at it:

[license]
file_path = "/etc/arc/license.json"   # or ARC_LICENSE_FILE_PATH

The file is verified from disk against Arc's pinned public key — no network calls of any kind. file_path wins over key; a rejected file means OSS mode (never a silent fallback to online licensing). Keep the file readable only by the Arc user (0600).

Scheduled edge sync (planned for 27.01.1)

Network spokes configured with edge_sync.spoke.enabled = true also start a scheduler when a valid paid license is present. Every paid tier qualifies. edge_sync.spoke.sync_interval defaults to 5m, and edge_sync.spoke.sync_retry_interval defaults to 30s. Both must be at least one second; retry must be shorter than the normal interval. The corresponding environment variables are ARC_EDGE_SYNC_SPOKE_SYNC_INTERVAL and ARC_EDGE_SYNC_SPOKE_SYNC_RETRY_INTERVAL.

See the edge-sync guide for hub URL, identities, environment-only credentials, retry backoff, and metrics. Manual network sync and bundle export remain available in OSS.

Configuration files

Primary: arc.toml

The main configuration file with production-ready defaults:

# Server Configuration
[server]
port = 8000

# Logging
[log]
level = "info"       # debug, info, warn, error
format = "console"   # json or console

# Database (query engine)
[database]
# Auto-detected if not set (recommended)
# max_connections = 28    # 2x CPU cores
# memory_limit = "8GB"    # unset: engine uses 80% of the cgroup limit
# thread_count = 14       # unset: engine uses the CPU quota
enable_wal = false

# Storage Backend
[storage]
backend = "local"           # local, s3, minio, azure, azblob
local_path = "./data/arc"

# Ingestion
[ingest]
max_buffer_size = 50000     # records before flush
max_buffer_age_ms = 5000    # ms before force flush
# flush_workers = 16        # async flush workers (auto-detected)
# flush_queue_size = 64     # pending flush queue (auto-detected)
# shard_count = 32          # buffer shards

# Compaction
[compaction]
enabled = true
hourly_enabled = true
hourly_min_age_hours = 0
hourly_min_files = 5

# Authentication
[auth]
enabled = true

# Delete Operations
[delete]
enabled = true
confirmation_threshold = 10000
max_rows_per_delete = 1000000

# Retention Policies
[retention]
enabled = true

# Continuous Queries
[continuous_query]
enabled = true

Environment variables

Override any setting via environment variables with the ARC_ prefix:

# Server
ARC_SERVER_PORT=8000
ARC_SERVER_TLS_ENABLED=false
ARC_SERVER_TLS_CERT_FILE=/path/to/cert.pem
ARC_SERVER_TLS_KEY_FILE=/path/to/key.pem
ARC_SERVER_MAX_PAYLOAD_SIZE=1GB

# Logging
ARC_LOG_LEVEL=info
ARC_LOG_FORMAT=json

# Database
ARC_DATABASE_MAX_CONNECTIONS=28
ARC_DATABASE_MEMORY_LIMIT=8GB
ARC_DATABASE_THREAD_COUNT=14

# Features
ARC_AUTH_ENABLED=true
ARC_COMPACTION_ENABLED=true
ARC_DELETE_ENABLED=true
ARC_RETENTION_ENABLED=true
ARC_CONTINUOUS_QUERY_ENABLED=true

# Ingestion Concurrency
ARC_INGEST_FLUSH_WORKERS=32
ARC_INGEST_FLUSH_QUEUE_SIZE=200
ARC_INGEST_SHARD_COUNT=64

Configuration priority

Settings are applied in this order (highest to lowest):

Environment variables (e.g., ARC_SERVER_PORT=8000)
arc.toml file
Built-in defaults

An environment variable set to the empty string counts as unset, not as an override: ARC_TIERED_STORAGE_COLD_S3_ENDPOINT="" does not clear an s3_endpoint the file sets. To blank a key the file sets, remove it from the file or give the variable the value you want.

Storage backends

Local Filesystem - Default, simplest option for single-node deployments.

[storage]
backend = "local"
local_path = "./data/arc"

Environment variables:

ARC_STORAGE_BACKEND=local
ARC_STORAGE_LOCAL_PATH=./data/arc

Key configuration areas

Server

Basic HTTP server settings:

[server]
port = 8000    # HTTP/HTTPS port to listen on

TLS/SSL (HTTPS)

Arc supports native HTTPS/TLS without requiring a reverse proxy:

[server]
port = 443
tls_enabled = true
tls_cert_file = "/etc/letsencrypt/live/example.com/fullchain.pem"
tls_key_file = "/etc/letsencrypt/live/example.com/privkey.pem"

Environment variables:

ARC_SERVER_TLS_ENABLED=true
ARC_SERVER_TLS_CERT_FILE=/path/to/cert.pem
ARC_SERVER_TLS_KEY_FILE=/path/to/key.pem

When to Use Native TLS

Native packages (deb/rpm): Use native TLS for simple deployments
Docker/Kubernetes: Use a reverse proxy (Traefik, nginx, Ingress) for TLS termination
Development: Use self-signed certificates for local HTTPS testing

When TLS is enabled, Arc automatically:

Adds the Strict-Transport-Security (HSTS) header
Validates certificate and key files on startup

Max payload size

Configure the maximum request payload size for write endpoints (msgpack, line protocol):

[server]
# Maximum payload size (applies to both compressed and decompressed)
# Supports units: B, KB, MB, GB
# Default: 1GB
max_payload_size = "1GB"

Environment variable:

ARC_SERVER_MAX_PAYLOAD_SIZE=2GB

Large Bulk Imports

If you're importing large datasets and encounter 413 errors, you can:

Increase max_payload_size (e.g., "2GB")
Batch your imports into smaller chunks (recommended for reliability)

Database (query engine)

Query engine connection pool and resource settings:

[database]
# AUTO-DETECTION: If not set:
#   - max_connections: 2x CPU cores (min 4, max 64), set by Arc
#   - memory_limit:    left to the query engine, which uses 80% of the
#                      container's cgroup limit (or of host memory when
#                      unlimited). Percent and unit-less forms are rejected.
#   - thread_count:    left to the query engine, which uses the container's
#                      CPU quota rather than the host's core count.

# Manual override examples:
max_connections = 28      # Connection pool size
memory_limit = "8GB"      # Query engine memory limit
thread_count = 14         # Query execution threads
enable_wal = false        # Query engine WAL (not Arc WAL)

Behaviour change in v26.09.3

memory_limit and thread_count are unset by default now, and that is deliberate: the query engine performs its own cgroup-aware detection and uses 80% of the memory limit it finds — your container's limit if you set one, host memory if you did not. It reads the CPU quota for threads the same way.

Before v26.09.3 Arc overrode both from the CPU count: min(cores, 32) GB for memory and the core count for threads. runtime.NumCPU() reflects CPU affinity but not a CFS quota, and Kubernetes limits.cpu and docker --cpus are quotas — so a pod limited to 2 CPUs on a 64-core node saw 64 cores and was given a 32 GB memory limit inside a 2 Gi container. The engine then allocated against that budget rather than spilling to temp_directory, and the kernel OOM-killed the container.

Measured after the change: --memory=512m yields 409.5 MiB, --memory=2g yields 1.5 GiB, and --cpus=2 yields 2 threads on an 8-core host.

Two things to check when you upgrade.

If you run with a container memory limit, the engine's limit now follows it. That is the fix, and in the example above it drops from 32 GB to about 1.6 GiB.

If you run with no memory limit — including the OSS Helm chart, which ships resources: {} — the engine sees the node's total memory and takes 80% of it. On a 16-core/64 GiB node that is a rise, from Arc's old 16 GB to roughly 51 GiB. Set container limits, or set memory_limit explicitly, if you were relying on the old accidental cap.

A smaller limit also makes spilling to temp_directory more likely. Spilling is slow but survivable where an OOM kill is not, so size that directory (or its volume) for the queries you run.

Ingestion

Buffer and concurrency settings for write performance:

[ingest]
# Maximum records to buffer before flushing to Parquet
max_buffer_size = 50000

# Maximum age (ms) before forcing a flush
max_buffer_age_ms = 5000

# Concurrency settings (auto-detected if not set)
# flush_workers = 16       # async flush workers (2x CPU cores, min 8, max 64)
# flush_queue_size = 64    # pending flush queue (4x workers, min 100)
# shard_count = 32         # buffer shards for lock distribution

Data flushes when either condition is met:

Buffer reaches max_buffer_size records
Buffer age exceeds max_buffer_age_ms

Behaviour change in v26.09.3

When every flush worker is busy and the flush queue is full, Arc keeps the batch in its in-memory buffer and flushes it as soon as a worker frees a slot. Before v26.09.3 that batch was discarded and the write still returned success — with wal.enabled = false, the default, the records were lost.

Two things follow for tuning. Deferred records are held in memory and Arc does not bound that, so a node with many active measurements and slow storage grows its resident set rather than shedding writes — watch arc_buffer_deferred_buffers and the sum of arc_buffer_records_buffered, arc_buffer_records_queued and arc_buffer_records_inflight (the last two from v27.01.1). Lowering either setting below makes flushes smaller and more frequent, which is the lever if that becomes a problem. And max_buffer_size is a per-measurement trigger, so the worst-case memory is roughly active_measurements x max_buffer_size x bytes_per_record; with max_buffer_age_ms at its default the age trigger normally fires long before that, which is what keeps it small in practice.

High Concurrency

For deployments with many concurrent clients (50+), increase flush_workers and flush_queue_size:

[ingest]
flush_workers = 32
flush_queue_size = 200
shard_count = 64

Compaction

Automatic file optimization:

[compaction]
enabled = true
hourly_enabled = true
hourly_min_age_hours = 0    # Files must be this old
hourly_min_files = 10       # Minimum files to trigger
daily_enabled = true        # Daily tier
daily_min_age_hours = 24
daily_min_files = 12
cycle_timeout = "30m"       # Budget for one cycle (v26.09.2+); positive Go duration

Authentication

Token-based API authentication:

[auth]
enabled = true              # Enable/disable auth
db_path = "./data/arc_auth.db"  # Token database
cache_ttl = 30              # Token cache TTL (seconds)
max_cache_size = 1000       # Max cached tokens
bootstrap_token = ""        # Pre-set admin token value (v26.04.1+)
force_bootstrap = false     # Add a recovery token without removing existing ones (v26.04.1+)

Available since v26.04.1

bootstrap_token — Set a known admin token at deploy time via ARC_AUTH_BOOTSTRAP_TOKEN instead of catching a randomly generated one from startup logs. On first run, Arc uses this value as the initial admin token. On subsequent restarts, it is a no-op.

force_bootstrap — Recovery path when the admin token is lost. Set ARC_AUTH_FORCE_BOOTSTRAP=true alongside ARC_AUTH_BOOTSTRAP_TOKEN to add a new arc-recovery admin token without removing existing tokens. Remove this flag after recovery. See the Authentication configuration guide for full details.

Delete operations

Safe deletion with confirmation:

[delete]
enabled = true
confirmation_threshold = 10000   # Require confirmation above this
max_rows_per_delete = 1000000    # Hard limit per operation

Query

Query execution limits and the query path's schema behavior. The schema anchor settings are explained in the stable field schema guide.

[query]
timeout = 300                 # Query execution timeout in seconds (0 = no timeout)
slow_query_threshold_ms = 0   # Log queries slower than this (0 = off)

# Range-independent field binding through schema anchors (v26.09.2+)
stable_schema = true
stable_schema_bootstrap = true            # build anchors for pre-26.09.2 measurements on first query
stable_schema_bootstrap_max_files = 500   # max files a bootstrap reads; larger measurements are sampled to this many
empty_range_anchor_scan = false           # EXPERIMENTAL: answer a proven-empty range from the anchor alone
# On a cluster whose nodes keep separate storage and receive each other's
# files by replication, leave empty_range_anchor_scan off (see the guide).

# File-level time pruning of the live hour (v26.09.2+, experimental, local backend)
file_time_pruning = false
file_time_pruning_margin_seconds = 300    # writer clock-skew allowance

Retention policies

Automatic data expiration:

[retention]
enabled = true
db_path = "./data/arc_retention.db"

Continuous queries

Scheduled automated queries:

[continuous_query]
enabled = true
db_path = "./data/arc_cq.db"

Write-Ahead Log (WAL)

Optional durability guarantee:

[wal]
enabled = false              # Enable for zero data loss
directory = "./data/wal"
sync_mode = "fdatasync"      # none, fdatasync, fsync
max_size_mb = 500
max_age_seconds = 3600

Metrics

Timeseries metrics collection:

[metrics]
timeseries_retention_minutes = 60
timeseries_interval_seconds = 10

Quick configuration examples

[server]
port = 8000

[log]
level = "debug"
format = "console"

[storage]
backend = "local"
local_path = "./dev_data"

[auth]
enabled = false

[compaction]
enabled = false

Best practices

1. Use arc.toml for permanent settings

Store configuration in arc.toml and version control it (without secrets):

[storage]
backend = "s3"
s3_bucket = "arc"
s3_region = "us-east-1"
# Credentials via environment variables

2. Use environment variables for secrets

export ARC_STORAGE_S3_ACCESS_KEY="your_access_key"
export ARC_STORAGE_S3_SECRET_KEY="your_secret_key"

3. Let Arc auto-detect resources

Arc automatically detects optimal query engine settings based on your system. Only override if you have specific requirements:

[database]
# Leave commented for auto-detection
# max_connections = 28
# memory_limit = "8GB"
# thread_count = 14

4. Enable features progressively

Start simple, add features as needed:

Basic configuration (storage + auth)
Compaction (for query optimization)
Retention policies (for data management)
WAL (for zero data loss guarantee)

5. Monitor configuration impact

Check metrics after configuration changes:

# Memory usage
curl http://localhost:8000/api/v1/metrics/memory

# Query performance
curl http://localhost:8000/api/v1/metrics/query-pool

# Compaction status
curl http://localhost:8000/api/v1/compaction/status

Troubleshooting

Configuration not loading

# Verify TOML syntax (use any TOML validator)
# Check file exists in expected location
ls -la arc.toml

# Arc looks for arc.toml in:
# 1. Current directory
# 2. /etc/arc/arc.toml (native install)

Environment variables not working

# Verify they're set
env | grep ARC_

# Use correct prefix and format
export ARC_SERVER_PORT=8000     # Correct
export SERVER_PORT=8000         # Wrong - missing ARC_ prefix

Resource issues

# Check current settings via metrics
curl http://localhost:8000/api/v1/metrics/memory

# Adjust in arc.toml:
[database]
memory_limit = "4GB"
max_connections = 16

Enterprise configuration

The following configuration sections are available with an Arc Enterprise license. See each feature's dedicated page for detailed configuration.

Clustering

See Clustering & High Availability for full configuration reference.

[cluster]
enabled = true
node_id = "writer-01"
role = "writer"
cluster_name = "production"
seeds = ["10.0.1.10:9000"]

Tiered storage

See Tiered Storage for full configuration reference.

[tiered_storage]
enabled = true
default_hot_max_age_days = 30

[tiered_storage.cold]
enabled = true
backend = "s3"
s3_bucket = "arc-archive"

Audit logging

See Audit Logging for full configuration reference.

[audit_log]
enabled = true
retention_days = 90

Query governance

See Query Governance for full configuration reference.

[governance]
enabled = true
default_rate_limit_per_min = 60

Query management

See Query Management for full configuration reference.

[query_management]
enabled = true

Next steps

Clustering & High Availability - Multi-node cluster configuration
Tiered Storage - Hot/cold storage tiering
Audit Logging - Compliance and security logging
Advanced Features - Compaction and WAL

On this page