Tiered Storage

Reduce storage costs with automatic hot/cold data tiering. Recent data stays on fast primary storage while older data moves to object storage that costs a fraction of block storage per gigabyte, and queries keep reading it in place.

Overview

Arc Enterprise implements a 2-tier storage model:

┌─────────────────────────────────────────────────────────────┐
│                    Arc Tiered Storage                        │
│                                                              │
│  HOT TIER (Local / Primary Storage Backend)                 │
│  ├── Recent data (configurable, default: 30 days)           │
│  ├── Optimized for low-latency queries                      │
│  └── Cost: $$$                                               │
│                                                              │
│           │ Automatic migration (age-based)                  │
│           ▼                                                  │
│                                                              │
│  COLD TIER (S3 / Azure Blob Storage)                        │
│  ├── Historical data (30+ days)                             │
│  ├── Optimized for cost efficiency                          │
│  └── Cost: $                                                 │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Key features:

• Age-based migration — Data older than a configurable threshold automatically moves to the cold tier
• Per-database policies — Override global defaults for specific databases
• Hot-only databases — Exclude specific databases from tiering entirely
• Scheduled migrations — Cron-based scheduler runs migrations automatically
• Manual migrations — Trigger migrations on-demand via API
• Zero recompression — Files move as-is with no re-encoding overhead
• Transparent queries — Queries automatically span both tiers

Configuration

Global settings

[tiered_storage]
enabled = true
migration_schedule = "0 2 * * *"     # Cron: run at 2am daily
migration_max_concurrent = 4          # Parallel file transfers
migration_batch_size = 100            # Files per migration batch
default_hot_max_age_days = 30         # Data older than this moves to cold

Cold tier backend

[tiered_storage.cold]
enabled = true
backend = "s3"
s3_bucket = "arc-archive"
s3_region = "us-east-1"
# s3_access_key = ""          # Use env: ARC_TIERED_STORAGE_COLD_S3_ACCESS_KEY
# s3_secret_key = ""          # Use env: ARC_TIERED_STORAGE_COLD_S3_SECRET_KEY
s3_use_ssl = true
s3_path_style = false

Environment variables

# Global tiering settings
ARC_TIERED_STORAGE_ENABLED=true
ARC_TIERED_STORAGE_MIGRATION_SCHEDULE="0 2 * * *"
ARC_TIERED_STORAGE_MIGRATION_MAX_CONCURRENT=4
ARC_TIERED_STORAGE_MIGRATION_BATCH_SIZE=100
ARC_TIERED_STORAGE_DEFAULT_HOT_MAX_AGE_DAYS=30

# Cold tier (S3)
ARC_TIERED_STORAGE_COLD_ENABLED=true
ARC_TIERED_STORAGE_COLD_BACKEND=s3
ARC_TIERED_STORAGE_COLD_S3_BUCKET=arc-archive
ARC_TIERED_STORAGE_COLD_S3_REGION=us-east-1
ARC_TIERED_STORAGE_COLD_S3_ACCESS_KEY=your_key
ARC_TIERED_STORAGE_COLD_S3_SECRET_KEY=your_secret
ARC_TIERED_STORAGE_COLD_S3_USE_SSL=true

# Cold tier (Azure)
ARC_TIERED_STORAGE_COLD_BACKEND=azure
ARC_TIERED_STORAGE_COLD_AZURE_CONTAINER=arc-archive
ARC_TIERED_STORAGE_COLD_AZURE_ACCOUNT_NAME=your_account
ARC_TIERED_STORAGE_COLD_AZURE_ACCOUNT_KEY=your_key

Per-database policies

Override the global default_hot_max_age_days for specific databases, or exclude databases from tiering entirely.

Create policy

curl -X POST http://localhost:8000/api/v1/tiering/policies \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "database": "telemetry",
    "hot_max_age_days": 7
  }'

Create hot-only policy

Exclude a database from tiering:

curl -X POST http://localhost:8000/api/v1/tiering/policies \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "database": "realtime",
    "hot_only": true
  }'

List policies

curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8000/api/v1/tiering/policies

Get policy

curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8000/api/v1/tiering/policies/telemetry

Update policy

curl -X PUT http://localhost:8000/api/v1/tiering/policies/telemetry \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"hot_max_age_days": 14}'

Delete policy

curl -X DELETE http://localhost:8000/api/v1/tiering/policies/telemetry \
  -H "Authorization: Bearer $TOKEN"

API reference

All tiering endpoints require admin authentication.

Get tiering status

curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8000/api/v1/tiering/status

Response:

{
  "success": true,
  "data": {
    "enabled": true,
    "cold_backend": "s3",
    "default_hot_max_age_days": 30,
    "migration_schedule": "0 2 * * *",
    "last_migration": "2026-02-13T02:00:00Z"
  }
}

List files by tier

# All files
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8000/api/v1/tiering/files

# Filter by tier
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8000/api/v1/tiering/files?tier=cold"

# Filter by database
curl -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8000/api/v1/tiering/files?database=telemetry&limit=50"

Trigger manual migration

curl -X POST http://localhost:8000/api/v1/tiering/migrate \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "success": true,
  "data": {
    "message": "Migration started",
    "files_eligible": 42
  }
}

On a cluster node that is not the primary writer the request is refused with 409 Conflict naming the node's role; see Clusters and shared storage.

Rescan tiers

curl -X POST http://localhost:8000/api/v1/tiering/scan \
  -H "Authorization: Bearer $TOKEN"

Response:

{
  "files_scanned": 1590,
  "files_registered": 3,
  "files_skipped": 1587,
  "errors": 0,
  "cold_synced": 0,
  "hot_retired": 2
}

The scan brings the node's tier metadata in line with storage; every migration cycle runs the same pass first. It lists the hot tier and registers files that have no row yet (files_registered; files already known count as files_skipped), and it retires the rows of hot files that are no longer in hot storage (hot_retired) — what compaction leaves behind once it has consumed a measurement's raw files — after a five-minute grace period, so a file flushed while the scan runs is never retired. Cold rows are never touched. Without that retirement, a measurement whose files had all moved to cold kept an empty hot tier in every read, and a query without a time range on it returned no rows. In a cluster the scan also records the files other nodes moved to cold as cold_synced, and reports "cold_sync_failed": true when the cold tier could not be listed; see Clusters and shared storage.

Get migration statistics

curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8000/api/v1/tiering/stats

Response:

{
  "success": true,
  "data": {
    "total_files_migrated": 1250,
    "total_bytes_migrated": 53687091200,
    "hot_files": 340,
    "cold_files": 1250,
    "last_migration_duration_ms": 45000,
    "last_migration_files": 42
  }
}

Clusters and shared storage

How tiering behaves in a cluster depends on the deployment pattern.

Shared object storage (Pattern 2, cluster.shared_storage_mode = true). The hot tier is the one bucket every node shares, so exactly one node moves files: migration runs on the Raft leader among the writers — the same primary-writer gate that retention and continuous queries use — and the gate is re-checked on every cycle, so a leader change takes effect at the next tick with no restart. Every other node, readers and the remaining writers alike, still runs a cycle on the same schedule, but only to bring its own tier metadata in line with storage: it lists the cold tier and records what the primary moved. Each node routes queries from its own metadata, so that sync is what keeps cold data in those nodes' query routing, and what stops a newly elected primary from re-migrating files its predecessor already moved.

Two consequences to plan for:

• All nodes fire on the same cron minute, so a non-primary learns a cycle's moves on its next cycle. Only a measurement's very first cold file is affected, and only for one cycle: once a node has any cold row for a measurement, its cold reads cover later moves.
• The sync only ever adds or flips rows to cold. A cold row whose object has since disappeared from the cold tier is reported in a warning; the primary's reconciliation reverts such a row to hot when it finds the hot copy still present, so the file is migrated again rather than lost. A migration cycle requested while one is already running on the same node answers 409 Conflict instead of overlapping it.

On a node that syncs but never migrates, GET /api/v1/tiering/status reports "role_gated": true under scheduler, and POST /api/v1/tiering/migrate answers 409 Conflict with the node's role — retry it against the primary writer. POST /api/v1/tiering/scan runs the same sync on any node and reports how many rows it learned as cold_synced; see Rescan tiers.

A node in shared-storage mode must have cluster.role set to writer, reader or compactor. A node left at the default standalone refuses to start: it could win Raft leadership without ever passing the primary-writer gate, which would stop every singleton task on the cluster.

Per-node local storage (Pattern 1). A per-node cluster whose nodes keep each other in step with file replication (cluster.replication_enabled = true) is gated the same way: only the primary writer migrates, every node syncs its tier metadata from the cold tier each cycle, and every file the primary moves to cold is removed from the cluster file manifest before its hot copy goes — which is what unlinks the replica on every other node and stops replication from pulling it back. A per-node cluster without replication shares nothing and keeps migrating per node.

Two requirements follow. Every replicating node must run tiering with the same cold backend: once the primary migrates a file, the only way a node can read it is through its own cold row and cold backend; a node with tiering off or no cold tier cannot see that data any more, and Arc says so at startup on such a node. And a node that has no cold row yet for a measurement does not read that measurement's first migrated file until its next cold sync — at the default 0 2 * * * schedule up to a day; shorten tiered_storage.migration_schedule on readers if that matters.

Best practices

1. Start with 30-day hot retention — This is a good default for most workloads. Monitor query patterns and adjust based on how often historical data is accessed.

2. Keep cold on S3 Standard or Azure Hot — Arc writes cold objects with no storage class or access tier set, so on S3 they are Standard and on Azure they take the storage account's default tier, Hot unless the account owner changed it. Leave it that way. The saving tiering is built for is block storage versus object storage: a hot tier on EBS or local NVMe costs several times what S3 Standard does per gigabyte, and cold data stays queryable at object-store latency. Cold data in Arc is old telemetry you still query, not an archive you file away, so cheaper classes are a false economy: STANDARD_IA and GLACIER_IR add a per-gigabyte retrieval fee to every query that touches cold files, and GLACIER, DEEP_ARCHIVE and Azure Archive objects cannot be read at all without a restore. That is why Arc has no storage-class or access-tier setting. Do not attach a bucket lifecycle rule to the cold prefix that transitions objects out of Standard.

3. Schedule migrations during off-peak hours — The default 0 2 * * * (2am daily) works well for most deployments.

4. Use per-database policies for different retention needs — Real-time dashboards may need 7-day hot data, while compliance databases may need 90 days.

5. Mark real-time databases as hot-only — Databases used exclusively for real-time dashboards should skip tiering so no query on them pays object-store latency.

6. Use IAM roles or managed identity — For cloud deployments, use IAM roles (AWS) or managed identity (Azure) instead of access keys.

Next steps

• Audit Logging — Track tiering operations for compliance
• Automated Scheduling — Combine tiering with scheduled retention policies