Tiered Storage
Age Arc Enterprise data from hot storage to S3, Azure Blob Storage or Google Cloud Storage with per-database policies; cold files stay queryable in place, with no retrieval step.
Reduce storage costs with automatic hot/cold data tiering. Recent data stays on fast primary storage while older data moves to object storage that costs a fraction of block storage per gigabyte, and queries keep reading it in place.
Overview
Arc Enterprise implements a 2-tier storage model:
┌─────────────────────────────────────────────────────────────┐
│ Arc Tiered Storage │
│ │
│ HOT TIER (Local / Primary Storage Backend) │
│ ├── Recent data (configurable, default: 30 days) │
│ ├── Optimized for low-latency queries │
│ └── Cost: $$$ │
│ │
│ │ Automatic migration (age-based) │
│ ▼ │
│ │
│ COLD TIER (S3 / Azure Blob Storage) │
│ ├── Historical data (30+ days) │
│ ├── Optimized for cost efficiency │
│ └── Cost: $ │
│ │
└─────────────────────────────────────────────────────────────┘Key features:
- Age-based migration — Data older than a configurable threshold automatically moves to the cold tier
- Per-database policies — Override global defaults for specific databases
- Hot-only databases — Exclude specific databases from tiering entirely
- Scheduled migrations — Cron-based scheduler runs migrations automatically
- Manual migrations — Trigger migrations on-demand via API
- Zero recompression — Files move as-is with no re-encoding overhead
- Transparent queries — Queries automatically span both tiers
Configuration
Global settings
[tiered_storage]
enabled = true
migration_schedule = "0 2 * * *" # Cron: run at 2am daily
migration_max_concurrent = 4 # Parallel file transfers
migration_batch_size = 100 # Files per migration batch
default_hot_max_age_days = 30 # Data older than this moves to cold
scan_timeout = "2h" # Budget for one tier scanscan_timeout bounds one tier scan on every path that runs one: the scan at startup, POST /api/v1/tiering/scan, and the pre-migration scan inside a migration cycle or POST /api/v1/tiering/migrate. Before v27.01.1 the first two were capped at 30 minutes with no way to change it, while a cycle gave its scan the cycle's own 2 hours (arc#1154).
Inside a cycle the scan runs on a budget derived from the cycle's, so it gets whichever of the two is shorter and migration keeps the remainder. At the default the two are equal and the derivation changes nothing; setting scan_timeout above 2 hours is harmless because the cycle still bounds the scan.
Setting it below the time a full scan takes is harmful rather than conservative. Hot-row retirement runs only after the whole walk, so a scan that always truncates never retires a row for a file that has left hot storage — and a measurement whose remaining rows are all cold drops that node's local files from reads. Size it from how long a full scan takes on your storage root, which scales with the number of objects under it rather than their size. Zero, negative and malformed durations are refused at startup.
Cold tier backend
[tiered_storage.cold]
enabled = true
backend = "s3"
s3_bucket = "arc-archive"
s3_region = "us-east-1"
# s3_access_key = "" # Use env: ARC_TIERED_STORAGE_COLD_S3_ACCESS_KEY
# s3_secret_key = "" # Use env: ARC_TIERED_STORAGE_COLD_S3_SECRET_KEY
s3_use_ssl = true
s3_path_style = falseCold objects are written with no storage class set, so they land in S3 Standard, and are queried in place. Arc has no storage-class setting, deliberately; see Best practices. Keep bucket lifecycle rules off the cold prefix: a transition to GLACIER or DEEP_ARCHIVE makes those files unreadable to queries until restored.
Environment variables
# Global tiering settings
ARC_TIERED_STORAGE_ENABLED=true
ARC_TIERED_STORAGE_MIGRATION_SCHEDULE="0 2 * * *"
ARC_TIERED_STORAGE_MIGRATION_MAX_CONCURRENT=4
ARC_TIERED_STORAGE_MIGRATION_BATCH_SIZE=100
ARC_TIERED_STORAGE_DEFAULT_HOT_MAX_AGE_DAYS=30
# Cold tier (S3)
ARC_TIERED_STORAGE_COLD_ENABLED=true
ARC_TIERED_STORAGE_COLD_BACKEND=s3
ARC_TIERED_STORAGE_COLD_S3_BUCKET=arc-archive
ARC_TIERED_STORAGE_COLD_S3_REGION=us-east-1
ARC_TIERED_STORAGE_COLD_S3_ACCESS_KEY=your_key
ARC_TIERED_STORAGE_COLD_S3_SECRET_KEY=your_secret
ARC_TIERED_STORAGE_COLD_S3_USE_SSL=true
# Cold tier (Azure)
ARC_TIERED_STORAGE_COLD_BACKEND=azure
ARC_TIERED_STORAGE_COLD_AZURE_CONTAINER=arc-archive
ARC_TIERED_STORAGE_COLD_AZURE_ACCOUNT_NAME=your_account
ARC_TIERED_STORAGE_COLD_AZURE_ACCOUNT_KEY=your_key
# Cold tier (GCS)
ARC_TIERED_STORAGE_COLD_BACKEND=gcs
ARC_TIERED_STORAGE_COLD_GCS_BUCKET=arc-archive
ARC_TIERED_STORAGE_COLD_GCS_PREFIX=cold27.01.1: a cold tier that fails to start is no longer skipped silently
When the cold tier is enabled but its backend cannot be built at startup (a bad endpoint or credentials file, any backend), a query whose time range reaches cold files this node's tier metadata records answers HTTP 503 naming the cold tier. Earlier releases left the cold files out and answered with the hot rows alone, as a success. The startup log names the failure. New in 27.01.1 as well: a cold tier that shares a location with primary storage, or sits inside it (or contains it), is refused at startup.
Use Environment Variables for Secrets
Store access keys and secret keys in environment variables rather than in arc.toml. This keeps secrets out of version control.
An empty environment variable does not clear a value set in arc.toml
Arc reads the environment through viper's AutomaticEnv, which treats an empty variable as unset. ARC_TIERED_STORAGE_COLD_S3_ENDPOINT="" therefore leaves whatever s3_endpoint the file sets in place — a cold tier meant for AWS silently keeps addressing that endpoint, every cold listing fails, and each migration cycle skips. To clear a key, remove it from the file or give the variable the value you actually want. The arc.toml shipped in the image set MinIO values (localhost:9000, static minioadmin keys, plain HTTP, path-style) under [tiered_storage.cold] through 26.09.2; since 26.09.3 those lines are commented out and Arc's built-in defaults apply — no endpoint, HTTPS, virtual-hosted addressing, credentials from the AWS chain. If you relied on the shipped MinIO values, uncomment that block as a set.
Per-database policies
Override the global default_hot_max_age_days for specific databases, or exclude databases from tiering entirely.
Set a policy
PUT /api/v1/tiering/policies/{database} creates the policy or replaces it. The database comes from the path; the body carries the policy:
curl -X PUT http://localhost:8000/api/v1/tiering/policies/telemetry \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"hot_max_age_days": 7}'| Field | Type | Description |
|---|---|---|
hot_only | bool | Exclude the database from tiering. Omitted means false. |
hot_max_age_days | int | Days a file stays hot. Omitted means the global default_hot_max_age_days. |
A PUT replaces the whole policy
A field left out of the body is reset, not kept. On a database pinned hot_only, PUT with only {"hot_max_age_days": 14} turns hot_only off, and the next migration cycle moves its older files to cold. Send every field you want to keep.
Since Arc 27.01.1 the database must exist: a PUT for a database that does
not answers 404 Database '<name>' not found and stores nothing. Earlier
versions stored the policy. A database exists by the same rule
GET /api/v1/databases/{name} answers with: it has a marker, a file, a
cold-tier record or, in a cluster, a manifest entry. Arc's own directories in
the storage root, and the Iceberg warehouse's (see
Apache Iceberg Export), are not
databases and answer 404 too. To set a policy before
the first write, create the database first with POST /api/v1/databases.
GET, DELETE and /effective do not check the database, so DELETE still
removes a policy that a deleted database left behind.
Pin a database to the hot tier
curl -X PUT http://localhost:8000/api/v1/tiering/policies/realtime \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"hot_only": true}'hot_only stops future migrations only. Files of that database already in cold storage stay there, and queries keep reading them from cold. Since 27.01.1 the response says how many there are (arc#1304), and its policy is the stored row with its id and timestamps. Earlier versions answered id: 0 and zero timestamps here, and in a GET of the policy until the next restart (arc#1619):
{
"message": "Policy updated successfully",
"database": "realtime",
"policy": {
"id": 3,
"database": "realtime",
"hot_only": true,
"hot_max_age_days": null,
"created_at": "2026-11-10T14:02:11Z",
"updated_at": "2026-11-12T09:30:45Z"
},
"cold_files": {
"recorded_before_update": 42,
"scope": "node_local_metadata",
"note": "hot_only affects future tiering only; it does not restore existing cold files."
}
}recorded_before_update counts the cold files recorded in the tier metadata of the node that answered, before the change, leaving out quarantined files. It is not a cluster-wide total. If Arc cannot count them, the request fails with 500 and the policy is not changed. cold_files appears only when the request sets hot_only: true.
List policies
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/tiering/policiesGet a policy
Returns 404 when the database has no custom policy:
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/tiering/policies/telemetryGet the effective policy
The policy Arc applies, with global defaults filled in; source is custom or global:
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/tiering/policies/telemetry/effectiveDelete a policy
The database reverts to the global defaults:
curl -X DELETE http://localhost:8000/api/v1/tiering/policies/telemetry \
-H "Authorization: Bearer $ARC_TOKEN"Status codes
PUT and DELETE answer:
| Status | When |
|---|---|
200 | Done. |
400 | The policy is invalid, for example a database name longer than 256 bytes (in a cluster). Since 27.01.1 also a PUT whose database name is not a valid storage path segment (invalid database name "<name>"), or, on a local volume that does not distinguish case, a name that exists on disk in another spelling. |
404 | DELETE of a database with no custom policy. Since 27.01.1 also a PUT for a database that does not exist. |
503 | A cluster cannot take the write right now: no leader, or the policies are being taken over. Retry. |
500 | Anything else, including (since 27.01.1) a PUT whose database check itself failed (Failed to check database). |
Before 27.01.1, DELETE answered 404 for any error, and a PUT stored the
policy for a database that did not exist or whose name was not valid.
Policies in a cluster
Since 27.01.1 a cluster keeps its tiering policies in the cluster state, not in each node's own database (arc#1324).
Before 27.01.1. A policy lived only on the node that received it, while only the primary writer migrates files to cold. So:
- a policy set on any other node was stored and answered
200, but never applied; - after a failover, the new primary ignored every policy set on the old one.
From 27.01.1.
- Set anywhere. Set a policy on any node; it applies cluster-wide. The
database check runs on the node that answers, though: on a cluster that
keeps data per node, a database that so far holds only the marker made by
POST /api/v1/databasesexists only on the writer, so set its policy on the writer, or after the first write. - Read anywhere. Every node answers
GETwith the same policies. - Survives a failover. A new primary applies the same policies.
- Backups carry them. Every node that runs tiering keeps its own
tiering_policiestable equal to the cluster's policies, so a backup of such a node carries the current ones.
A standalone node is unchanged.
Taking over the existing policies. Once every node runs 27.01.1, the
cluster turns on cluster policies by itself (cluster_policies.enabled in
GET /api/v1/cluster). It then takes over the policies each node already holds:
- While it collects them, a tiering policy write answers
503for up to a few seconds. Retry it. - Every node must answer. The leader first checks it can reach every node. If one is down it takes nothing over, the policies stay on each node as before, and it tries again after 30 seconds, doubling the wait up to 10 minutes.
cluster_policies.kinds.tieringinGET /api/v1/clustershowsseeded,missing_nodes,last_errorandfailed_attempts.- A node that is gone for good must be removed from the cluster before the policies can be taken over.
How the nodes' policies are merged, in the direction that moves less data:
- A database
hot_onlyon any node stayshot_only. - Otherwise it keeps the longest hot age any node applied to it: the policy's
hot_max_age_days, or that node'sdefault_hot_max_age_dayswhere the policy left it unset.
No rolling back a single node
Once cluster policies are on, a node running an earlier release can no longer join. Roll back the whole cluster from a backup, never one node in place.
API reference
All tiering endpoints require admin authentication.
Get tiering status
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/tiering/statusResponse:
{
"success": true,
"data": {
"enabled": true,
"cold_backend": "s3",
"default_hot_max_age_days": 30,
"migration_schedule": "0 2 * * *",
"last_migration": "2026-02-13T02:00:00Z"
}
}On a cluster node with peer file replication the status also carries replication_events — {"applied", "dropped", "failed"} — counting the tier rows this node wrote for files it did not ingest itself: files the replication puller pulled, and local copies removed because another node moved them to cold (since 26.09.3, see Clusters and shared storage). dropped is zero on a healthy node; a non-zero dropped or failed means some rows wait for the next tier scan, which costs that node partition pruning until then. The field is absent on a node that has never received such a report.
List files by tier
# All files
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/tiering/files
# Filter by tier
curl -H "Authorization: Bearer $ARC_TOKEN" \
"http://localhost:8000/api/v1/tiering/files?tier=cold"
# Filter by database
curl -H "Authorization: Bearer $ARC_TOKEN" \
"http://localhost:8000/api/v1/tiering/files?database=telemetry&limit=50"
# Filter by database and tier: the database's files in that tier
curl -H "Authorization: Bearer $ARC_TOKEN" \
"http://localhost:8000/api/v1/tiering/files?database=telemetry&tier=cold"tier and database can be combined. Before 27.01.1, tier was ignored when database was set, and the list held the database's files in every tier (arc#1430).
tier must be hot or cold, in lower case. Since 27.01.1 any other value, such as COLD or warm, is answered 400 invalid tier; before, it listed the hot files with a 200.
limit defaults to 100 and keeps the first entries of the list, whose order depends on the filters:
| Filters | Order |
|---|---|
| none | hot files, then cold files, each oldest partition first |
tier | oldest partition first |
database | newest partition first |
database and tier | newest partition first |
So with the default limit, ?tier=cold returns the oldest cold files and ?database=telemetry&tier=cold the newest. The response's count is the number of files returned after limit, not the number that matched.
Cold files missing from cold storage
Since v27.01.1 (arc#1551), every cold-tier metadata sync checks each cold file Arc has a record of but the cold listing did not return, and reports two counts:
- lost: no cold object and no hot copy. These files cannot be read, and nothing in Arc restores them. Queries still answer, without those rows.
- hot_copy: the cold object is gone, but the hot copy is still read. No data is missing.
They appear in:
- the
cold_missing_objectsfield ofGET /api/v1/tiering/status(lost,hot_copy,unchecked,sample,checked_at); /healthundertiering(counts and check time only);- the gauges
arc_tiering_cold_rows_missing_objectsandarc_tiering_cold_rows_missing_objects_hot_copy.
Alert on arc_tiering_cold_rows_missing_objects > 0. Arc also logs an Error naming a sample of paths.
The usual cause is a bucket lifecycle or expiry rule, or a delete made outside Arc, on the cold prefix. To fix it, restore the objects from a backup, or delete that data through Arc so its records are retired. The count updates at the next sync: at startup, on POST /api/v1/tiering/scan, or in the migration cycle.
Notes:
- Reported on standalone nodes and on the cluster's primary writer only. On readers the field is absent and the gauges are 0 (arc#1563).
- A DELETE or retention run whose record cleanup failed (it logs that) leaves records that count as lost until a later run retires them.
- Up to 1,000 files are checked per sync, within 30 seconds. A check that could not examine every file never lowers the lost count.
Trigger manual migration
curl -X POST http://localhost:8000/api/v1/tiering/migrate \
-H "Authorization: Bearer $ARC_TOKEN"Response:
{
"success": true,
"data": {
"message": "Migration started",
"files_eligible": 42
}
}On a cluster node that is not the primary writer the request is refused with 409 Conflict naming the node's role; see Clusters and shared storage.
Rescan tiers
curl -X POST http://localhost:8000/api/v1/tiering/scan \
-H "Authorization: Bearer $ARC_TOKEN"Response:
{
"files_scanned": 1590,
"files_registered": 3,
"files_skipped": 1587,
"errors": 0,
"cold_synced": 0,
"hot_retired": 2,
"hot_rows_written": 3
}The scan brings the node's tier metadata in line with storage. Every migration cycle runs the same pass first, and since 26.09.3 every node runs it once at startup too, in the background, so files that arrived while the node was down — or were already on disk when upgrading — have rows as soon as the node is up; the log line Startup tier scan completed carries the same counts. It lists the hot tier and accounts for every hot file (files_registered; files whose row already says cold count as files_skipped), writing a row only where one is missing or out of date (hot_rows_written, zero on a node whose rows already match its disk), and it retires the rows of hot files that are no longer in hot storage (hot_retired) — what compaction leaves behind once it has consumed a measurement's raw files — after a five-minute grace period and a check that the file really is gone, so a file flushed or pulled while the scan runs is never retired. Cold rows are never touched. Without that retirement, a measurement whose files had all moved to cold kept an empty hot tier in every read, and a query without a time range on it returned no rows. In a cluster with local storage, a node also retires the hot row of a file as soon as the cluster deletes it, including a file that node had removed itself first (its own compaction inputs, or the files its retention run removed), so those rows no longer wait for the next scan (since 27.01.1, #1271). With a cold tier configured, the scan first lists it and records the cold files this node has no row for as cold_synced: in a cluster, the files other nodes moved to cold; on any node, standalone included (since 27.01.1, #1179), the cold rows a restored or lost arc.db no longer holds. It writes those rows in batches of 1,000 and clears the node's tier cache once per measurement per batch rather than once per file, which during a first sync of a large bucket made every query running meanwhile repeat its tier lookup (since 27.01.1, #1188). It reports "cold_sync_failed": true when the cold sync did not finish — the cold tier could not be listed, or the scan ran out of its budget part way through; see Clusters and shared storage.
When a scan runs out of its budget
A scan bounded by scan_timeout can stop part way, and from v27.01.1 it says so rather than looking like a clean pass (arc#1154). The rows it wrote before stopping are real; what it has not done is retire any stale hot row, because retirement runs only after the whole walk.
POST /api/v1/tiering/scan answers 503 Service Unavailable with the counts it reached:
{
"error": "The tier scan ran out of its budget and stopped part way",
"message": "The rows below were written, but the scan is incomplete and no stale hot rows were retired. Raise tiered_storage.scan_timeout above the time a full scan takes on this node.",
"result": {
"files_scanned": 25974,
"files_registered": 25974,
"files_skipped": 0,
"errors": 0,
"cold_synced": 0,
"hot_retired": 0,
"hot_rows_written": 25974,
"truncated": true
}
}Before v27.01.1 the same case returned 500 with context deadline exceeded and no counts at all. A scan cancelled because the node is shutting down also answers 503, with its own message — that one is not a budget problem and raising scan_timeout will not help it.
The startup scan has no response to carry any of this, so GET /api/v1/tiering/status reports the last scan this node ran:
{
"last_scan": {
"files_scanned": 15738,
"files_registered": 15738,
"files_skipped": 0,
"errors": 0,
"cold_synced": 0,
"hot_retired": 0,
"hot_rows_written": 15738,
"truncated": true
},
"last_scan_at": "2026-10-08T20:40:33.880463Z"
}"truncated": true there means this node's tier rows are incomplete and no stale hot row was retired, which costs it partition pruning until a scan completes. Read it rather than calling the scan endpoint to find out, because calling that endpoint starts a scan.
Only one scan at a time
A second POST /api/v1/tiering/scan while one is running is refused with 409 Conflict rather than starting a parallel scan:
{
"error": "A tier scan is already running on this node",
"message": "Wait for it to finish and read its result from GET /api/v1/tiering/status, which reports the last scan."
}The endpoint is synchronous and a scan over a large storage root can take a while, and Arc does not learn that a client disconnected — so without this an operator who gave up on a slow scan and retried would have started another full one over the same storage root, against the same metadata database, each invalidating the tier cache the query path reads. A scan inside a migration cycle is deliberately not refused: the cycle must scan before it migrates, or it would migrate from stale rows.
Get migration statistics
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/tiering/statsResponse:
{
"success": true,
"data": {
"total_files_migrated": 1250,
"total_bytes_migrated": 53687091200,
"hot_files": 340,
"cold_files": 1250,
"last_migration_duration_ms": 45000,
"last_migration_files": 42
}
}Clusters and shared storage
How tiering behaves in a cluster depends on the deployment pattern.
Shared object storage (Pattern 2, cluster.shared_storage_mode = true). The hot tier is the one bucket every node shares, so exactly one node moves files: migration runs on the Raft leader among the writers — the same primary-writer gate that retention and continuous queries use — and the gate is re-checked on every cycle, so a leader change takes effect at the next tick with no restart. Every other node, readers and the remaining writers alike, still runs a cycle on the same schedule, but only to bring its own tier metadata in line with storage: it lists the cold tier and records what the primary moved. Each node routes queries from its own metadata, so that sync is what keeps cold data in those nodes' query routing, and what stops a newly elected primary from re-migrating files its predecessor already moved.
Two consequences to plan for:
- All nodes fire on the same cron minute, so a non-primary learns a cycle's moves on its next cycle. Only a measurement's very first cold file is affected, and only for one cycle: once a node has any cold row for a measurement, its cold reads cover later moves. A node that restarts learns them at boot instead, from the startup tier scan (since 26.09.3), which on shared storage lists the shared hot bucket once per node per start — the same listing the nightly cycle already performs.
- The sync only ever adds or flips rows to cold. A cold row whose object has since disappeared from the cold tier is reported in a warning; the primary's reconciliation reverts such a row to hot when it finds the hot copy still present, so the file is migrated again rather than lost. A migration cycle requested while one is already running on the same node answers
409 Conflictinstead of overlapping it.
On a node that syncs but never migrates, GET /api/v1/tiering/status reports "role_gated": true under scheduler, and POST /api/v1/tiering/migrate answers 409 Conflict with the node's role — retry it against the primary writer. POST /api/v1/tiering/scan runs the same sync on any node and reports how many rows it learned as cold_synced; see Rescan tiers.
A node in shared-storage mode must have cluster.role set to writer, reader or compactor. A node left at the default standalone refuses to start: it could win Raft leadership without ever passing the primary-writer gate, which would stop every singleton task on the cluster.
Per-node local storage (Pattern 1). A per-node cluster whose nodes keep each other in step with file replication (cluster.replication_enabled = true) is gated the same way: only the primary writer migrates, every node syncs its tier metadata from the cold tier each cycle, and every file the primary moves to cold is removed from the cluster file manifest before its hot copy goes — which is what unlinks the replica on every other node and stops replication from pulling it back. A per-node cluster without replication shares nothing and keeps migrating per node.
One requirement follows. Every replicating node must run tiering with the same cold backend: once the primary migrates a file, the only way a node can read it is through its own cold row and cold backend; a node with tiering off or no cold tier cannot see that data any more, and Arc says so at startup on such a node.
Since 26.09.3 a node's tier rows follow its own disk, not only what it ingested. A file the replication puller pulls is registered as hot the moment it lands, and when the primary moves a file to cold the node that removes its local copy confirms the object is in the cold tier before flipping its row to cold — so a measurement's first migrated file is readable on every node within seconds of the migration, not at the next cycle, and nothing needs tuning for it. Two counters show the path working: tier_registered beside pulled in replication_catchup_status on /api/v1/cluster (it tracks pulled where tiering is on, and stays at zero where it is not), and replication_events on /api/v1/tiering/status, whose dropped is zero on a healthy node.
Before 26.09.3: a node's tier rows only described what it ingested
On 26.09.2 and earlier, a node recorded tier rows only for files it flushed itself. Files pulled from a peer had no row until the next scan, and a file another node migrated was not marked cold locally until the next cold sync — at the default 0 2 * * * schedule up to a day. Because the query layer routes reads from those rows, a node not currently ingesting a measurement lost partition pruning for it (GET /api/v1/tiering/files showed far fewer files on it than on the ingesting node), and a measurement with a cold row and no hot row dropped that node's local files from reads entirely: a windowed count(*) over data on its disk returned 0 with success: true. POST /api/v1/tiering/scan on each affected node closes the gap until the next migration cycle reopens it; upgrading removes it.
Reads while the cold tier is unreachable
Since v27.01.1 a read of recent data keeps working while the cold tier cannot be listed: it answers from hot data when Arc can prove the range holds no cold data. A read that does need cold data still fails rather than answer without it, and so does a read with no stated start time.
-
Standalone (arc#1550): the node's own tier metadata decides, once a complete cold-tier sync has recorded every cold object.
-
Cluster (arc#1552): the decision uses a cold mark: the newest day per measurement that may hold cold data, replicated through Raft. It is raised before every cold write, and a read leaves cold out only when its start is after that day. The mark is used only when it is trusted:
- every node runs a release that maintains it, and the mark was seeded from a complete cold listing;
- every member confirmed it within the last 15 minutes. A node rolled back to an older release, or gone silent, makes it untrusted.
- the reading node uses the same cold tier (backend, bucket, prefix, endpoint) as the one the mark was seeded from.
Before a read leaves cold out, the node confirms it is caught up with the Raft leader.
Do not roll a node back to an older release once the mark is in use. Until the mark turns untrusted (up to 15 minutes), that node can write to the cold tier without raising it, and a read in that window can miss those files. If it happens, upgrade the node again and reset the seed (below) so the next complete sync records what it wrote.
While the cold tier is coming back, a read can still return 503 if a listing has not recovered yet; retry it.
GET /api/v1/cluster shows cold_mark.trusted and, when it is not trusted, untrusted_reason. Reads answered this way count in arc_query_cold_mark_skipped_total (standalone: arc_query_cold_unlisted_skipped_total).
If the mark reports a backend conflict, or you changed the cold tier configuration, make every node use the same cold tier first, then reset the seed. The next complete cold-tier sync seeds it again, and marks already recorded are kept:
curl -X POST -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/cluster/cold-mark/resetDeleting rows in cold storage
A row DELETE (POST /api/v1/delete) deletes a cold file when every row in it
matches the clause, and leaves a cold file where only some rows match,
reporting it in the response (207, cold_files). It does not rewrite part
of a cold file. See Delete operations.
A DELETE also checks cold storage for a copy of every daily file it rewrites,
whatever the database's current policy, so a file with a copy left in cold
storage (for example after a database was pinned to the hot tier) is left as
it is until that copy is removed. While cold storage is unreachable, a DELETE
that touches a daily file answers 503; retry once it is back.
Best practices
-
Start with 30-day hot retention — This is a good default for most workloads. Monitor query patterns and adjust based on how often historical data is accessed.
-
Keep cold on S3 Standard or Azure Hot — Arc writes cold objects with no storage class or access tier set, so on S3 they are Standard and on Azure they take the storage account's default tier, Hot unless the account owner changed it. Leave it that way. The saving tiering is built for is block storage versus object storage: a hot tier on EBS or local NVMe costs several times what S3 Standard does per gigabyte, and cold data stays queryable at object-store latency. Cold data in Arc is old telemetry you still query, not an archive you file away, so cheaper classes are a false economy:
STANDARD_IAandGLACIER_IRadd a per-gigabyte retrieval fee to every query that touches cold files, andGLACIER,DEEP_ARCHIVEand Azure Archive objects cannot be read at all without a restore. That is why Arc has no storage-class or access-tier setting. Do not attach a bucket lifecycle rule to the cold prefix that transitions objects out of Standard. Expiry and delete rules on the cold prefix are worse: they remove data Arc still lists. Remove data through Arc's retention or DELETE instead, and watcharc_tiering_cold_rows_missing_objects. -
Schedule migrations during off-peak hours — The default
0 2 * * *(2am daily) works well for most deployments. -
Use per-database policies for different retention needs — Real-time dashboards may need 7-day hot data, while compliance databases may need 90 days.
-
Mark real-time databases as hot-only — Databases used exclusively for real-time dashboards should skip tiering so no query on them pays object-store latency.
-
Use IAM roles or managed identity — For cloud deployments, use IAM roles (AWS) or managed identity (Azure) instead of access keys.
Next steps
- Audit Logging — Track tiering operations for compliance
- Automated Scheduling — Combine tiering with scheduled retention policies
Delete Operations
Delete data from an Arc Enterprise cluster by rewriting Parquet files: enable delete in arc.toml, scope by measurement and time range, and audit every deletion.
Advanced
Internals that shape an Arc Enterprise cluster's behaviour: the write-ahead log, file compaction, query caching, and data-time partitioning of Parquet files.