Retention Policies
Define per-measurement retention in Arc Enterprise and let the licensed scheduler enforce it automatically on a cron schedule instead of triggering the API by hand.
Retention policies allow you to automatically manage data lifecycle by defining how long data should be kept.
Automatic Scheduling Available
Arc Enterprise includes automatic retention enforcement. See Automated Scheduling to configure scheduled execution.
Overview
Retention policies in Arc help you:
- Define data retention periods at database or measurement level
- Automatically clean up old data through manual execution
- Reduce storage costs by removing unnecessary historical data
- Maintain compliance with data retention requirements
- Test deletion operations safely with dry-run mode
How it works
Arc implements retention through physical file deletion:
- Scanning: Examines Parquet files in measurement directories
- Metadata Analysis: Reads file metadata to find maximum timestamps
- Identification: Locates files where all rows are older than the cutoff date
- Deletion: Physically removes entire files from disk
Cutoff Calculation: cutoff_date = today - retention_days - buffer_days
Tiered storage
With tiered storage, a policy applies to the cold tier as well as to local files. After each measurement's hot files, retention checks the cold files whose partition starts before the cutoff. Each one is read for its newest timestamp, deleted from cold storage if every row is older than the cutoff, and then removed from the tier metadata. A measurement whose data has all moved to cold storage is processed too.
The execute response reports the cold tier on its own. deleted_count and
files_deleted include these figures.
| Field | Meaning |
|---|---|
cold_files_deleted | Cold files deleted (in a dry run: that would be) |
cold_rows_deleted | Rows in those files |
cold_files_skipped | Cold files left in place |
cold_skipped_reason | Why they were left in place |
A cold file is left in place and counted, never deleted blindly, when:
- it cannot be read or deleted, for example while the cold store is unreachable;
- tiering has quarantined it;
tiered_storage.cold.enabledis off on this node. Retention then handles local files only, as without tiering.
The run still completes, with status completed_with_errors and the reason in
error_message. Retention never stops local deletions because of the cold
tier.
A tier row whose cold object is already gone is cleaned up. If the same file is still in local storage, the row is pointed back at it instead. A dry run reports the cold files it would delete, but not these row cleanups.
Retention runs only on the primary writer. In a cluster, other nodes keep their tier rows for the deleted cold files until they become primary and run retention themselves. Queries are not affected: they read cold storage directly.
Policies in a cluster
Since 27.01.1 a cluster keeps its retention policies in the cluster state, not in each node's own database (arc#1324).
Before 27.01.1. A policy lived only on the node that received it, while only the primary writer runs retention. So:
- a policy created on any other node answered
201but never ran; - after a failover, the new primary ran only its own policies. A policy that was deleting data stopped, and one that had never run could start.
From 27.01.1.
- Write anywhere. Create, update or delete a policy on any node; it applies cluster-wide.
- Read anywhere. Every node answers
GETwith the same policies and IDs. - Runs read the cluster's policy. A run, scheduled or manual, reads the policy as the cluster holds it when the run starts.
- Survives a failover. A new primary runs the same policies.
- History stays per node. The execution history and
last_execution_*stay on the node that ran the policy. After a failover, the new primary shows none until it runs the policy. - Backups carry them. Every node that runs retention keeps its own
retention_policiestable equal to the cluster's policies, so a backup of that node carries the current ones.
A standalone node is unchanged.
Taking over the existing policies
Once every node runs 27.01.1, the cluster turns on cluster-wide retention by
itself: cluster_policies.retention_enabled in GET /api/v1/cluster. Until then the leader lists the nodes still on an
earlier release in retention_blocking_nodes.
The leader then reads every node's own policies. It takes them over by itself only when:
- every node answers;
- no policy name has two different definitions on two nodes;
- every active policy has already completed a run on some node since it was last changed. Inactive policies are taken over as they are.
A policy that never ran is never activated without an admin choosing it: it may have been created on a node that never runs retention.
Otherwise each node keeps its own policies, and the primary goes on running
its own, as before. cluster_policies.kinds.retention.last_error in
GET /api/v1/cluster says why. The leader checks again every 30 seconds,
doubling the wait up to 10 minutes. An admin can choose which node's policies
to take over:
# On the Raft leader: every node's own policies, and a hash of each node's set
curl -H "Authorization: Bearer $ARC_ADMIN_TOKEN" \
http://leader:8000/api/v1/cluster/policies
# Take over one node's policies, naming the hash GET returned for it
curl -X POST -H "Authorization: Bearer $ARC_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"kind": "retention", "node": "arc-writer-0", "hash": "<hash>"}' \
http://leader:8000/api/v1/cluster/policies/adoptBoth endpoints are admin-only and run on the Raft leader. Once
retention_enabled is on, any other node answers 409 with the leader's ID
in leader. POST answers 202: cluster_policies.kinds.retention in
GET /api/v1/cluster shows the outcome (seeded, or last_error).
The node's policies are taken over only if it still holds exactly the ones the
hash names. If they changed, nothing is taken over and last_error says so:
look again and choose again. Checking the hash takes a collect, so retention
writes answer 503 for a few seconds either way.
When the policies are taken over:
- IDs. If they come from one node, and no other node ever issued those
IDs, they keep their IDs. Otherwise every policy gets a new ID above any ID a node
held, and the old IDs answer
404. An old ID never names a different policy. - The other nodes' policies. A policy the cluster did not take over is
moved, with its history, to
retention_policies_dormantandretention_executions_dormantin that node's database. It never runs again. - Writes wait a few seconds. While the leader reads the nodes' policies, a
retention write answers
503. Retry it. - Restoring a backup. In a cluster, restoring
arc.dbfrom a backup no longer restores retention policies: the cluster's policies win, and the backup's own rows go to the dormant tables. A full disaster recovery onto a fresh Raft cluster takes the restored policies over again, with new IDs, because every node then holds the same rows. - A restarted node waits. A node whose own table already holds the
cluster's policies answers retention requests with
503, and runs nothing, until it has caught up with the cluster after a restart (seconds).
A node where retention is disabled counts too: its database may still hold the policies of when retention ran there.
A node that is gone for good must be removed from the cluster before the policies can be taken over.
No rolling back a single node
Once cluster-wide retention is on, a node running an earlier release can no longer join. Roll back the whole cluster from a backup, never one node in place.
Status codes
Create, update and delete answer:
| Status | When |
|---|---|
201 / 200 | Done. |
202 | Create only: the cluster took the policy, but this node has not applied it yet. List the policies to see it. |
400 | The policy is invalid (including a name, database or measurement longer than 256 bytes), its name is taken, or retention_days is not greater than buffer_days. |
404 | No such policy. |
503 | The cluster cannot take the write right now (no leader, or the policies are being taken over). Retry. |
API endpoints
Create policy
Create a new retention policy:
POST /api/v1/retentionRequest Body:
{
"name": "delete_old_metrics",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": true
}Parameters:
name(string, required): Unique policy identifierdatabase(string, required): Target database namemeasurement(string, optional): Target measurement. Omit it, or sendnullor"", for a database-wide policy (stored and returned asnull).retention_days(integer, required): Number of days to retain data; greater than 0 and greater thanbuffer_daysbuffer_days(integer, optional, default0): Safety margin in days, added toretention_days; may not be negativeis_active(boolean, optional, defaultfalse): Whether the policy runs. Sendtrueto enable it. The default staysfalseso that a client that leaves it out never creates a policy that starts deleting.
List policies
Retrieve all retention policies:
GET /api/v1/retentionResponse:
[
{
"id": 1,
"name": "delete_old_metrics",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": true,
"last_execution_time": "2024-01-20T02:00:00Z",
"last_execution_status": "completed",
"last_deleted_count": 1500,
"created_at": "2024-01-15T10:30:00Z",
"updated_at": "2024-01-15T10:30:00Z"
}
]Get single policy
Retrieve a specific retention policy:
GET /api/v1/retention/{policy_id}Update policy
Update an existing retention policy:
PUT /api/v1/retention/{policy_id}Request Body: any subset of the create fields. A field you leave out keeps its stored value; a field you send is checked with the create rules and written. To pause a policy:
{ "is_active": false }To make a measurement-pinned policy database-wide, send "measurement": null. Leaving measurement out keeps the pin. null for any other field, or a body with no fields, is refused with 400.
retention_days must stay greater than buffer_days. If you send only one of them, it is checked against the stored value of the other.
Delete policy
Remove a retention policy:
DELETE /api/v1/retention/{policy_id}Execute policy
Manually trigger a retention policy:
POST /api/v1/retention/{policy_id}/executeRequest Body:
{
"dry_run": false,
"confirm": true
}Dry Run Example:
{
"dry_run": true,
"confirm": false
}Response:
{
"policy_id": 1,
"policy_name": "cpu-30d",
"deleted_count": 125000,
"files_deleted": 2,
"execution_time_ms": 84,
"dry_run": true,
"cutoff_date": "2023-10-22T00:00:00Z",
"affected_measurements": ["cpu"]
}In a dry run, deleted_count (rows) and files_deleted are what a real run
would delete. skipped_files and skipped_reason appear only when some files
could not be processed.
With tiered storage, the cold-tier fields described in
Tiered storage appear too.
View execution history
View past executions of a retention policy:
GET /api/v1/retention/{policy_id}/executions?limit=50Response:
{
"policy_id": 1,
"executions": [
{
"id": 7,
"policy_id": 1,
"execution_time": "2024-01-20T02:00:00Z",
"status": "completed",
"deleted_count": 1500,
"cutoff_date": "2023-12-21T02:00:00Z",
"execution_duration_ms": 2500,
"error_message": ""
}
]
}Executions are listed newest first. status is running while the run is in
progress, then completed, completed_with_errors when some files were left
in place (error_message says why), or failed. A finished run's
error_message is an empty string when there is nothing to report.
A running entry has execution_duration_ms: 0 and error_message: null
until the run finishes; check status, not the duration, to tell whether a
run is done. If Arc stops in the middle of a run, nothing marks that entry
finished: it stays running. Before v27.01.1 the history left running
executions out entirely.
Since Arc 27.01.1, a policy ID with no retention policy behind it, mistyped or
deleted, answers 404 Retention policy not found, as GET of the policy does,
and a policy that has not run yet lists "executions": []. In a cluster, a
node that has not yet caught up with the cluster's retention policies after a
restart answers 503; retry it. Earlier versions answered 200 with
"executions": null for an unknown ID and for a policy that had not run, and
in a cluster a deleted policy's history could still be listed for a moment
after the delete.
Configuration parameters
Retention days
The number of days to keep data before it becomes eligible for deletion. Choose based on:
- Business requirements
- Compliance regulations
- Storage capacity
- Query patterns
Example: retention_days: 90 keeps data for 90 days.
Buffer days
A safety margin added to the retention period to prevent accidental deletion of recent data.
Recommended Values:
- Development: 7 days
- Production: 14-30 days
Example: With retention_days: 90 and buffer_days: 7, data older than 97 days will be deleted.
Database vs measurement level
Database-wide policy:
{
"database": "telegraf",
"measurement": null,
"retention_days": 365
}Measurement-specific policy:
{
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90
}Use measurement-specific policies for granular control over different data types.
Usage examples
Example 1: clean old metrics
import os
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create a retention policy for old CPU metrics
response = requests.post(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "cpu_cleanup",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": True
}
)
policy_id = response.json()["id"]
# Test with dry run first
dry_run = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"dry_run": True, "confirm": False}
)
print(f"Would delete {dry_run.json()['files_deleted']} files")
# Execute for real
if input("Proceed? (yes/no): ") == "yes":
result = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"dry_run": False, "confirm": True}
)
print(f"Deleted {result.json()['files_deleted']} files")Example 2: database-wide retention
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Apply retention to all measurements in a database
response = requests.post(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "database_cleanup",
"database": "telegraf",
"measurement": None, # Apply to all measurements
"retention_days": 180,
"buffer_days": 14,
"is_active": True
}
)Example 3: list and monitor policies
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# List all policies
policies = requests.get(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"}
)
for policy in policies.json():
print(f"Policy: {policy['name']}")
print(f" Last executed: {policy['last_execution_time']}")
print(f" Last deleted: {policy['last_deleted_count']} rows")
# Get execution history
history = requests.get(
f"http://localhost:8000/api/v1/retention/{policy['id']}/executions?limit=10",
headers={"Authorization": f"Bearer {ARC_TOKEN}"}
)
print(f" Recent executions: {len(history.json()['executions'])}")Best practices
1. Always test first
Use dry-run mode before executing retention policies:
# Always start with dry run
result = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
json={"dry_run": True, "confirm": False}
)
# Review what will be deleted
print(f"Would delete {result.json()['files_deleted']} files ({result.json()['deleted_count']} rows)")2. Use buffer days
Implement a safety buffer to prevent accidental deletion:
{
"retention_days": 90,
"buffer_days": 14 // 14-day safety margin
}3. Start conservative
Begin with longer retention periods and gradually shorten:
// Start here
{"retention_days": 365, "buffer_days": 30}
// After monitoring, reduce if needed
{"retention_days": 180, "buffer_days": 14}4. Test in non-production
Create and test policies in a development environment first:
# Development environment
export ARC_ENV=dev
# Test policies thoroughly before production5. Monitor execution history
Regularly check the last_deleted_count field:
# Check if deletion counts are as expected
policy = requests.get(f"/api/v1/retention/{policy_id}").json()
if policy['last_deleted_count'] > 10000:
print("Warning: Large deletion detected!")6. Use measurement-specific policies
Create granular policies for different data types:
# High-frequency metrics - shorter retention
{"measurement": "cpu", "retention_days": 30}
# Business metrics - longer retention
{"measurement": "revenue", "retention_days": 730}Important limitations
Storage backends
Retention runs against the configured storage backend: local disk, S3 or Azure Blob Storage. With tiered storage it also covers the cold tier; see Tiered storage.
File-level granularity
Retention operates at the file level, not row level. A file is only deleted if all rows are older than the cutoff date.
Compaction affects eligibility
For optimal retention policy effectiveness, ensure your data is properly compacted. Files with mixed timestamps may not be eligible for deletion.
No rollback
Deleted data cannot be recovered. Always:
- Use dry-run mode first
- Maintain backups of critical data
- Test in non-production environments
Sequential processing
Retention policies process measurements sequentially. Large databases may take time to process.
Works best with compacted files
Retention policies are most effective when files contain data from similar time periods. Enable automatic compaction for better results.
Troubleshooting
No files being deleted
Problem: Dry run shows 0 files to delete.
Solutions:
- Check that data actually exists older than
retention_days + buffer_days - Verify the policy targets the correct database and measurement
- Ensure files are fully older than the cutoff (file-level granularity)
Policy not executing
Problem: Manual execution returns an error.
Solutions:
- Verify the policy
is_activeis set totrue - Check that
confirm: trueis set for actual execution - Ensure you have write permissions on the data directory
Unexpected file count
Problem: More/fewer files than expected are being deleted.
Solutions:
- Remember: Only files where all rows are older than cutoff are deleted
- Check file timestamps using
ls -lon the measurement directory - Review recent compaction activity that may have merged files
Related topics
- Delete Operations - Manual delete operations for specific data
- Continuous Queries - Downsample data before deletion
- Compaction - Optimize file structure for better retention