BasekickLabs

Retention Policies

Define per-measurement retention in Arc Enterprise and let the licensed scheduler enforce it automatically on a cron schedule instead of triggering the API by hand.

Retention policies allow you to automatically manage data lifecycle by defining how long data should be kept.

Automatic Scheduling Available

Arc Enterprise includes automatic retention enforcement. See Automated Scheduling to configure scheduled execution.

Overview

Retention policies in Arc help you:

  • Define data retention periods at database or measurement level
  • Automatically clean up old data through manual execution
  • Reduce storage costs by removing unnecessary historical data
  • Maintain compliance with data retention requirements
  • Test deletion operations safely with dry-run mode

How it works

Arc implements retention through physical file deletion:

  1. Scanning: Examines Parquet files in measurement directories
  2. Metadata Analysis: Reads file metadata to find maximum timestamps
  3. Identification: Locates files where all rows are older than the cutoff date
  4. Deletion: Physically removes entire files from disk

Cutoff Calculation: cutoff_date = today - retention_days - buffer_days

Tiered storage

With tiered storage, a policy applies to the cold tier as well as to local files. After each measurement's hot files, retention checks the cold files whose partition starts before the cutoff. Each one is read for its newest timestamp, deleted from cold storage if every row is older than the cutoff, and then removed from the tier metadata. A measurement whose data has all moved to cold storage is processed too.

The execute response reports the cold tier on its own. deleted_count and files_deleted include these figures.

FieldMeaning
cold_files_deletedCold files deleted (in a dry run: that would be)
cold_rows_deletedRows in those files
cold_files_skippedCold files left in place
cold_skipped_reasonWhy they were left in place

A cold file is left in place and counted, never deleted blindly, when:

  • it cannot be read or deleted, for example while the cold store is unreachable;
  • tiering has quarantined it;
  • tiered_storage.cold.enabled is off on this node. Retention then handles local files only, as without tiering.

The run still completes, with status completed_with_errors and the reason in error_message. Retention never stops local deletions because of the cold tier.

A tier row whose cold object is already gone is cleaned up. If the same file is still in local storage, the row is pointed back at it instead. A dry run reports the cold files it would delete, but not these row cleanups.

Retention runs only on the primary writer. In a cluster, other nodes keep their tier rows for the deleted cold files until they become primary and run retention themselves. Queries are not affected: they read cold storage directly.

Policies in a cluster

Since 27.01.1 a cluster keeps its retention policies in the cluster state, not in each node's own database (arc#1324).

Before 27.01.1. A policy lived only on the node that received it, while only the primary writer runs retention. So:

  • a policy created on any other node answered 201 but never ran;
  • after a failover, the new primary ran only its own policies. A policy that was deleting data stopped, and one that had never run could start.

From 27.01.1.

  • Write anywhere. Create, update or delete a policy on any node; it applies cluster-wide.
  • Read anywhere. Every node answers GET with the same policies and IDs.
  • Runs read the cluster's policy. A run, scheduled or manual, reads the policy as the cluster holds it when the run starts.
  • Survives a failover. A new primary runs the same policies.
  • History stays per node. The execution history and last_execution_* stay on the node that ran the policy. After a failover, the new primary shows none until it runs the policy.
  • Backups carry them. Every node that runs retention keeps its own retention_policies table equal to the cluster's policies, so a backup of that node carries the current ones.

A standalone node is unchanged.

Taking over the existing policies

Once every node runs 27.01.1, the cluster turns on cluster-wide retention by itself: cluster_policies.retention_enabled in GET /api/v1/cluster. Until then the leader lists the nodes still on an earlier release in retention_blocking_nodes.

The leader then reads every node's own policies. It takes them over by itself only when:

  • every node answers;
  • no policy name has two different definitions on two nodes;
  • every active policy has already completed a run on some node since it was last changed. Inactive policies are taken over as they are.

A policy that never ran is never activated without an admin choosing it: it may have been created on a node that never runs retention.

Otherwise each node keeps its own policies, and the primary goes on running its own, as before. cluster_policies.kinds.retention.last_error in GET /api/v1/cluster says why. The leader checks again every 30 seconds, doubling the wait up to 10 minutes. An admin can choose which node's policies to take over:

# On the Raft leader: every node's own policies, and a hash of each node's set
curl -H "Authorization: Bearer $ARC_ADMIN_TOKEN" \
  http://leader:8000/api/v1/cluster/policies

# Take over one node's policies, naming the hash GET returned for it
curl -X POST -H "Authorization: Bearer $ARC_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"kind": "retention", "node": "arc-writer-0", "hash": "<hash>"}' \
  http://leader:8000/api/v1/cluster/policies/adopt

Both endpoints are admin-only and run on the Raft leader. Once retention_enabled is on, any other node answers 409 with the leader's ID in leader. POST answers 202: cluster_policies.kinds.retention in GET /api/v1/cluster shows the outcome (seeded, or last_error).

The node's policies are taken over only if it still holds exactly the ones the hash names. If they changed, nothing is taken over and last_error says so: look again and choose again. Checking the hash takes a collect, so retention writes answer 503 for a few seconds either way.

When the policies are taken over:

  • IDs. If they come from one node, and no other node ever issued those IDs, they keep their IDs. Otherwise every policy gets a new ID above any ID a node held, and the old IDs answer 404. An old ID never names a different policy.
  • The other nodes' policies. A policy the cluster did not take over is moved, with its history, to retention_policies_dormant and retention_executions_dormant in that node's database. It never runs again.
  • Writes wait a few seconds. While the leader reads the nodes' policies, a retention write answers 503. Retry it.
  • Restoring a backup. In a cluster, restoring arc.db from a backup no longer restores retention policies: the cluster's policies win, and the backup's own rows go to the dormant tables. A full disaster recovery onto a fresh Raft cluster takes the restored policies over again, with new IDs, because every node then holds the same rows.
  • A restarted node waits. A node whose own table already holds the cluster's policies answers retention requests with 503, and runs nothing, until it has caught up with the cluster after a restart (seconds).

A node where retention is disabled counts too: its database may still hold the policies of when retention ran there.

A node that is gone for good must be removed from the cluster before the policies can be taken over.

No rolling back a single node

Once cluster-wide retention is on, a node running an earlier release can no longer join. Roll back the whole cluster from a backup, never one node in place.

Status codes

Create, update and delete answer:

StatusWhen
201 / 200Done.
202Create only: the cluster took the policy, but this node has not applied it yet. List the policies to see it.
400The policy is invalid (including a name, database or measurement longer than 256 bytes), its name is taken, or retention_days is not greater than buffer_days.
404No such policy.
503The cluster cannot take the write right now (no leader, or the policies are being taken over). Retry.

API endpoints

Create policy

Create a new retention policy:

POST /api/v1/retention

Request Body:

{
  "name": "delete_old_metrics",
  "database": "telegraf",
  "measurement": "cpu",
  "retention_days": 90,
  "buffer_days": 7,
  "is_active": true
}

Parameters:

  • name (string, required): Unique policy identifier
  • database (string, required): Target database name
  • measurement (string, optional): Target measurement. Omit it, or send null or "", for a database-wide policy (stored and returned as null).
  • retention_days (integer, required): Number of days to retain data; greater than 0 and greater than buffer_days
  • buffer_days (integer, optional, default 0): Safety margin in days, added to retention_days; may not be negative
  • is_active (boolean, optional, default false): Whether the policy runs. Send true to enable it. The default stays false so that a client that leaves it out never creates a policy that starts deleting.

List policies

Retrieve all retention policies:

GET /api/v1/retention

Response:

[
  {
    "id": 1,
    "name": "delete_old_metrics",
    "database": "telegraf",
    "measurement": "cpu",
    "retention_days": 90,
    "buffer_days": 7,
    "is_active": true,
    "last_execution_time": "2024-01-20T02:00:00Z",
    "last_execution_status": "completed",
    "last_deleted_count": 1500,
    "created_at": "2024-01-15T10:30:00Z",
    "updated_at": "2024-01-15T10:30:00Z"
  }
]

Get single policy

Retrieve a specific retention policy:

GET /api/v1/retention/{policy_id}

Update policy

Update an existing retention policy:

PUT /api/v1/retention/{policy_id}

Request Body: any subset of the create fields. A field you leave out keeps its stored value; a field you send is checked with the create rules and written. To pause a policy:

{ "is_active": false }

To make a measurement-pinned policy database-wide, send "measurement": null. Leaving measurement out keeps the pin. null for any other field, or a body with no fields, is refused with 400.

retention_days must stay greater than buffer_days. If you send only one of them, it is checked against the stored value of the other.

Delete policy

Remove a retention policy:

DELETE /api/v1/retention/{policy_id}

Execute policy

Manually trigger a retention policy:

POST /api/v1/retention/{policy_id}/execute

Request Body:

{
  "dry_run": false,
  "confirm": true
}

Dry Run Example:

{
  "dry_run": true,
  "confirm": false
}

Response:

{
  "policy_id": 1,
  "policy_name": "cpu-30d",
  "deleted_count": 125000,
  "files_deleted": 2,
  "execution_time_ms": 84,
  "dry_run": true,
  "cutoff_date": "2023-10-22T00:00:00Z",
  "affected_measurements": ["cpu"]
}

In a dry run, deleted_count (rows) and files_deleted are what a real run would delete. skipped_files and skipped_reason appear only when some files could not be processed. With tiered storage, the cold-tier fields described in Tiered storage appear too.

View execution history

View past executions of a retention policy:

GET /api/v1/retention/{policy_id}/executions?limit=50

Response:

{
  "policy_id": 1,
  "executions": [
    {
      "id": 7,
      "policy_id": 1,
      "execution_time": "2024-01-20T02:00:00Z",
      "status": "completed",
      "deleted_count": 1500,
      "cutoff_date": "2023-12-21T02:00:00Z",
      "execution_duration_ms": 2500,
      "error_message": ""
    }
  ]
}

Executions are listed newest first. status is running while the run is in progress, then completed, completed_with_errors when some files were left in place (error_message says why), or failed. A finished run's error_message is an empty string when there is nothing to report.

A running entry has execution_duration_ms: 0 and error_message: null until the run finishes; check status, not the duration, to tell whether a run is done. If Arc stops in the middle of a run, nothing marks that entry finished: it stays running. Before v27.01.1 the history left running executions out entirely.

Since Arc 27.01.1, a policy ID with no retention policy behind it, mistyped or deleted, answers 404 Retention policy not found, as GET of the policy does, and a policy that has not run yet lists "executions": []. In a cluster, a node that has not yet caught up with the cluster's retention policies after a restart answers 503; retry it. Earlier versions answered 200 with "executions": null for an unknown ID and for a policy that had not run, and in a cluster a deleted policy's history could still be listed for a moment after the delete.

Configuration parameters

Retention days

The number of days to keep data before it becomes eligible for deletion. Choose based on:

  • Business requirements
  • Compliance regulations
  • Storage capacity
  • Query patterns

Example: retention_days: 90 keeps data for 90 days.

Buffer days

A safety margin added to the retention period to prevent accidental deletion of recent data.

Recommended Values:

  • Development: 7 days
  • Production: 14-30 days

Example: With retention_days: 90 and buffer_days: 7, data older than 97 days will be deleted.

Database vs measurement level

Database-wide policy:

{
  "database": "telegraf",
  "measurement": null,
  "retention_days": 365
}

Measurement-specific policy:

{
  "database": "telegraf",
  "measurement": "cpu",
  "retention_days": 90
}

Use measurement-specific policies for granular control over different data types.

Usage examples

Example 1: clean old metrics

import os
import requests

ARC_TOKEN = os.environ["ARC_TOKEN"]

# Create a retention policy for old CPU metrics
response = requests.post(
    "http://localhost:8000/api/v1/retention",
    headers={"Authorization": f"Bearer {ARC_TOKEN}"},
    json={
        "name": "cpu_cleanup",
        "database": "telegraf",
        "measurement": "cpu",
        "retention_days": 90,
        "buffer_days": 7,
        "is_active": True
    }
)

policy_id = response.json()["id"]

# Test with dry run first
dry_run = requests.post(
    f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
    headers={"Authorization": f"Bearer {ARC_TOKEN}"},
    json={"dry_run": True, "confirm": False}
)

print(f"Would delete {dry_run.json()['files_deleted']} files")

# Execute for real
if input("Proceed? (yes/no): ") == "yes":
    result = requests.post(
        f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
        headers={"Authorization": f"Bearer {ARC_TOKEN}"},
        json={"dry_run": False, "confirm": True}
    )
    print(f"Deleted {result.json()['files_deleted']} files")

Example 2: database-wide retention

import os

ARC_TOKEN = os.environ["ARC_TOKEN"]

# Apply retention to all measurements in a database
response = requests.post(
    "http://localhost:8000/api/v1/retention",
    headers={"Authorization": f"Bearer {ARC_TOKEN}"},
    json={
        "name": "database_cleanup",
        "database": "telegraf",
        "measurement": None,  # Apply to all measurements
        "retention_days": 180,
        "buffer_days": 14,
        "is_active": True
    }
)

Example 3: list and monitor policies

import os

ARC_TOKEN = os.environ["ARC_TOKEN"]

# List all policies
policies = requests.get(
    "http://localhost:8000/api/v1/retention",
    headers={"Authorization": f"Bearer {ARC_TOKEN}"}
)

for policy in policies.json():
    print(f"Policy: {policy['name']}")
    print(f"  Last executed: {policy['last_execution_time']}")
    print(f"  Last deleted: {policy['last_deleted_count']} rows")

    # Get execution history
    history = requests.get(
        f"http://localhost:8000/api/v1/retention/{policy['id']}/executions?limit=10",
        headers={"Authorization": f"Bearer {ARC_TOKEN}"}
    )
    print(f"  Recent executions: {len(history.json()['executions'])}")

Best practices

1. Always test first

Use dry-run mode before executing retention policies:

# Always start with dry run
result = requests.post(
    f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
    json={"dry_run": True, "confirm": False}
)

# Review what will be deleted
print(f"Would delete {result.json()['files_deleted']} files ({result.json()['deleted_count']} rows)")

2. Use buffer days

Implement a safety buffer to prevent accidental deletion:

{
  "retention_days": 90,
  "buffer_days": 14  // 14-day safety margin
}

3. Start conservative

Begin with longer retention periods and gradually shorten:

// Start here
{"retention_days": 365, "buffer_days": 30}

// After monitoring, reduce if needed
{"retention_days": 180, "buffer_days": 14}

4. Test in non-production

Create and test policies in a development environment first:

# Development environment
export ARC_ENV=dev
# Test policies thoroughly before production

5. Monitor execution history

Regularly check the last_deleted_count field:

# Check if deletion counts are as expected
policy = requests.get(f"/api/v1/retention/{policy_id}").json()
if policy['last_deleted_count'] > 10000:
    print("Warning: Large deletion detected!")

6. Use measurement-specific policies

Create granular policies for different data types:

# High-frequency metrics - shorter retention
{"measurement": "cpu", "retention_days": 30}

# Business metrics - longer retention
{"measurement": "revenue", "retention_days": 730}

Important limitations

Storage backends

Retention runs against the configured storage backend: local disk, S3 or Azure Blob Storage. With tiered storage it also covers the cold tier; see Tiered storage.

File-level granularity

Retention operates at the file level, not row level. A file is only deleted if all rows are older than the cutoff date.

Compaction affects eligibility

For optimal retention policy effectiveness, ensure your data is properly compacted. Files with mixed timestamps may not be eligible for deletion.

No rollback

Deleted data cannot be recovered. Always:

  1. Use dry-run mode first
  2. Maintain backups of critical data
  3. Test in non-production environments

Sequential processing

Retention policies process measurements sequentially. Large databases may take time to process.

Works best with compacted files

Retention policies are most effective when files contain data from similar time periods. Enable automatic compaction for better results.

Troubleshooting

No files being deleted

Problem: Dry run shows 0 files to delete.

Solutions:

  • Check that data actually exists older than retention_days + buffer_days
  • Verify the policy targets the correct database and measurement
  • Ensure files are fully older than the cutoff (file-level granularity)

Policy not executing

Problem: Manual execution returns an error.

Solutions:

  • Verify the policy is_active is set to true
  • Check that confirm: true is set for actual execution
  • Ensure you have write permissions on the data directory

Unexpected file count

Problem: More/fewer files than expected are being deleted.

Solutions:

  • Remember: Only files where all rows are older than cutoff are deleted
  • Check file timestamps using ls -l on the measurement directory
  • Review recent compaction activity that may have merged files

On this page