Retention Policies

Retention policies allow you to automatically manage data lifecycle by defining how long data should be kept.

Overview

Retention policies in Arc help you:

• Define data retention periods at database or measurement level
• Automatically clean up old data through manual execution
• Reduce storage costs by removing unnecessary historical data
• Maintain compliance with data retention requirements
• Test deletion operations safely with dry-run mode

How it works

Arc implements retention through physical file deletion:

1. Scanning: Examines Parquet files in measurement directories
2. Metadata Analysis: Reads file metadata to find maximum timestamps
3. Identification: Locates files where all rows are older than the cutoff date
4. Deletion: Physically removes entire files from disk

Cutoff Calculation: cutoff_date = today - retention_days - buffer_days

API endpoints

Create policy

Create a new retention policy:

POST /api/v1/retention

Request Body:

{
  "name": "delete_old_metrics",
  "database": "telegraf",
  "measurement": "cpu",
  "retention_days": 90,
  "buffer_days": 7,
  "is_active": true
}

Parameters:

• name (string, required): Unique policy identifier
• database (string, required): Target database name
• measurement (string, optional): Target measurement. Omit it, or send null or "", for a database-wide policy (stored and returned as null).
• retention_days (integer, required): Number of days to retain data; greater than 0 and greater than buffer_days
• buffer_days (integer, optional, default 0): Safety margin in days, added to retention_days; may not be negative
• is_active (boolean, optional, default false): Whether the policy runs. Send true to enable it. The default stays false so that a client that leaves it out never creates a policy that starts deleting.

List policies

Retrieve all retention policies:

GET /api/v1/retention

Response:

[
  {
    "id": 1,
    "name": "delete_old_metrics",
    "database": "telegraf",
    "measurement": "cpu",
    "retention_days": 90,
    "buffer_days": 7,
    "is_active": true,
    "last_execution_time": "2024-01-20T02:00:00Z",
    "last_execution_status": "completed",
    "last_deleted_count": 1500,
    "created_at": "2024-01-15T10:30:00Z",
    "updated_at": "2024-01-15T10:30:00Z"
  }
]

Get single policy

Retrieve a specific retention policy:

GET /api/v1/retention/{policy_id}

Update policy

Update an existing retention policy:

PUT /api/v1/retention/{policy_id}

Request Body: any subset of the create fields. A field you leave out keeps its stored value; a field you send is checked with the create rules and written. To pause a policy:

{ "is_active": false }

To make a measurement-pinned policy database-wide, send "measurement": null. Leaving measurement out keeps the pin. null for any other field, or a body with no fields, is refused with 400.

retention_days must stay greater than buffer_days. If you send only one of them, it is checked against the stored value of the other.

Delete policy

Remove a retention policy:

DELETE /api/v1/retention/{policy_id}

Execute policy

Manually trigger a retention policy:

POST /api/v1/retention/{policy_id}/execute

Request Body:

{
  "dry_run": false,
  "confirm": true
}

Dry Run Example:

{
  "dry_run": true,
  "confirm": false
}

Response:

{
  "policy_id": 1,
  "policy_name": "cpu-30d",
  "deleted_count": 125000,
  "files_deleted": 2,
  "execution_time_ms": 84,
  "dry_run": true,
  "cutoff_date": "2023-10-22T00:00:00Z",
  "affected_measurements": ["cpu"]
}

In a dry run, deleted_count (rows) and files_deleted are what a real run would delete. skipped_files and skipped_reason appear only when some files could not be processed.

View execution history

View past executions of a retention policy:

GET /api/v1/retention/{policy_id}/executions?limit=50

Response:

{
  "policy_id": 1,
  "executions": [
    {
      "id": 7,
      "policy_id": 1,
      "execution_time": "2024-01-20T02:00:00Z",
      "status": "completed",
      "deleted_count": 1500,
      "cutoff_date": "2023-12-21T02:00:00Z",
      "execution_duration_ms": 2500,
      "error_message": null
    }
  ]
}

status is completed, completed_with_errors when some files were left in place (error_message says why), or failed.

Configuration parameters

Retention days

The number of days to keep data before it becomes eligible for deletion. Choose based on:

• Business requirements
• Compliance regulations
• Storage capacity
• Query patterns

Example: retention_days: 90 keeps data for 90 days.

Buffer days

A safety margin added to the retention period to prevent accidental deletion of recent data.

Recommended Values:

• Development: 7 days
• Production: 14-30 days

Example: With retention_days: 90 and buffer_days: 7, data older than 97 days will be deleted.

Database vs measurement level

Database-wide policy:

{
  "database": "telegraf",
  "measurement": null,
  "retention_days": 365
}

Measurement-specific policy:

{
  "database": "telegraf",
  "measurement": "cpu",
  "retention_days": 90
}

Use measurement-specific policies for granular control over different data types.

Usage examples

Example 1: clean old metrics

import os
import requests

ARC_TOKEN = os.environ["ARC_TOKEN"]

# Create a retention policy for old CPU metrics
response = requests.post(
    "http://localhost:8000/api/v1/retention",
    headers={"Authorization": f"Bearer {ARC_TOKEN}"},
    json={
        "name": "cpu_cleanup",
        "database": "telegraf",
        "measurement": "cpu",
        "retention_days": 90,
        "buffer_days": 7,
        "is_active": True
    }
)

policy_id = response.json()["id"]

# Test with dry run first
dry_run = requests.post(
    f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
    headers={"Authorization": f"Bearer {ARC_TOKEN}"},
    json={"dry_run": True, "confirm": False}
)

print(f"Would delete {dry_run.json()['files_deleted']} files")

# Execute for real
if input("Proceed? (yes/no): ") == "yes":
    result = requests.post(
        f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
        headers={"Authorization": f"Bearer {ARC_TOKEN}"},
        json={"dry_run": False, "confirm": True}
    )
    print(f"Deleted {result.json()['files_deleted']} files")

Example 2: database-wide retention

import os

ARC_TOKEN = os.environ["ARC_TOKEN"]

# Apply retention to all measurements in a database
response = requests.post(
    "http://localhost:8000/api/v1/retention",
    headers={"Authorization": f"Bearer {ARC_TOKEN}"},
    json={
        "name": "database_cleanup",
        "database": "telegraf",
        "measurement": None,  # Apply to all measurements
        "retention_days": 180,
        "buffer_days": 14,
        "is_active": True
    }
)

Example 3: list and monitor policies

import os

ARC_TOKEN = os.environ["ARC_TOKEN"]

# List all policies
policies = requests.get(
    "http://localhost:8000/api/v1/retention",
    headers={"Authorization": f"Bearer {ARC_TOKEN}"}
)

for policy in policies.json():
    print(f"Policy: {policy['name']}")
    print(f"  Last executed: {policy['last_execution_time']}")
    print(f"  Last deleted: {policy['last_deleted_count']} rows")

    # Get execution history
    history = requests.get(
        f"http://localhost:8000/api/v1/retention/{policy['id']}/executions?limit=10",
        headers={"Authorization": f"Bearer {ARC_TOKEN}"}
    )
    print(f"  Recent executions: {len(history.json())}")

Best practices

1. Always test first

Use dry-run mode before executing retention policies:

# Always start with dry run
result = requests.post(
    f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
    json={"dry_run": True, "confirm": False}
)

# Review what will be deleted
print(f"Would delete {result.json()['files_deleted']} files ({result.json()['deleted_count']} rows)")

2. Use buffer days

Implement a safety buffer to prevent accidental deletion:

{
  "retention_days": 90,
  "buffer_days": 14  // 14-day safety margin
}

3. Start conservative

Begin with longer retention periods and gradually shorten:

// Start here
{"retention_days": 365, "buffer_days": 30}

// After monitoring, reduce if needed
{"retention_days": 180, "buffer_days": 14}

4. Test in non-production

Create and test policies in a development environment first:

# Development environment
export ARC_ENV=dev
# Test policies thoroughly before production

5. Monitor execution history

Regularly check the last_deleted_count field:

# Check if deletion counts are as expected
policy = requests.get(f"/api/v1/retention/{policy_id}").json()
if policy['last_deleted_count'] > 10000:
    print("Warning: Large deletion detected!")

6. Use measurement-specific policies

Create granular policies for different data types:

# High-frequency metrics - shorter retention
{"measurement": "cpu", "retention_days": 30}

# Business metrics - longer retention
{"measurement": "revenue", "retention_days": 730}

Important limitations

Storage backends

Retention runs against the configured storage backend: local disk, S3 or Azure Blob Storage.

File-level granularity

Retention operates at the file level, not row level. A file is only deleted if all rows are older than the cutoff date.

No rollback

Deleted data cannot be recovered. Always:

1. Use dry-run mode first
2. Maintain backups of critical data
3. Test in non-production environments

Sequential processing

Retention policies process measurements sequentially. Large databases may take time to process.

Works best with compacted files

Retention policies are most effective when files contain data from similar time periods. Enable automatic compaction for better results.

Troubleshooting

No files being deleted

Problem: Dry run shows 0 files to delete.

Solutions:

• Check that data actually exists older than retention_days + buffer_days
• Verify the policy targets the correct database and measurement
• Ensure files are fully older than the cutoff (file-level granularity)

Policy not executing

Problem: Manual execution returns an error.

Solutions:

• Verify the policy is_active is set to true
• Check that confirm: true is set for actual execution
• Ensure you have write permissions on the data directory

Unexpected file count

Problem: More/fewer files than expected are being deleted.

Solutions:

• Remember: Only files where all rows are older than cutoff are deleted
• Check file timestamps using ls -l on the measurement directory
• Review recent compaction activity that may have merged files

Related topics

• Delete Operations - Manual delete operations for specific data
• Continuous Queries - Downsample data before deletion
• Compaction - Optimize file structure for better retention