Automated scheduling
Configure paid scheduling for Arc continuous queries and retention policies, plus planned 27.01.1 edge-sync replication, with license checks, cluster eligibility, and monitoring.
Run continuous queries and retention policies on configurable schedules. Built-in edge-sync replication scheduling is merged for the planned 27.01.1 release.
Overview
Arc OSS provides continuous queries and retention policies with manual API-triggered execution. Arc Enterprise adds automatic scheduling — define your schedules once, and Arc handles execution automatically.
Scheduling requires a valid paid license in every tier, including grace periods. Manual edge sync and air-gap bundle export remain available in OSS.
Schedulers:
| Scheduler | Purpose | Default schedule |
|---|---|---|
| CQ scheduler | Runs continuous queries at their configured intervals | Per-CQ interval |
| Retention scheduler | Enforces retention policies on a cron schedule | Daily at 3am (0 3 * * *) |
| Edge-sync scheduler (planned 27.01.1) | Replicates network spoke files to a hub | 5m after a complete pass; initial failure retry 30s |
CQ scheduler
The CQ Scheduler automatically executes continuous queries at their configured intervals. Each continuous query runs independently on its own schedule.
How it works
- Define continuous queries with intervals via the CQ API
- Enable the CQ scheduler (requires enterprise license)
- Arc automatically executes each CQ at its configured interval
- Results are written to the destination measurement
Configuration
The CQ Scheduler is enabled when continuous queries are enabled and a valid enterprise license is present:
[continuous_query]
enabled = trueARC_CONTINUOUS_QUERY_ENABLED=trueEach continuous query defines its own execution interval when created through the API.
Retention scheduler
The Retention Scheduler automatically enforces retention policies on a cron schedule, deleting data that has exceeded its retention period.
How it works
- Define retention policies via the Retention API
- Enable the retention scheduler (requires enterprise license)
- Arc evaluates all active policies on the configured schedule
- Expired data is automatically deleted
Configuration
[retention]
enabled = true
[scheduler]
retention_schedule = "0 3 * * *" # Cron: daily at 3amARC_RETENTION_ENABLED=true
ARC_SCHEDULER_RETENTION_SCHEDULE="0 3 * * *"Cron Schedule Syntax
The schedule uses standard 5-field cron syntax: minute hour day-of-month month day-of-week.
| Schedule | Meaning |
|---|---|
0 3 * * * | Daily at 3:00 AM |
0 */6 * * * | Every 6 hours |
0 2 * * 0 | Weekly on Sunday at 2:00 AM |
30 1 1 * * | Monthly on the 1st at 1:30 AM |
Edge-sync scheduler (planned for 27.01.1)
A configured network spoke starts automatic replication when Arc starts with
a valid paid license. edge_sync.spoke.sync_interval defaults to 5m, and
edge_sync.spoke.sync_retry_interval starts failure retries at 30s, with
exponential backoff capped at the normal interval. Both durations must be at
least one second; retry must be shorter than the normal interval.
The spoke checks licensing and primary-writer eligibility on every tick and cancels an active scheduled pass when eligibility is lost. Scheduled and manual passes cannot overlap; a busy manual request returns HTTP 409. Bundle export remains manual. A hub does not need a paid license simply to receive a licensed spoke's scheduled transfers.
Use the edge-sync configuration and monitoring reference for setup, environment overrides, retry behavior, and metrics. Edge sync uses these spoke-specific intervals; it is not configured through retention's cron setting or the CQ job API.
Data lifecycle pipeline
Combine CQ and retention scheduling to build a complete data lifecycle pipeline:
Raw Data (1-second resolution)
│
│ CQ: 1-minute aggregation (runs every minute)
▼
1-Minute Data
│
│ CQ: 1-hour aggregation (runs every hour)
▼
1-Hour Data
│
│ CQ: 1-day aggregation (runs daily)
▼
1-Day Data
Retention Schedule (runs daily at 3am):
├── Delete raw data older than 7 days
├── Delete 1-minute data older than 30 days
├── Delete 1-hour data older than 365 days
└── Keep 1-day data indefinitelyExample setup
1. Create continuous queries for downsampling:
# 1-minute aggregation
curl -X POST http://localhost:8000/api/v1/continuous_queries \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "downsample_1min",
"database": "telemetry",
"source_measurement": "sensors_raw",
"destination_measurement": "sensors_1min",
"query": "SELECT time_bucket('\''1 minute'\'', time) as time, device_id, AVG(temperature) as temperature, MAX(pressure) as pressure FROM telemetry.sensors_raw WHERE time >= {start_time} AND time < {end_time} GROUP BY 1, 2",
"interval": "1m",
"is_active": true
}'
# 1-hour aggregation
curl -X POST http://localhost:8000/api/v1/continuous_queries \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "downsample_1hr",
"database": "telemetry",
"source_measurement": "sensors_1min",
"destination_measurement": "sensors_1hr",
"query": "SELECT time_bucket('\''1 hour'\'', time) as time, device_id, AVG(temperature) as temperature, MAX(pressure) as pressure FROM telemetry.sensors_1min WHERE time >= {start_time} AND time < {end_time} GROUP BY 1, 2",
"interval": "1h",
"is_active": true
}'2. Create retention policies:
# Delete raw data after 7 days
curl -X POST http://localhost:8000/api/v1/retention \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "raw_7d",
"database": "telemetry",
"measurement": "sensors_raw",
"retention_days": 7,
"is_active": true
}'
# Delete 1-minute data after 30 days
curl -X POST http://localhost:8000/api/v1/retention \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "1min_30d",
"database": "telemetry",
"measurement": "sensors_1min",
"retention_days": 30,
"is_active": true
}'With enterprise scheduling enabled, these queries and policies run automatically — no cron jobs, no external orchestration.
Best practices
-
Schedule retention during off-peak hours — File deletion generates I/O. The default 3am schedule avoids impacting daytime workloads.
-
Add buffer days to retention policies — Use the
buffer_daysparameter in retention policies to provide a safety margin before deletion. -
Test CQ queries manually first — Before enabling automatic execution, run your continuous query SQL manually to verify correct results.
-
Combine with tiered storage — Use tiered storage to move data to cold storage before retention deletes it, keeping long-term archives at low cost.
-
Monitor CQ execution — Check Arc logs for CQ execution results and errors. Failed CQ executions are logged at WARN level.
Next steps
- Continuous Queries — Create and manage continuous queries (OSS docs)
- Retention Policies — Create and manage retention policies (OSS docs)
- Tiered Storage — Combine scheduling with tiered storage for optimal cost management
Telemetry
What anonymous usage telemetry an Arc Enterprise node reports, what is deliberately excluded, and how to disable it in arc.toml or by environment variable across a cluster.
Profiling with pprof
Enable Go pprof on an Arc Enterprise node with ARC_DEBUG_PPROF to capture heap, goroutine, and CPU profiles, and the loopback and firewall gates required to expose it.