BasekickLabs

Automated scheduling

Configure paid scheduling for Arc continuous queries and retention policies, plus planned 27.01.1 edge-sync replication, with license checks, cluster eligibility, and monitoring.

Run continuous queries and retention policies on configurable schedules. Built-in edge-sync replication scheduling is merged for the planned 27.01.1 release.

Overview

Arc OSS provides continuous queries and retention policies with manual API-triggered execution. Arc Enterprise adds automatic scheduling — define your schedules once, and Arc handles execution automatically.

Scheduling requires a valid paid license in every tier, including grace periods. Manual edge sync and air-gap bundle export remain available in OSS.

Schedulers:

SchedulerPurposeDefault schedule
CQ schedulerRuns continuous queries at their configured intervalsPer-CQ interval
Retention schedulerEnforces retention policies on a cron scheduleDaily at 3am (0 3 * * *)
Edge-sync scheduler (planned 27.01.1)Replicates network spoke files to a hub5m after a complete pass; initial failure retry 30s

CQ scheduler

The CQ Scheduler automatically executes continuous queries at their configured intervals. Each continuous query runs independently on its own schedule.

How it works

  1. Define continuous queries with intervals via the CQ API
  2. Enable the CQ scheduler (requires enterprise license)
  3. Arc automatically executes each CQ at its configured interval
  4. Results are written to the destination measurement

Configuration

The CQ Scheduler is enabled when continuous queries are enabled and a valid enterprise license is present:

[continuous_query]
enabled = true
ARC_CONTINUOUS_QUERY_ENABLED=true

Each continuous query defines its own execution interval when created through the API.

Retention scheduler

The Retention Scheduler automatically enforces retention policies on a cron schedule, deleting data that has exceeded its retention period.

How it works

  1. Define retention policies via the Retention API
  2. Enable the retention scheduler (requires enterprise license)
  3. Arc evaluates all active policies on the configured schedule
  4. Expired data is automatically deleted

Configuration

[retention]
enabled = true

[scheduler]
retention_schedule = "0 3 * * *"   # Cron: daily at 3am
ARC_RETENTION_ENABLED=true
ARC_SCHEDULER_RETENTION_SCHEDULE="0 3 * * *"

Cron Schedule Syntax

The schedule uses standard 5-field cron syntax: minute hour day-of-month month day-of-week.

ScheduleMeaning
0 3 * * *Daily at 3:00 AM
0 */6 * * *Every 6 hours
0 2 * * 0Weekly on Sunday at 2:00 AM
30 1 1 * *Monthly on the 1st at 1:30 AM

Edge-sync scheduler (planned for 27.01.1)

A configured network spoke starts automatic replication when Arc starts with a valid paid license. edge_sync.spoke.sync_interval defaults to 5m, and edge_sync.spoke.sync_retry_interval starts failure retries at 30s, with exponential backoff capped at the normal interval. Both durations must be at least one second; retry must be shorter than the normal interval.

The spoke checks licensing and primary-writer eligibility on every tick and cancels an active scheduled pass when eligibility is lost. Scheduled and manual passes cannot overlap; a busy manual request returns HTTP 409. Bundle export remains manual. A hub does not need a paid license simply to receive a licensed spoke's scheduled transfers.

Use the edge-sync configuration and monitoring reference for setup, environment overrides, retry behavior, and metrics. Edge sync uses these spoke-specific intervals; it is not configured through retention's cron setting or the CQ job API.

Data lifecycle pipeline

Combine CQ and retention scheduling to build a complete data lifecycle pipeline:

Raw Data (1-second resolution)
    │
    │ CQ: 1-minute aggregation (runs every minute)
    ▼
1-Minute Data
    │
    │ CQ: 1-hour aggregation (runs every hour)
    ▼
1-Hour Data
    │
    │ CQ: 1-day aggregation (runs daily)
    ▼
1-Day Data

Retention Schedule (runs daily at 3am):
  ├── Delete raw data older than 7 days
  ├── Delete 1-minute data older than 30 days
  ├── Delete 1-hour data older than 365 days
  └── Keep 1-day data indefinitely

Example setup

1. Create continuous queries for downsampling:

# 1-minute aggregation
curl -X POST http://localhost:8000/api/v1/continuous_queries \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "downsample_1min",
    "database": "telemetry",
    "source_measurement": "sensors_raw",
    "destination_measurement": "sensors_1min",
    "query": "SELECT time_bucket('\''1 minute'\'', time) as time, device_id, AVG(temperature) as temperature, MAX(pressure) as pressure FROM telemetry.sensors_raw WHERE time >= {start_time} AND time < {end_time} GROUP BY 1, 2",
    "interval": "1m",
    "is_active": true
  }'

# 1-hour aggregation
curl -X POST http://localhost:8000/api/v1/continuous_queries \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "downsample_1hr",
    "database": "telemetry",
    "source_measurement": "sensors_1min",
    "destination_measurement": "sensors_1hr",
    "query": "SELECT time_bucket('\''1 hour'\'', time) as time, device_id, AVG(temperature) as temperature, MAX(pressure) as pressure FROM telemetry.sensors_1min WHERE time >= {start_time} AND time < {end_time} GROUP BY 1, 2",
    "interval": "1h",
    "is_active": true
  }'

2. Create retention policies:

# Delete raw data after 7 days
curl -X POST http://localhost:8000/api/v1/retention \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "raw_7d",
    "database": "telemetry",
    "measurement": "sensors_raw",
    "retention_days": 7,
    "is_active": true
  }'

# Delete 1-minute data after 30 days
curl -X POST http://localhost:8000/api/v1/retention \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "1min_30d",
    "database": "telemetry",
    "measurement": "sensors_1min",
    "retention_days": 30,
    "is_active": true
  }'

With enterprise scheduling enabled, these queries and policies run automatically — no cron jobs, no external orchestration.

Best practices

  1. Schedule retention during off-peak hours — File deletion generates I/O. The default 3am schedule avoids impacting daytime workloads.

  2. Add buffer days to retention policies — Use the buffer_days parameter in retention policies to provide a safety margin before deletion.

  3. Test CQ queries manually first — Before enabling automatic execution, run your continuous query SQL manually to verify correct results.

  4. Combine with tiered storage — Use tiered storage to move data to cold storage before retention deletes it, keeping long-term archives at low cost.

  5. Monitor CQ execution — Check Arc logs for CQ execution results and errors. Failed CQ executions are logged at WARN level.

Next steps

On this page