# Changelog (/arc/changelog)
Release history for Arc.
## 26.04.1 [#26041]
Released: April 2026
Major performance release with DuckDB native Arrow query path (+59% JSON, +157% Arrow IPC throughput), automatic compaction deduplication, native Decimal128 type support, and 10 bug fixes.
### New features [#new-features]
#### Native Decimal128 type support [#native-decimal128-type-support]
Per-measurement configuration for exact decimal precision — financial data, scientific measurements, and cryptocurrency prices are stored as native Parquet DECIMAL type instead of float64. Supports float64, int64, and string input with up to 38 significant digits. DuckDB reads DECIMAL natively with no query changes.
Documentation: [Decimal Precision](/arc/guides/decimal-precision/)
#### DuckDB native Arrow query path [#duckdb-native-arrow-query-path]
Query results bypass `database/sql` row-by-row scanning entirely using DuckDB's native Arrow API. JSON queries are 59% faster (2.28M rows/sec), Arrow IPC queries are 157% faster (6.29M rows/sec).
#### Automatic compaction deduplication [#automatic-compaction-deduplication]
Compaction now automatically deduplicates rows with identical tag values and timestamps (last-write-wins). Tag columns are auto-detected from Parquet metadata — zero configuration required.
#### WAL dropped entries metric [#wal-dropped-entries-metric]
WAL buffer drops exposed as Prometheus counter (`arc_wal_dropped_entries_total`). Buffer size now configurable via `ARC_WAL_BUFFER_SIZE`.
#### Slow query logging [#slow-query-logging]
Configurable slow query detection with WARN-level logging and Prometheus counter (`arc_slow_queries_total`). Covers all query paths.
#### S3 path prefix support [#s3-path-prefix-support]
`ARC_STORAGE_S3_PREFIX` enables shared-bucket multi-tenant deployments with path-based isolation.
### Performance [#performance]
* **Typed JSON streaming** -- 2.3x faster serialization, constant memory usage regardless of result set size
* **Basekick-Labs/msgpack v6** -- 8.6% higher sustained throughput, 13% flatter degradation curve under GC pressure
* **Bulk UTF-8 pre-validation** -- Single-pass payload validation on Line Protocol path (58-84 GB/s on arm64)
### Bug fixes [#bug-fixes]
* **CQ scheduler reload on update** -- Updated continuous query definitions now take effect immediately without restart
* **Atomic CQ execution recording** -- SQLite transaction wraps execution recording and time window update
* **Database delete batch fallback** -- Falls back to individual file deletion when S3/Azure batch delete fails
* **S3 delete file rewrite streaming** -- Streams temp file to S3 instead of loading into memory (prevents OOM)
* **CQ scheduler graceful shutdown** -- Cancels in-flight queries on stop instead of waiting for 10-minute timeout
* **Compactor subprocess signal handling** -- Responds to SIGTERM/SIGINT for prompt cancellation
* **Streaming backup restore** -- Streams Parquet files through temp file instead of loading into memory
* **Local storage optimizations** -- WriteReader directory cache, context-aware file listing, batch delete error reporting
* **Token expiration display fix** -- Non-expiring tokens correctly display as "Never expires" instead of "Expired"
### Security [#security]
* **Admin authorization on mutating endpoints** -- `RequireAdmin` middleware on CQ, delete, retention, compaction, and scheduler endpoints
* **Hardened delete WHERE clause** -- Expanded forbidden keyword list blocks `UNION`, `SELECT`, `CREATE`, `COPY`, `ATTACH`, `LOAD`, `PRAGMA`, `CALL`, `SET`
* **Temp directory permissions** -- Changed from `0755` to `0700` (owner-only)
### Dependencies [#dependencies]
* **DuckDB 1.4.3 → 1.4.4** -- Parquet UTF-8 tolerance, Arrow string view pushdown fix, `mode()` use-after-free fix, secret secure clear
* **gRPC 1.79.1 → 1.79.3** -- Authorization bypass fix for malformed `:path` headers
* **Arrow Go v18.4.1 → v18.5.2** -- Large string Parquet writes fix, decompression regression fix, reduced GC pressure
***
## 26.03.1 [#26031]
Released: March 2026
Major quality release with backup/restore, 14 bug fixes, security hardening, and Go 1.26 upgrade.
### New features [#new-features-1]
#### Backup & restore API [#backup--restore-api]
Full backup and restore system via REST API. Backups capture parquet data files, SQLite metadata (auth, audit, MQTT config), and the `arc.toml` configuration file. Async operations with real-time progress tracking and selective restore.
Documentation: [Backup & Restore](/arc/operations/backup-restore/)
### Bug fixes [#bug-fixes-1]
* **Null handling in LP ingestion** -- Missing fields were stored as `0` instead of `NULL`. Introduced `TypedColumnBatch` with validity bitmaps throughout the ingestion pipeline.
* **Stale cache after compaction** -- Queries failed with 404 after compaction deleted old S3 parquet files. Added post-compaction cache invalidation for DuckDB caches and partition pruner. Extended to enterprise clustering with cross-node broadcast.
* **Descriptive query error messages** -- All query endpoints now return actual DuckDB errors instead of generic "Query execution failed".
* **time\_bucket / date\_trunc bucketing** -- GROUP BY queries returned one row per second instead of proper buckets. Fixed DuckDB float division (`/`) to integer division (`//`).
* **WAL recovery after flush failure** -- Recovery replayed already-flushed data causing duplication. Added `PurgeOlderThan` before recovery to limit replay window.
* **Self-adjusting flush timer** -- Replaced fixed-period ticker with adaptive timer. Worst-case flush delay drops from \~1.5x to \~1.0x of `max_buffer_age_ms`.
* **MQTT CleanSession default** -- Changed from `true` to `false` to preserve at-least-once delivery across reconnects.
* **Delete API partial failure** -- Now returns HTTP 207 with `failed_files` list instead of `success: true` when some files fail.
* **Replication observability** -- Added Prometheus metrics for dropped entries and sequence gaps.
* **Orphaned hot file cleanup** -- Reconciliation pass after tiering migration detects and removes orphaned hot copies.
* **Compaction manifest cleanup** -- Fixed three related bugs that could leave orphaned input files alongside compacted output.
* **Unified cache\_httpfs TTLs** -- Metadata/file handle TTLs now match `s3_cache_ttl_seconds`. Glob TTL fixed at 10s.
### Security [#security-1]
* **Database name validation** -- Added `isValidDatabaseName()` to LP write, CSV/Parquet import, and MsgPack handlers to prevent path traversal.
* **Backup restore permissions** -- Files now written with `0600` instead of `0644`.
### Performance [#performance-1]
* **Pooled gzip** -- CSV, Parquet, and TLE import endpoints now use pooled klauspost gzip (3-5x faster decompression).
* **Single-pass LP unescape** -- Byte scanner replaces three sequential `ReplaceAll` calls in the ingestion hot path.
* **Single-pass SQL regex** -- 7 separate regex passes consolidated into one alternation pattern.
### Infrastructure [#infrastructure]
* **Go 1.26** -- 10-40% GC overhead reduction (Green Tea GC), 30% faster cgo calls (DuckDB/SQLite), 2x faster `io.ReadAll`.
***
## 26.02.1 [#26021]
Released: February 2026
Major feature release adding MQTT integration, TLE satellite data ingestion, and bulk import endpoints for CSV, Parquet, and Line Protocol files.
### New features [#new-features-2]
#### MQTT integration [#mqtt-integration]
Native MQTT subscriber with API-driven subscription management. Connect to IoT devices, industrial sensors, and message brokers without middleware.
* Dynamic subscription management via REST API (create, update, delete, start/stop)
* Multiple simultaneous brokers with topic wildcards
* Auto-detection of JSON and MessagePack message formats
* \~6M records/sec with MessagePack columnar format
* TLS/SSL support with encrypted credentials at rest (AES-256-GCM)
* QoS 0, 1, and 2
Documentation: [MQTT Integration](/arc/integrations/mqtt/)
#### TLE (satellite orbital data) ingestion [#tle-satellite-orbital-data-ingestion]
Native support for ingesting satellite orbital data in the standard Two-Line Element format used by Space-Track.org, CelesTrak, and ground station pipelines.
* **Streaming ingestion** (`POST /api/v1/write/tle`) for continuous feeds and cron jobs
* **Bulk import** (`POST /api/v1/import/tle`) for historical backfill
* Pure Go parser with both 2-line and 3-line format support
* Derived orbital metrics (semi-major axis, period, apogee, perigee, orbit classification)
* \~3.5M records/sec via typed columnar fast path
Documentation: [TLE Integration](/arc/integrations/tle/)
#### Bulk import endpoints [#bulk-import-endpoints]
New REST API endpoints for importing data from files:
| Endpoint | Format | Description |
| ----------------------------- | ------------- | ---------------------------------------------------------------------------- |
| `POST /api/v1/import/csv` | CSV | Import CSV/TSV files with configurable delimiter, time column, and skip rows |
| `POST /api/v1/import/parquet` | Parquet | Import Parquet files directly (DuckDB native read) |
| `POST /api/v1/import/lp` | Line Protocol | Import InfluxDB LP exports with precision support |
All import endpoints support:
* Gzip auto-detection via magic bytes
* RBAC write permission checks
* 500 MB size limit
* Hourly data partitioning
Documentation: [CSV Import](/arc/data-import/csv/) | [Parquet Import](/arc/data-import/parquet/) | [Line Protocol Import](/arc/data-import/line-protocol/)
### Breaking changes [#breaking-changes]
None
***
## 26.01.2 [#26012]
Released: January 2026
Bugfix release addressing Azure Blob Storage backend issues and authentication configuration.
### Bug fixes [#bug-fixes-2]
#### Azure Blob Storage backend [#azure-blob-storage-backend]
* **Fix queries failing with Azure backend** - Queries were incorrectly using local filesystem paths (`./data/...`) instead of Azure blob paths (`azure://...`) when using Azure Blob Storage as the storage backend.
* **Fix compaction subprocess Azure authentication** - Compaction subprocess was failing with "DefaultAzureCredential: failed to acquire token" because credentials weren't being passed to the subprocess. Now passes `AZURE_STORAGE_KEY` via environment variable.
#### Configuration [#configuration]
* **Authentication enabled by default** - `auth.enabled` is now `true` by default in arc.toml for improved security out of the box.
### Upgrade notes [#upgrade-notes]
If you were relying on authentication being disabled by default, you'll need to explicitly set `auth.enabled = false` in your arc.toml.
***
## 26.01.1 [#26011]
Released: January 2026
### New features [#new-features-3]
#### Official Python SDK [#official-python-sdk]
The official Python SDK for Arc is now available on PyPI as `arc-tsdb-client`.
```bash
pip install arc-tsdb-client
# With DataFrame support
pip install arc-tsdb-client[pandas] # pandas
pip install arc-tsdb-client[polars] # polars
pip install arc-tsdb-client[all] # all optional dependencies
```
**Key features:**
* High-performance MessagePack columnar ingestion (10M+ records/sec)
* Query support with JSON, Arrow IPC, pandas, polars, and PyArrow responses
* Full async API with httpx
* Buffered writes with automatic batching (size and time thresholds)
* Complete management API (retention policies, continuous queries, delete operations, authentication)
Documentation: [Python SDK](/arc/sdks/python/)
#### Azure Blob Storage backend [#azure-blob-storage-backend-1]
Arc now supports Azure Blob Storage as a storage backend, enabling deployment on Microsoft Azure infrastructure.
**Configuration:**
```toml
[storage]
backend = "azure"
azure_container = "arc-data"
azure_account_name = "mystorageaccount"
azure_use_managed_identity = true
```
**Authentication options:**
* Connection string
* Account key
* SAS token
* Managed Identity (recommended for Azure deployments)
#### Native TLS/SSL support [#native-tlsssl-support]
Arc now supports native HTTPS/TLS without requiring a reverse proxy.
**Configuration:**
```toml
[server]
port = 443
tls_enabled = true
tls_cert_file = "/etc/letsencrypt/live/example.com/fullchain.pem"
tls_key_file = "/etc/letsencrypt/live/example.com/privkey.pem"
```
**Environment variables:** `ARC_SERVER_TLS_ENABLED`, `ARC_SERVER_TLS_CERT_FILE`, `ARC_SERVER_TLS_KEY_FILE`
#### Configurable ingestion concurrency [#configurable-ingestion-concurrency]
Ingestion concurrency settings are now configurable for high-concurrency deployments.
**Configuration:**
```toml
[ingest]
flush_workers = 32 # Async flush worker pool size
flush_queue_size = 200 # Pending flush queue capacity
shard_count = 64 # Buffer shards for lock distribution
```
Defaults scale dynamically with CPU cores.
#### Data-time partitioning [#data-time-partitioning]
Parquet files are now organized by the data's timestamp instead of ingestion time, enabling proper backfill of historical data.
**Key features:**
* Historical data lands in correct time-based partitions
* Batches spanning multiple hours are automatically split into separate files
* Data is sorted by timestamp within each Parquet file
* Enables accurate partition pruning for time-range queries
Documentation: [Data-Time Partitioning](/arc/advanced/data-time-partitioning/)
*Contributed by [@schotime](https://github.com/schotime)*
#### Compaction API triggers [#compaction-api-triggers]
Hourly and daily compaction can now be triggered manually via API.
| Method | Endpoint | Description |
| ------ | --------------------------- | ------------------------- |
| `POST` | `/api/v1/compaction/hourly` | Trigger hourly compaction |
| `POST` | `/api/v1/compaction/daily` | Trigger daily compaction |
**Configuration:**
```toml
[compaction]
hourly_schedule = "0 * * * *" # Every hour
daily_schedule = "0 2 * * *" # Daily at 2 AM
```
*Contributed by [@schotime](https://github.com/schotime)*
#### Configurable max payload size [#configurable-max-payload-size]
The maximum request payload size is now configurable, with the default increased from 100MB to 1GB.
```toml
[server]
max_payload_size = "2GB"
```
Supports human-readable units: B, KB, MB, GB.
#### Database management API [#database-management-api]
New REST API endpoints for managing databases programmatically.
| Method | Endpoint | Description |
| -------- | -------------------------------------- | --------------------- |
| `GET` | `/api/v1/databases` | List all databases |
| `POST` | `/api/v1/databases` | Create a new database |
| `GET` | `/api/v1/databases/:name` | Get database info |
| `GET` | `/api/v1/databases/:name/measurements` | List measurements |
| `DELETE` | `/api/v1/databases/:name` | Delete a database |
#### DuckDB S3 query support [#duckdb-s3-query-support]
Arc now configures the DuckDB httpfs extension automatically, enabling direct queries against Parquet files stored in S3.
### Improvements [#improvements]
#### Ingestion pipeline [#ingestion-pipeline]
* **Zstd compression support** - 9.57M rec/sec with only 5% overhead vs uncompressed. Auto-detected via magic bytes.
* **O(n log n) column sorting** - Replaced O(n²) bubble sort with `sort.Slice()` for schema inference.
* **Single-pass timestamp normalization** - Reduced from 2-3 passes to single pass.
* **Result:** 7% throughput improvement (9.47M → 10.1M rec/s), 63% p50 latency reduction, 84% p99 latency reduction.
#### Authentication performance [#authentication-performance]
* **Token lookup index** - O(1) token lookup instead of O(n) full table scan.
* **Atomic cache counters** - Eliminated lock contention on cache hit/miss tracking.
* **Auth metrics integration** - Prometheus metrics for authentication requests and cache performance.
#### Query performance [#query-performance]
* **Arrow IPC throughput boost** - 5.2M rows/sec (80% improvement from 2.88M rows/sec).
* **SQL transform caching** - 60-second TTL cache for SQL-to-storage-path transformations (49-104x speedup on cache hits).
* **Partition path caching** - 60-second TTL cache saving 50-100ms per recurring query.
* **Glob result caching** - 30-second TTL cache saving 5-10ms per query for large partition sets.
#### Storage roundtrip optimizations [#storage-roundtrip-optimizations]
* Fixed N+1 query pattern in database listing (90% reduction for 20 databases).
* Optimized database existence checks via direct marker file lookup.
* Batch row counting in delete handler.
### Bug fixes [#bug-fixes-3]
* Fixed DuckDB S3 credentials not persisting across connection pool
* Fixed compaction subprocess failing with large file counts
* **Fixed CTE (Common Table Expressions) support** - CTE names are now properly recognized as virtual table references
* **Fixed JOIN clause table resolution** - `JOIN database.table` syntax now correctly converts to storage paths
* **Fixed string literal corruption in queries** - String literals containing SQL keywords are no longer incorrectly rewritten
* **Fixed SQL comment handling** - Comments containing table references are no longer incorrectly converted
* **Added LATERAL JOIN support** - All LATERAL join variants now work correctly
* **Fixed UTC consistency in path generation** - Storage paths now consistently use UTC time
### Performance [#performance-2]
Tested at **10.1M records/second** with:
* p50 latency: 3.09ms
* p95 latency: 5.16ms
* p99 latency: 6.73ms
* p999 latency: 9.29ms
### Breaking changes [#breaking-changes-1]
None
### Upgrade notes [#upgrade-notes-1]
1. **S3 credentials** - For S3 storage backend, credentials are now also passed to DuckDB for httpfs queries. Ensure AWS credentials are configured.
2. **Azure backend** - New storage backend option. No changes required for existing deployments.
3. **Token prefix migration** - Existing API tokens are automatically migrated on startup. No action required.
### Contributors [#contributors]
* [@schotime](https://github.com/schotime) (Adam Schroder) - Data-time partitioning, compaction API triggers, UTC fixes
### Dependencies [#dependencies-1]
* Added `github.com/Azure/azure-sdk-for-go/sdk/storage/azblob` for Azure Blob Storage
* Added `github.com/Azure/azure-sdk-for-go/sdk/azidentity` for Azure authentication
***
## 25.12.1 [#25121]
Released: December 2025
**Major Release: Complete rewrite from Python to Go**
### Migration highlights [#migration-highlights]
This release marks the complete migration from Python to Go, delivering:
#### Performance improvements [#performance-improvements]
* **9.47M records/sec** MessagePack ingestion (125% faster than Python's 4.21M)
* **1.92M records/sec** Line Protocol ingestion (76% faster than Python's 1.09M)
* **2.88M rows/sec** Arrow query throughput
#### Reliability [#reliability]
* **Memory stable** - No memory leaks (Python leaked 372MB per 500 queries)
* **Single binary** - No Python dependencies, pip, or virtual environments
* **Type-safe** - Strong typing catches bugs at compile time
#### Full feature parity [#full-feature-parity]
* Authentication (user/password)
* Automatic Compaction (Parquet optimization)
* Write-Ahead Log (WAL for durability)
* Retention Policies (automatic data expiration)
* Continuous Queries (real-time aggregations)
* Delete API (selective data removal)
* S3/MinIO storage backend
* Arrow IPC query responses
### Breaking changes [#breaking-changes-2]
* **Python version** - The Python implementation is preserved in the `python-legacy` branch
* **Configuration** - TOML config format (unchanged, but verify your arc.toml)
### Upgrading from Python [#upgrading-from-python]
1. Stop existing Arc service
2. Backup your data directory
3. Install the new Go binary (same config format)
4. Start Arc - data is automatically migrated
***
## 25.11.1 [#25111]
Released: November 2025
**Initial public release**
One database for metrics, logs, traces, and events. Query all your observability data with SQL. Built on DuckDB + Parquet.
### Features [#features]
#### High-performance ingestion [#high-performance-ingestion]
* **6.57M records/sec unified** - Ingest metrics, logs, traces, and events simultaneously through one endpoint
* **MessagePack columnar protocol** - Zero-copy ingestion optimized for throughput
* **InfluxDB Line Protocol** - 240K records/sec for Telegraf compatibility and easy migration
#### Query and analytics [#query-and-analytics]
* **DuckDB SQL engine** - Full analytical SQL with window functions, CTEs, joins, and aggregations
* **Cross-database queries** - Join metrics, logs, and traces in a single SQL query
* **Query caching** - Configurable result caching for repeated analytical queries
* **Apache Arrow format** - Zero-copy columnar data transfer for Pandas/Polars pipelines
#### Storage and scalability [#storage-and-scalability]
* **Columnar Parquet storage** - 3-5x compression ratios, optimized for analytical queries
* **Flexible backends** - Local filesystem, MinIO, AWS S3/R2, Google Cloud Storage, or any S3-compatible storage
* **Multi-database architecture** - Organize data by environment, tenant, or application with database namespaces
* **Automatic compaction** - Merges small files into optimized 512MB files for 10-50x faster queries
#### Data management [#data-management]
* **Retention policies** - Time-based data lifecycle management with automatic cleanup
* **Continuous queries** - Downsampling and materialized views for long-term data aggregation
* **GDPR-compliant deletion** - Precise deletion with zero overhead on writes/queries
* **Write-Ahead Log (WAL)** - Optional durability feature for zero data loss
#### Integrations and tools [#integrations-and-tools]
* **VSCode Extension** - Full-featured database manager with query editor, notebooks, CSV import, and alerting
* **Apache Superset** - Native dialect for BI dashboards and visualizations
* **Grafana** - Native Data Source
* **Prometheus** - Ingest via Telegraf bridge
* **OpenTelemetry** - Ingest via OTEL Collector
#### Operations and monitoring [#operations-and-monitoring]
* **Health checks** - `/health` and `/ready` endpoints for orchestration
* **Prometheus metrics** - Export operational metrics for monitoring
* **Authentication** - Token-based API authentication with cache for performance
* **Production ready** - Docker, native deployment, and systemd service management
### Performance [#performance-3]
**Unified Ingestion Benchmark** (Apple M3 Max, 14 cores):
* Metrics: 2.91M/sec
* Logs: 1.55M/sec
* Traces: 1.50M/sec
* Events: 1.54M/sec
* **Total: 6.57M records/sec** (all data types simultaneously)
**ClickBench Results** (AWS c6a.4xlarge, 100M rows):
* Cold run: 120.25s
* Warm run: 35.70s
* 12.4x faster than TimescaleDB
* 1.2x faster than QuestDB (Combined and Cold Run)
# Getting Started (/arc/getting-started)
Get Arc up and running in 5 minutes.
## Prerequisites [#prerequisites]
* 4GB RAM minimum, 8GB+ recommended
* Docker, Kubernetes, or Linux (Debian/RHEL)
## Quick start [#quick-start]
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
```bash
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
helm install arc https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc-${LATEST}.tgz
kubectl port-forward svc/arc 8000:8000
```
```bash
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc_${LATEST}_amd64.deb
sudo dpkg -i arc_${LATEST}_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
```bash
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc-${LATEST}-1.x86_64.rpm
sudo rpm -i arc-${LATEST}-1.x86_64.rpm
sudo systemctl enable arc && sudo systemctl start arc
```
**Verify it's running:**
```bash
curl http://localhost:8000/health
```
## Get your admin token [#get-your-admin-token]
Arc generates an admin token on first startup. **Copy it immediately - you won't see it again!**
```bash
docker logs arc 2>&1 | grep -i "admin"
```
```bash
kubectl logs -l app=arc | grep -i "admin"
```
```bash
sudo journalctl -u arc | grep -i "admin"
```
You'll see:
```bash
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Initial admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save it:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
## Write data [#write-data]
```python
import msgpack
import requests
from datetime import datetime
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
token = os.getenv("ARC_TOKEN")
data = {
"m": "cpu",
"columns": {
# time must be a numeric Unix epoch (microseconds recommended).
# Strings (e.g. "2024-01-01T00:00:00Z") and nulls are rejected.
"time": [int(datetime.now().timestamp() * 1_000_000)],
"host": ["server01"],
"usage_idle": [95.0],
"usage_user": [3.2]
}
}
response = requests.post(
"http://localhost:8000/api/v1/write/msgpack",
headers={
"Authorization": f"Bearer {token}",
"Content-Type": "application/msgpack",
"x-arc-database": "default"
},
data=msgpack.packb(data)
)
```
```bash
# InfluxDB 1.x compatible endpoint
curl -X POST "http://localhost:8000/write?db=default&p=$ARC_TOKEN" \
--data-binary "cpu,host=server01 usage_idle=95.0,usage_user=3.2"
# Or with Authorization header
curl -X POST "http://localhost:8000/write?db=default" \
-H "Authorization: Bearer $ARC_TOKEN" \
--data-binary "cpu,host=server01 usage_idle=95.0,usage_user=3.2"
```
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
client.write.write_columnar(
measurement="cpu",
columns={
# numeric epoch in microseconds (2024-01-01 00:00:00 UTC)
"time": [1704067200000000],
"host": ["server01"],
"usage_idle": [95.0],
},
)
```
## Query data [#query-data]
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM default.cpu LIMIT 10", "format": "json"}'
```
```python
import requests
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
token = os.getenv("ARC_TOKEN")
response = requests.post(
"http://localhost:8000/api/v1/query",
headers={"Authorization": f"Bearer {token}"},
json={"sql": "SELECT * FROM default.cpu LIMIT 10", "format": "json"}
)
print(response.json())
```
```python
import requests
import pyarrow as pa
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
token = os.getenv("ARC_TOKEN")
response = requests.post(
"http://localhost:8000/api/v1/query/arrow",
headers={"Authorization": f"Bearer {token}"},
json={"sql": "SELECT * FROM default.cpu LIMIT 10000"}
)
reader = pa.ipc.open_stream(response.content)
df = reader.read_all().to_pandas()
print(df.head())
```
## Next steps [#next-steps]
* **[Python SDK](/arc/sdks/python/)** - Official client with DataFrame support
* **[Telegraf Integration](/arc/integrations/telegraf/)** - Collect system metrics
* **[Apache Superset](/arc/integrations/superset/)** - Build dashboards
* **[Configuration](/arc/configuration/overview/)** - Tune Arc for your workload
## Troubleshooting [#troubleshooting]
```bash
docker logs arc
```
```bash
kubectl logs -l app=arc
kubectl describe pod -l app=arc
```
```bash
sudo journalctl -u arc -n 50
sudo systemctl status arc
```
### Common issues [#common-issues]
**Authentication errors**: Make sure `ARC_TOKEN` is set and included in headers.
**No data returned**: Data may not be flushed yet. Force flush:
```bash
curl -X POST http://localhost:8000/api/v1/write/line-protocol/flush \
-H "Authorization: Bearer $ARC_TOKEN"
```
## Need help? [#need-help]
* [Discord Community](https://discord.gg/nxnWfUxsdm)
* [GitHub Issues](https://github.com/basekick-labs/arc/issues)
# Arc OSS (/arc)
**Open, SQL-native time-series database**
Arc is a SQL-native time-series database that stores data as Parquet with flexible storage backends. Use it for analytics, observability, AI/ML, IoT, and log management. High-throughput columnar ingestion, fast analytical queries. Single Go binary. S3/Azure native. No vendor lock-in. AGPL-3.0.
## Key features [#key-features]
* **High-throughput ingestion**: Columnar MessagePack write path
* **Fast Analytical Queries**: Vectorized columnar execution with full SQL support
* **Flexible Storage**: Local filesystem, MinIO, AWS S3, Azure Blob Storage
* **Multi-Database Architecture**: Organize data by environment, tenant, or application
* **Automatic Compaction**: Tiered (hourly/daily) file merging to keep queries fast
* **Optional WAL**: Zero data loss with Write-Ahead Log
* **Data Lifecycle**: Retention policies, continuous queries, GDPR-compliant delete
* **Production Ready**: Prometheus metrics, structured logging, graceful shutdown
* **MQTT Integration**: Direct MQTT broker subscription for streaming data
* **Python SDK**: Native Python client with DataFrame support (Pandas, Polars, PyArrow)
* **Bulk Import**: CSV and Parquet import with auto-partitioning
* **Native TLS/HTTPS**: Built-in TLS support, no reverse proxy needed
## Why Arc? [#why-arc]
**The Problem**: Modern data workloads generate massive volumes at scale:
* **Product Analytics**: Billions of events from user interactions, funnels, and sessions
* **Observability**: Metrics, logs, and traces from distributed systems
* **AI/ML Pipelines**: Feature stores, training data, and model inference logs
* **IoT & Industrial**: Sensor telemetry from factories, vehicles, and infrastructure
* **Log Management**: Application logs, security events, and audit trails
Traditional databases can't keep up. They're slow, expensive, and lock your data in proprietary formats.
**Arc solves this: high-throughput ingestion, sub-second queries on billions of rows, portable Parquet files you own.**
```sql
-- Analyze page views and session funnels
SELECT
time_bucket(INTERVAL '1 hour', time) AS bucket,
page_url,
COUNT(DISTINCT session_id) AS unique_sessions,
COUNT(*) AS page_views,
AVG(time_on_page_ms) AS avg_time_on_page,
SUM(CASE WHEN converted THEN 1 ELSE 0 END)::FLOAT / COUNT(*) AS conversion_rate
FROM data.page_views
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY bucket, page_url
ORDER BY page_views DESC;
```
```sql
-- Analyze error rates and patterns
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
service_name,
level,
COUNT(*) AS log_count,
COUNT(*) FILTER (WHERE level = 'ERROR') AS error_count,
COUNT(DISTINCT trace_id) AS affected_traces
FROM data.app_logs
WHERE time > NOW() - INTERVAL '1 hour'
AND level IN ('ERROR', 'WARN')
GROUP BY bucket, service_name, level
ORDER BY error_count DESC;
```
```sql
-- Analyze equipment anomalies across facilities
SELECT
device_id,
facility_name,
AVG(temperature) OVER (
PARTITION BY device_id
ORDER BY timestamp
ROWS BETWEEN 10 PRECEDING AND CURRENT ROW
) as temp_moving_avg,
MAX(pressure) as peak_pressure,
STDDEV(vibration) as vibration_variance
FROM data.iot_sensors
WHERE timestamp > NOW() - INTERVAL '24 hours'
AND facility_id IN ('mining_site_42', 'plant_7')
GROUP BY device_id, facility_name, timestamp
HAVING MAX(pressure) > 850 OR STDDEV(vibration) > 2.5;
```
**Standard analytical SQL. Window functions, CTEs, joins. No proprietary query language.**
## Quick example [#quick-example]
```python
import os
import msgpack
import requests
from datetime import datetime
ARC_TOKEN = os.environ["ARC_TOKEN"]
# COLUMNAR FORMAT (RECOMMENDED)
# All data organized as columns (arrays), not rows
data = {
"m": "cpu", # measurement name
"columns": { # columnar data structure
# time is a numeric Unix epoch in microseconds (strings/nulls rejected)
"time": [
int(datetime.now().timestamp() * 1_000_000),
int(datetime.now().timestamp() * 1_000_000) + 1_000_000,
int(datetime.now().timestamp() * 1_000_000) + 2_000_000
],
"host": ["server01", "server02", "server03"],
"region": ["us-east", "us-west", "eu-central"],
"datacenter": ["aws", "gcp", "azure"],
"usage_idle": [95.0, 85.0, 92.0],
"usage_user": [3.2, 10.5, 5.8],
"usage_system": [1.8, 4.5, 2.2]
}
}
# Send columnar data
response = requests.post(
"http://localhost:8000/api/v1/write/msgpack",
headers={
"Authorization": f"Bearer {ARC_TOKEN}",
"Content-Type": "application/msgpack",
"x-arc-database": "default" # Optional: specify database
},
data=msgpack.packb(data)
)
# Check response (returns 204 No Content on success)
if response.status_code == 204:
print(f"Successfully wrote {len(data['columns']['time'])} records!")
else:
print(f"Error {response.status_code}: {response.text}")
# Query data
response = requests.post(
"http://localhost:8000/api/v1/query",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"sql": "SELECT * FROM default.cpu LIMIT 10", "format": "json"}
)
```
{/* TODO(diagram): replace the ASCII box-flow below with a real diagram of the write path (client -> API -> buffer -> Parquet -> object storage) and the read path (the query engine reading Parquet in place). The separation of compute and storage is Arc's central design claim and deserves better than monospace arrows. */}
## Architecture [#architecture]
```text
Client → Arc API → Buffer → Parquet → Storage (S3/MinIO/Azure/Local)
↓
Query Engine
```
Arc separates compute and storage, allowing you to scale them independently. Data is stored as Parquet files on object storage, queried directly by Arc's vectorized columnar engine.
## Performance [#performance]
Arc is built for high-throughput ingestion and sub-second analytical queries over
billions of rows, with Parquet's columnar compression keeping stored data far smaller
than row-oriented or JSON representations.
Published ClickBench results, per-database comparisons, and the hardware each run used
are on the [benchmarks page](/arc/performance/benchmarks/).
## Arc Enterprise [#arc-enterprise]
Need clustering, RBAC, tiered storage, audit logging, or automated scheduling? [Arc Enterprise](/arc-enterprise/) extends Arc with production-grade features for scale, security, and compliance. Same binary, same performance — add a license key and enable the features you need.
## Next steps [#next-steps]
* [Getting Started](/arc/getting-started/) - Install and run Arc in 5 minutes
* [Installation Guide](/arc/installation/docker/) - Docker, native packages, and source
* [Arc Enterprise](/arc-enterprise/) - Enterprise features for production at scale
* [GitHub Repository](https://github.com/basekick-labs/arc) - Star us on GitHub
## Support [#support]
* [Discord Community](https://discord.gg/nxnWfUxsdm)
* [GitHub Issues](https://github.com/basekick-labs/arc/issues)
* Enterprise: [enterprise@basekick.net](mailto:enterprise@basekick.net)
# Getting Started (/arc-enterprise/getting-started)
Get Arc up and running in 5 minutes.
## Prerequisites [#prerequisites]
* 4GB RAM minimum, 8GB+ recommended
* Docker, Kubernetes, or Linux (Debian/RHEL)
## License key setup [#license-key-setup]
Before enabling enterprise features, configure your license key:
A license key is required for enterprise features (clustering, RBAC, tiered storage, audit logging, query governance, query management, automated scheduling). Core features like ingestion, querying, and compaction work without a license.
Add to `arc.toml`:
```toml
[license]
key = "ARC-XXXX-XXXX-XXXX-XXXX"
```
Or set the environment variable:
```bash
export ARC_LICENSE_KEY="ARC-XXXX-XXXX-XXXX-XXXX"
```
On 26.09.1+ you can license Arc with an **offline license file** instead of a key — no route to the activation server needed. Set `file_path` under `[license]` (or `ARC_LICENSE_FILE_PATH`). See [Configuration Overview → Air-gapped: offline license file](/arc-enterprise/configuration/overview/#air-gapped-offline-license-file-26091).
## Quick start [#quick-start]
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
```bash
helm install arc https://github.com/basekick-labs/arc/releases/latest/download/arc-26.09.1.tgz
kubectl port-forward svc/arc 8000:8000
```
```bash
wget https://github.com/basekick-labs/arc/releases/latest/download/arc_26.09.1_amd64.deb
sudo dpkg -i arc_26.09.1_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
```bash
wget https://github.com/basekick-labs/arc/releases/latest/download/arc-26.09.1-1.x86_64.rpm
sudo rpm -i arc-26.09.1-1.x86_64.rpm
sudo systemctl enable arc && sudo systemctl start arc
```
**Verify it's running:**
```bash
curl http://localhost:8000/health
```
## Get your admin token [#get-your-admin-token]
Arc generates an admin token on first startup. **Copy it immediately - you won't see it again!**
```bash
docker logs arc 2>&1 | grep -i "admin"
```
```bash
kubectl logs -l app=arc | grep -i "admin"
```
```bash
sudo journalctl -u arc | grep -i "admin"
```
You'll see:
```text
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Initial admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save it:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
## Write data [#write-data]
```python
import msgpack
import requests
from datetime import datetime
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
token = os.getenv("ARC_TOKEN")
data = {
"m": "cpu",
"columns": {
"time": [int(datetime.now().timestamp() * 1000)],
"host": ["server01"],
"usage_idle": [95.0],
"usage_user": [3.2]
}
}
response = requests.post(
"http://localhost:8000/api/v1/write/msgpack",
headers={
"Authorization": f"Bearer {token}",
"Content-Type": "application/msgpack",
"x-arc-database": "default"
},
data=msgpack.packb(data)
)
```
```bash
# InfluxDB 1.x compatible endpoint
curl -X POST "http://localhost:8000/write?db=default&p=$ARC_TOKEN" \
--data-binary "cpu,host=server01 usage_idle=95.0,usage_user=3.2"
# Or with Authorization header
curl -X POST "http://localhost:8000/write?db=default" \
-H "Authorization: Bearer $ARC_TOKEN" \
--data-binary "cpu,host=server01 usage_idle=95.0,usage_user=3.2"
```
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
client.write.write_columnar(
measurement="cpu",
columns={
"time": [1704067200000],
"host": ["server01"],
"usage_idle": [95.0],
},
)
```
## Query data [#query-data]
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM default.cpu LIMIT 10", "format": "json"}'
```
```python
import requests
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
token = os.getenv("ARC_TOKEN")
response = requests.post(
"http://localhost:8000/api/v1/query",
headers={"Authorization": f"Bearer {token}"},
json={"sql": "SELECT * FROM default.cpu LIMIT 10", "format": "json"}
)
print(response.json())
```
```python
import requests
import pyarrow as pa
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
token = os.getenv("ARC_TOKEN")
response = requests.post(
"http://localhost:8000/api/v1/query/arrow",
headers={"Authorization": f"Bearer {token}"},
json={"sql": "SELECT * FROM default.cpu LIMIT 10000"}
)
reader = pa.ipc.open_stream(response.content)
df = reader.read_all().to_pandas()
print(df.head())
```
## Next steps [#next-steps]
* **[Python SDK](/arc-enterprise/sdks/python/)** - Official client with DataFrame support
* **[Telegraf Integration](/arc-enterprise/integrations/telegraf/)** - Collect system metrics
* **[Apache Superset](/arc-enterprise/integrations/superset/)** - Build dashboards
* **[Configuration](/arc-enterprise/configuration/overview/)** - Tune Arc for your workload
## Troubleshooting [#troubleshooting]
```bash
docker logs arc
```
```bash
kubectl logs -l app=arc
kubectl describe pod -l app=arc
```
```bash
sudo journalctl -u arc -n 50
sudo systemctl status arc
```
### Common issues [#common-issues]
**Authentication errors**: Make sure `ARC_TOKEN` is set and included in headers.
**No data returned**: Data may not be flushed yet. Force flush:
```bash
curl -X POST http://localhost:8000/api/v1/write/line-protocol/flush \
-H "Authorization: Bearer $ARC_TOKEN"
```
## Need help? [#need-help]
* [Discord Community](https://discord.gg/nxnWfUxsdm)
* [GitHub Issues](https://github.com/basekick-labs/arc/issues)
# Arc Enterprise (/arc-enterprise)
**Production-grade time-series database infrastructure for scale, security, and compliance.**
Arc Enterprise extends the [Arc](/arc/) time-series database with enterprise features designed for production deployments. Same binary, same performance — with clustering, access control, tiered storage, audit logging, and more.
## OSS vs Enterprise [#oss-vs-enterprise]
| Capability | Arc OSS | Arc Enterprise |
| -------------------------------------------------------------- | ------- | -------------- |
| **Deployment & Scale** | | |
| Single-node deployment | Yes | Yes |
| Clustering (multi-node) | - | Yes |
| Role separation (writer / reader / compactor) | - | Yes |
| Automatic failover (writer + compactor) | - | Yes |
| Shared-storage deployment (S3 / MinIO / Azure) | Yes | Yes |
| Local-storage deployment with peer replication | - | Yes |
| Heartbeat-based cluster health detection | - | Yes |
| **Security & Access** | | |
| Token-based authentication | Yes | Yes |
| Organizations, teams, and RBAC | - | Yes |
| Measurement-level permissions | - | Yes |
| Cluster TLS encryption + shared secret auth | - | Yes |
| Audit logging | - | Yes |
| **Data Lifecycle** | | |
| Continuous queries (manual execution) | Yes | Yes |
| Retention policies (manual execution) | Yes | Yes |
| Automatic scheduling (continuous queries + retention policies) | - | Yes |
| Auto-aggregation | - | Yes |
| Hot/cold tiered storage with per-database policies | - | Yes |
| Backup & restore | - | Yes |
| **Observability & Operations** | | |
| Prometheus metrics | Yes | Yes |
| Active query management | - | Yes |
| Query governance (rate limits, quotas, row limits) | - | Yes |
| SLA monitoring | - | Yes |
| **Integrations** | | |
| MQTT subscription management | Yes | Yes |
| **Storage** | | |
| S3 / Azure / MinIO storage | Yes | Yes |
## Getting started [#getting-started]
Arc Enterprise uses the same binary as Arc OSS. Enable enterprise features by adding your license key:
```toml
# arc.toml
[license]
key = "ARC-XXXX-XXXX-XXXX-XXXX"
```
Or via environment variable:
```bash
export ARC_LICENSE_KEY="ARC-XXXX-XXXX-XXXX-XXXX"
```
On startup, Arc validates your license and enables the features included in your plan. No data migration is required — your existing databases, measurements, and configurations are fully preserved.
Upgrading from Arc OSS to Enterprise requires no data migration. Add your license key, enable the features you need, and restart. Your existing data and configuration remain intact.
## Features [#features]
## Configuration example [#configuration-example]
A complete enterprise configuration combining multiple features:
```toml
# arc.toml — Enterprise deployment example
[license]
key = "ARC-XXXX-XXXX-XXXX-XXXX"
[server]
port = 8000
[storage]
backend = "s3"
s3_bucket = "arc-production"
s3_region = "us-east-1"
[auth]
enabled = true
[cluster]
enabled = true
node_id = "writer-01"
role = "writer"
cluster_name = "production"
seeds = ["10.0.1.10:9000", "10.0.1.11:9000"]
[tiered_storage]
enabled = true
default_hot_max_age_days = 30
[tiered_storage.cold]
enabled = true
backend = "s3"
s3_bucket = "arc-archive"
s3_region = "us-east-1"
s3_storage_class = "GLACIER"
[audit_log]
enabled = true
retention_days = 90
[governance]
enabled = true
default_rate_limit_per_min = 60
default_max_queries_per_hour = 500
[query_management]
enabled = true
```
## Contact [#contact]
To get started with Arc Enterprise, request a trial, or learn more about licensing:
* **Email**: [enterprise@basekick.net](mailto:enterprise@basekick.net)
* **Discord**: [Join our community](https://discord.gg/nxnWfUxsdm)
# Arc Launchpad (/launchpad)
**A self-hosted web UI for Arc. Query and operate your instances from the browser.**
Arc Launchpad is a browser-based control panel for [Arc](/arc/), the open, SQL-native time-series database. It connects to one or more Arc instances you already run and gives you a UI to run SQL, explore schemas, browse logs, manage tokens and retention policies, set up alerts and continuous queries, manage MQTT ingestion, and invite teammates into shared organizations.
## What Launchpad is (and isn't) [#what-launchpad-is-and-isnt]
Launchpad connects to Arc instances you **already run**. You point it at an Arc endpoint (URL + admin token) and it gives you a UI to both *query* and *operate* that instance.
It does **not** provision or host databases. It doesn't spin up servers, allocate storage, or run Arc for you. You bring your own Arc. Once connected, Launchpad drives Arc's admin API to manage tokens, retention, alerts, continuous queries, and MQTT ingestion on your behalf.
Launchpad stores only your **connection records** and your **accounts/teams** in a local SQLite database. It never copies your data out of Arc: every request goes straight to your instance through a built-in proxy.
## Features [#features]
* **SQL console**: schema explorer, query history, multi-statement scripts, and result export (CSV/JSON/Markdown/chart)
* **Log viewer**: for the logs you store in Arc, with pattern detection and trace extraction
* **Monitoring**: Arc's own self-observability: ingestion, queries, and internal metrics
* **Retention policies**: automatic data-retention rules with a dry-run preview
* **Continuous queries**: rollups and downsampling
* **Alerts**: threshold alerts evaluated by Launchpad, with webhook notifications
* **MQTT ingestion**: manage broker subscriptions that ingest topics into Arc
* **Token management**: full lifecycle for Arc API tokens
* **Organizations & teams**: invite users and assign roles
* **Local auth**: email/password with optional MFA (TOTP) and passkeys (WebAuthn)
## How it fits together [#how-it-fits-together]
```text
┌──────────┐ ┌──────────────┐ ┌──────────────┐
│ Browser │ ─────▶ │ Launchpad │ ─────▶ │ Arc server │
│ (you) │ │ (proxy + UI) │ │ (your data) │
└──────────┘ └──────────────┘ └──────────────┘
│
▼
┌──────────┐
│ SQLite │ accounts, teams, connection records
└──────────┘
```
Launchpad holds accounts, teams, and the connection records (endpoint + token) in SQLite. All queries and admin actions are proxied to the Arc server you registered; your analytical data stays in Arc.
## Get started [#get-started]
* **[Installation](/launchpad/getting-started/installation/)**: Docker Compose, standalone Docker, Helm, or from source
* **[First-run setup](/launchpad/getting-started/first-run-setup/)**: create the admin account
* **[Connecting to Arc](/launchpad/getting-started/connecting-to-arc/)**: register your first Arc server
## Links [#links]
* **Source:** [github.com/Basekick-Labs/launchpad](https://github.com/Basekick-Labs/launchpad)
* **License:** Apache-2.0
* **Issues & requests:** [GitHub issues](https://github.com/Basekick-Labs/launchpad/issues)
# Query Caching (/arc/advanced/caching)
Arc implements multiple caching layers to optimize query performance, particularly for dashboard and monitoring use cases where the same queries are executed repeatedly.
## Cache layers [#cache-layers]
Arc uses three complementary caches that work together:
| Cache | TTL | Purpose | Savings |
| -------------------- | --- | -------------------------------------- | --------------------------------- |
| SQL Transform Cache | 60s | Caches SQL-to-storage-path conversions | Avoids re-parsing per query |
| Partition Path Cache | 60s | Caches `OptimizeTablePath()` results | Avoids repeated path resolution |
| Glob Cache | 30s | Caches filesystem glob results | Avoids repeated directory listing |
## SQL transform cache [#sql-transform-cache]
When you execute a query like:
```sql
SELECT * FROM mydb.cpu WHERE time > now() - INTERVAL '1 hour'
```
Arc converts the table reference `mydb.cpu` to a `read_parquet()` call:
```sql
SELECT * FROM read_parquet('./data/mydb/cpu/**/*.parquet') WHERE time > now() - INTERVAL '1 hour'
```
This string transformation uses regex matching and happens on every query. The SQL Transform Cache stores the result so repeated queries skip this processing.
### Performance impact [#performance-impact]
| Scenario | Time | Speedup |
| --------------------------- | ------- | ------- |
| Without cache (first query) | 13-37μs | - |
| With cache (repeated query) | \~300ns | 49-104x |
### When it helps [#when-it-helps]
The SQL Transform Cache is most beneficial for:
* **Dashboard refresh**: Same queries every 30s-5min
* **Monitoring alerts**: Repeated threshold checks
* **API integrations**: Clients polling the same metrics
* **Multi-user dashboards**: Shared queries across users
### Cache behavior [#cache-behavior]
* **Key**: SHA256 hash of the raw SQL string
* **TTL**: 60 seconds (matches partition cache)
* **Max entries**: 10,000 queries
* **Eviction**: Expired entries removed first, then oldest
## Partition path cache [#partition-path-cache]
After SQL transformation, Arc optimizes the storage path by applying time-based partition pruning. This cache stores the optimized paths.
### Example [#example]
Query with time filter:
```sql
SELECT * FROM mydb.cpu WHERE time > 1704067200000000
```
Without cache: Scans partition metadata to find relevant directories.
With cache: Returns pre-computed path like `./data/mydb/cpu/2024/01/**/*.parquet`.
### Performance impact [#performance-impact-1]
Saves 50-100ms per query on large datasets with many partitions.
## Glob cache [#glob-cache]
After determining the partition path, Arc uses filesystem globs to find matching Parquet files. The Glob Cache stores these file listings.
### Performance impact [#performance-impact-2]
Saves 5-10ms per query by avoiding repeated filesystem operations.
## Cache statistics [#cache-statistics]
Monitor cache performance via the pruner stats:
```go
stats := pruner.GetAllCacheStats()
```
Returns:
```json
{
"partition_cache": {
"size": 150,
"hits": 12847,
"misses": 423,
"hit_rate_percent": 96.8
},
"glob_cache": {
"size": 89,
"hits": 8234,
"misses": 312,
"hit_rate_percent": 96.3
}
}
```
## Best practices [#best-practices]
### 1. Use consistent query strings [#1-use-consistent-query-strings]
Cache keys are based on exact SQL text. These are different cache entries:
```sql
SELECT * FROM mydb.cpu WHERE time > 1704067200000000
SELECT * FROM mydb.cpu WHERE time > 1704067200000000 -- extra space
select * from mydb.cpu where time > 1704067200000000 -- lowercase
```
Normalize your queries for better cache hit rates.
### 2. Use parameterized time ranges [#2-use-parameterized-time-ranges]
For dashboard queries, use relative time:
```sql
-- Good: Same query text on every refresh
SELECT * FROM mydb.cpu WHERE time > now() - INTERVAL '1 hour'
-- Less efficient: Different timestamp each time
SELECT * FROM mydb.cpu WHERE time > 1704067200000000
```
### 3. Monitor hit rates [#3-monitor-hit-rates]
Healthy dashboards should see 60-80%+ cache hit rates. Low hit rates may indicate:
* Too many unique queries
* Query text variations
* TTL too short for your refresh interval
## Configuration [#configuration]
Cache parameters are currently fixed but tuned for typical workloads:
| Parameter | Value | Rationale |
| ---------------------- | ------ | -------------------------------------- |
| SQL Transform TTL | 60s | Covers 1-2 dashboard refresh cycles |
| SQL Transform Max Size | 10,000 | Handles large multi-tenant deployments |
| Partition Cache TTL | 60s | Balance freshness vs. performance |
| Glob Cache TTL | 30s | Files change less frequently |
## Cache invalidation [#cache-invalidation]
Caches automatically expire based on TTL. Manual invalidation happens when:
* New data is ingested (invalidates partition/glob caches for affected measurements)
* Compaction runs (file paths change)
The SQL Transform Cache is not invalidated by data changes since the transformation logic doesn't depend on data content.
## Technical details [#technical-details]
### Thread safety [#thread-safety]
All caches use `sync.RWMutex` for concurrent access:
* Multiple readers allowed
* Exclusive write lock for updates
* Lock-free atomic counters for hit/miss tracking
### Memory usage [#memory-usage]
Approximate memory per cache:
| Cache | Entry Size | Max Entries | Max Memory |
| -------------- | ----------- | ----------- | ---------- |
| SQL Transform | \~500 bytes | 10,000 | \~5MB |
| Partition Path | \~200 bytes | 1,000 | \~200KB |
| Glob | \~1KB | 1,000 | \~1MB |
Total cache overhead: \~6MB typical, \~10MB maximum.
## Next steps [#next-steps]
* **[Compaction](/arc/advanced/compaction/)** - Optimize query performance through file merging
* **[WAL](/arc/advanced/wal/)** - Write-ahead log for durability
* **[Performance Benchmarks](/arc/performance/benchmarks/)** - Benchmark methodology and results
# File Compaction (/arc/advanced/compaction)
Arc's automatic compaction system merges small Parquet files into larger, optimized files for dramatically faster queries.
## Overview [#overview]
Compaction is Arc's file optimization system that **merges small files into larger ones**, substantially improving query performance.
**Key Features:**
* **Automatic** - Runs on schedule (default: hourly at :05)
* **Safe** - Locked partitions prevent concurrent compaction
* **Efficient** - Parallel, sorted merging by the query engine
* **Non-blocking** - Queries work during compaction
* **Enabled by default** - Essential for production
Compaction is **enabled by default** and runs automatically every hour.
## Why compaction matters [#why-compaction-matters]
### The small file problem [#the-small-file-problem]
Arc's high-throughput ingestion creates many small files:
```bash
At high ingest rates with a 5-second flush interval:
→ 12 files per minute per measurement
→ 720 files per hour per measurement
→ 17,280 files per day per measurement
```
**Impact on Queries:**
* **Slow queries** - The query engine must open/scan hundreds of files
* **High costs** - More S3/MinIO API calls
* **Poor compression** - Small files compress less efficiently
* **Reduced pruning** - Less effective partition elimination
### After compaction [#after-compaction]
**Real Production Test Results:**
```text
Before: 2,704 small files (Snappy) = 3.7 GB
After: 3 compacted files (ZSTD) = 724 MB
Compression: 80.4% space savings
File reduction: 901x fewer files (2,704 → 3)
Compaction time: 5 seconds
```
**Per-Measurement Breakdown:**
* **mem**: 888 files → 1 file, 1,213 MB → 239 MB (80.3% compression)
* **disk**: 906 files → 1 file, 1,237 MB → 242 MB (80.4% compression)
* **cpu**: 910 files → 1 file, 1,246 MB → 243 MB (80.5% compression)
**Query Performance:**
* **Far fewer file opens** - Single file scan vs hundreds
* **99% fewer API calls** - Massive cost reduction (2,704 → 3 LIST operations)
* **80.4% compression** - ZSTD compaction vs Snappy writes
* **Effective pruning** - The query engine can skip entire files
## How it works [#how-it-works]
### Compaction flow [#compaction-flow]
```text
1. Scheduler wakes up (cron: "5 * * * *")
↓
2. Scan storage for eligible partitions
↓
3. For each partition:
- Check age (>1 hour old?)
- Check file count (≥10 files?)
- Check if already compacted?
↓
4. Acquire partition lock (SQLite)
↓
5. Download small files to temp directory
↓
6. Compact via the query engine (parallel, sorted)
↓
7. Upload compacted file to storage
↓
8. Delete old small files
↓
9. Release lock & cleanup temp files
↓
10. Repeat for next partition
```
### Partition structure [#partition-structure]
Data is organized by hour:
```text
arc/ # Bucket
├── default/ # Database
│ └── cpu/ # Measurement
│ └── 2025/10/08/ # Date
│ ├── 14/ # Hour (2 PM) - Eligible for compaction
│ │ ├── file1.parquet (50 MB)
│ │ ├── file2.parquet (48 MB)
│ │ └── ...
│ ├── 15/ # Hour (3 PM) - Eligible for compaction
│ └── 16/ # Hour (4 PM) - CURRENT, skip!
```
Compaction merges all files in a partition (e.g., `2025/10/08/14/`) into one optimized file.
## Configuration [#configuration]
### Default configuration [#default-configuration]
Compaction is **enabled by default** in `arc.toml`:
```toml
[compaction]
enabled = true
# Hourly tier
hourly_enabled = true
hourly_schedule = "5 * * * *" # Cron schedule: every hour at :05
hourly_min_age_hours = 1 # Wait 1 hour before compacting (let the hour complete)
hourly_min_files = 10 # Only compact if >=10 files exist
# Daily tier
daily_enabled = true
daily_schedule = "0 3 * * *" # Cron schedule: 3 AM daily
daily_min_age_hours = 24 # Wait 24 hours
daily_min_files = 12 # Only compact if >=12 files exist
max_concurrent = 2 # Run 2 compactions in parallel
```
### Configuration options [#configuration-options]
#### Schedule [#schedule]
```toml
[compaction]
hourly_schedule = "5 * * * *" # Every hour at :05 (default)
daily_schedule = "0 3 * * *" # 3 AM daily (default)
# hourly_schedule = "0 */2 * * *" # Every 2 hours at :00
```
**Cron format:** `minute hour day month weekday`
#### Minimum age [#minimum-age]
```toml
[compaction]
hourly_min_age_hours = 1 # Don't compact the current hour (default)
daily_min_age_hours = 24 # Daily tier waits a full day (default)
# hourly_min_age_hours = 2 # Wait 2 hours (more conservative)
# hourly_min_age_hours = 0 # Compact immediately (aggressive)
```
Setting `hourly_min_age_hours = 0` can compact the current hour while data is still being written, potentially creating many compacted files.
#### Minimum files [#minimum-files]
```toml
[compaction]
hourly_min_files = 10 # Only compact if >=10 files (default)
daily_min_files = 12 # Daily tier threshold (default)
# hourly_min_files = 50 # Only compact with many files
# hourly_min_files = 5 # Compact more aggressively
```
#### Concurrent jobs [#concurrent-jobs]
```toml
[compaction]
max_concurrent = 2 # Run 2 compactions in parallel (default)
# max_concurrent = 4 # More parallelism (uses more CPU/memory)
# max_concurrent = 1 # Sequential (lower resource usage)
```
#### Memory limit and threads (per subprocess) [#memory-limit-and-threads-per-subprocess]
`memory_limit` and `threads` are configurable starting in Arc **v26.09.1**. On earlier versions each compaction subprocess inherits the full `database.memory_limit` and uses all CPU cores.
Each compaction job runs in an isolated subprocess with its own query engine instance. These keys bound that instance's resources:
```toml
[compaction]
memory_limit = "" # Per-subprocess engine memory limit; "" (default) = auto
threads = 0 # Per-subprocess engine threads; 0 (default) = auto
# memory_limit = "2GB" # Explicit cap
# threads = 4 # Explicit thread count
```
Env vars: `ARC_COMPACTION_MEMORY_LIMIT`, `ARC_COMPACTION_THREADS`.
**Auto behavior:**
* `memory_limit` derives as `database.memory_limit / max_concurrent`, so all concurrent compaction jobs together stay within roughly one `database.memory_limit`. With `database.memory_limit = "8GB"` and the default concurrency of 2, each subprocess gets `4GB`.
* `threads` defaults to half the CPU cores (minimum 1), so the default two concurrent jobs together use about one machine's worth of cores, leaving headroom for ingest and queries.
Accepted `memory_limit` forms are absolute sizes with a unit: `"8GB"`, `"512MB"`, `"0.5GB"`. Percent and unit-less forms are rejected at startup (DuckDB's `SET memory_limit` does not support them), as are other invalid values. The effective values appear in the startup log (`subprocess_memory_limit`, `subprocess_threads`).
When a job exceeds its memory limit, DuckDB spills to a `duckdb-spill/` directory inside the job's temp directory (under `compaction.temp_directory`) — size that volume for your largest partitions. Spill files are removed by normal job cleanup and by the crash sweeps on startup.
Lower these when compaction competes with ingest for RAM during backfill catch-up (many partitions become candidates at once); raise them to make individual large compactions faster on dedicated compactor nodes.
#### Files per batch [#files-per-batch]
`max_files_per_batch` is configurable starting in Arc **v26.09.1**. On earlier versions the batch size is fixed at 30 files and this setting has no effect.
A partition with more files than this is split into several batches, each compacted as an independent job producing its own output file.
```toml
[compaction]
max_files_per_batch = 30 # Files per compaction job (default)
# max_files_per_batch = 5 # Smaller outputs, more jobs per partition
# max_files_per_batch = 60 # Fewer, larger outputs
```
Valid range is **2–500**. Values outside it fall back to the default with a startup warning; `1` is rejected because compaction's adaptive retry cannot process a single-file batch.
This bounds the **file count** per job, not the output size in bytes — compacted file size tracks input file size, which follows your ingest buffer settings. The main reason to lower it is transferring compacted files over a constrained or intermittent link (edge deployments), where smaller, independently-transferable files resume better after an interruption. The trade-off is more compaction jobs per partition, and in cluster mode proportionally more Raft manifest entries.
The upper bound exists because a single `read_parquet()` call spanning too many files can abort.
#### Compression [#compression]
Compaction always writes its output with ZSTD, which is why compacted files are
substantially smaller than the freshly-ingested files they replace. This is not
configurable per tier.
The compression used for **incoming** writes is separate, and is set by
`ingest.compression` (default `snappy`) — see the
[configuration overview](/arc/configuration/overview/).
### Disable compaction [#disable-compaction]
```toml
[compaction]
enabled = false
```
**When to disable:**
* Testing ingestion performance
* Very low write volume (\<10 files/hour)
* Debugging compaction issues
Disabling compaction will cause queries to slow down significantly as files accumulate.
## Monitoring [#monitoring]
### Check compaction status [#check-compaction-status]
```bash
curl http://localhost:8000/api/compaction/status \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response:**
```json
{
"enabled": true,
"running": false,
"last_run": "2025-10-08T14:05:00Z",
"next_run": "2025-10-08T15:05:00Z",
"stats": {
"total_jobs": 42,
"successful_jobs": 40,
"failed_jobs": 2,
"total_files_compacted": 12580,
"total_bytes_saved": 8589934592
}
}
```
### Get detailed statistics [#get-detailed-statistics]
```bash
curl http://localhost:8000/api/compaction/stats \
-H "Authorization: Bearer $ARC_TOKEN"
```
### List eligible partitions [#list-eligible-partitions]
```bash
curl http://localhost:8000/api/compaction/candidates \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response:**
```json
{
"candidates": [
{
"partition": "default/cpu/2025/10/08/14",
"file_count": 150,
"total_size_mb": 7500,
"age_hours": 2.5,
"eligible": true
},
{
"partition": "default/mem/2025/10/08/14",
"file_count": 120,
"total_size_mb": 6000,
"age_hours": 2.5,
"eligible": true
}
],
"total_candidates": 2
}
```
### Manually trigger compaction [#manually-trigger-compaction]
```bash
curl -X POST http://localhost:8000/api/compaction/trigger \
-H "Authorization: Bearer $ARC_TOKEN"
```
### View active jobs [#view-active-jobs]
```bash
curl http://localhost:8000/api/compaction/jobs \
-H "Authorization: Bearer $ARC_TOKEN"
```
### View job history [#view-job-history]
```bash
curl http://localhost:8000/api/compaction/history \
-H "Authorization: Bearer $ARC_TOKEN"
```
## Performance impact [#performance-impact]
### Compaction performance [#compaction-performance]
**Test Environment:** Apple M3 Max (14 cores, 36GB RAM)
| Files | Size | Compaction Time | Final Size | Compression |
| ----- | ------ | --------------- | ---------- | ----------- |
| 888 | 1.2 GB | 2.1s | 239 MB | 80.3% |
| 906 | 1.2 GB | 2.2s | 242 MB | 80.4% |
| 910 | 1.2 GB | 2.3s | 243 MB | 80.5% |
**Total:** 2,704 files (3.7 GB) → 3 files (724 MB) in **6.6 seconds**
### Query performance [#query-performance]
**Before Compaction:**
```sql
SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL 1 HOUR;
-- 5.2 seconds (scan 720 files)
```
**After Compaction:**
```sql
SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL 1 HOUR;
-- 0.05 seconds (scan 1 file)
```
### Storage savings [#storage-savings]
```bash
Original files (Snappy): 3.7 GB
Compacted files (ZSTD): 724 MB
Space saved: 80.4%
```
## Best practices [#best-practices]
### 1. Let compaction run automatically [#1-let-compaction-run-automatically]
The default schedule (hourly) works well for most use cases:
```toml
[compaction]
enabled = true
hourly_schedule = "5 * * * *"
```
### 2. Monitor compaction jobs [#2-monitor-compaction-jobs]
Set up alerts for:
* Failed compaction jobs
* Partitions with >1000 files
* Compaction taking >10 minutes
### 3. Adjust based on write volume [#3-adjust-based-on-write-volume]
**High volume** (>10M records/sec):
```toml
[compaction]
hourly_min_files = 100 # Wait for more files
max_concurrent = 4 # More parallelism
```
**Low volume** (\<100K records/sec):
```toml
[compaction]
hourly_min_files = 5 # Compact with fewer files
hourly_schedule = "0 */6 * * *" # Every 6 hours
```
### 4. Tune files per batch [#4-tune-files-per-batch]
```toml
[compaction]
max_files_per_batch = 30 # Files per compaction job (default)
# max_files_per_batch = 60 # Fewer, larger outputs
# max_files_per_batch = 5 # Smaller outputs, more jobs per partition
```
### 5. Reduce file generation at source [#5-reduce-file-generation-at-source]
**Best practice:** Increase buffer sizes to generate fewer files:
```toml
[ingest]
max_buffer_size = 200000 # Up from 50,000 (4x fewer files)
max_buffer_age_ms = 10000 # Up from 5000 (2x fewer files)
```
**Impact:**
* Files generated: 2,000/hour → 250/hour (8x reduction)
* Compaction time: substantially reduced
* Memory usage: +300MB per worker
This is the **most effective optimization** - fewer files means faster compaction AND faster queries.
## Troubleshooting [#troubleshooting]
### Compaction not running [#compaction-not-running]
**Check status:**
```bash
curl http://localhost:8000/api/compaction/status
```
**Verify configuration:**
```bash
# Check if enabled
grep "enabled" arc.toml
# Check schedule
grep "schedule" arc.toml
```
**Check logs:**
```bash
# Docker
docker logs arc | grep compaction
# Native
sudo journalctl -u arc | grep compaction
```
### Partition skipped: No `time` column [#partition-skipped-no-time-column]
**Symptoms:** A warning in the logs:
```bash
Skipping compaction: no 'time' column in any input file (data was not written by Arc ingest); leaving source files in place
```
**Cause:** Compaction requires a `time` column — it normalizes the column's type and sorts output by it. Arc's ingest path always writes one, so this only happens for Parquet files placed into the storage directory by external tools (custom loaders, bulk copies from other systems).
**What happens:** The partition is left untouched and the job counts as completed, not failed. The warning repeats each cycle as long as the partition stays above the compaction file-count threshold.
**Solutions:**
1. **Rewrite the data through Arc ingest** so it carries a proper `time` column, or
2. **Rewrite the files in place** with the timestamp column renamed/cast to `time` (type `TIMESTAMP WITH TIME ZONE`), or
3. **Leave it as-is** — the data stays queryable; it just won't be compacted.
Partitions where only *some* files lack `time` are not skipped: they compact normally, and rows from files without the column get `NULL` time values.
### Compaction taking too long [#compaction-taking-too-long]
**Symptoms:** Compaction jobs running for >30 minutes
**Solutions:**
1. **Reduce files per batch:**
```toml
[compaction]
max_files_per_batch = 10 # Smaller compaction jobs
```
2. **Increase parallelism:**
```toml
[compaction]
max_concurrent = 4
```
3. **Reduce files at source:**
```toml
[ingest]
max_buffer_size = 200000
```
### Out of disk space during compaction [#out-of-disk-space-during-compaction]
**Symptoms:** Compaction fails with disk space errors
**Solutions:**
1. **Use temp directory on larger disk:**
```bash
export TMPDIR=/mnt/large-disk/tmp
```
2. **Reduce concurrent jobs:**
```toml
[compaction]
max_concurrent = 1
```
3. **Clean up old compacted files manually:**
```bash
# Remove small files that were already compacted
find ./data -name "*.parquet" -size -10M -delete
```
### Compaction locks not releasing [#compaction-locks-not-releasing]
**Symptoms:** Partitions stuck in "locked" state
**Check locks:**
```bash
# View active locks
sqlite3 ./data/arc.db "SELECT * FROM compaction_locks;"
```
**Clear stale locks:**
```bash
# Locks expire automatically after 2 hours
# Or manually clear:
sqlite3 ./data/arc.db "DELETE FROM compaction_locks WHERE expires_at < datetime('now');"
```
## API reference [#api-reference]
### GET /api/v1/compaction/status [#get-apiv1compactionstatus]
Get current compaction status.
**Response:**
```json
{
"enabled": true,
"running": false,
"last_run": "2025-10-08T14:05:00Z",
"next_run": "2025-10-08T15:05:00Z"
}
```
### GET /api/v1/compaction/stats [#get-apiv1compactionstats]
Get detailed compaction statistics.
### GET /api/v1/compaction/candidates [#get-apiv1compactioncandidates]
List partitions eligible for compaction.
### POST /api/v1/compaction/trigger [#post-apiv1compactiontrigger]
Manually trigger compaction.
**Response:**
```json
{
"message": "Compaction triggered",
"job_id": "comp_1696775400"
}
```
### GET /api/v1/compaction/jobs [#get-apiv1compactionjobs]
View active compaction jobs.
### GET /api/v1/compaction/history [#get-apiv1compactionhistory]
View compaction job history.
## Summary [#summary]
Compaction is essential for production deployments:
**Benefits:**
* Faster queries
* 80% storage savings
* 99% fewer API calls
* Automatic and safe
**Default configuration works for most cases:**
```toml
[compaction]
enabled = true
hourly_schedule = "5 * * * *"
hourly_min_age_hours = 1
hourly_min_files = 10
```
**Monitor regularly:**
* Check `/api/v1/compaction/status`
* Alert on failed jobs
* Watch for partitions with >1000 files
## Next steps [#next-steps]
* **[Monitor Compaction](/arc/operations/telemetry/)** - Set up health checks
* **[Configure WAL](/arc/advanced/wal/)** - Add durability guarantees
* **[Configuration Reference](/arc/configuration/overview/)** - Tune settings for your workload
# Data-Time Partitioning (/arc/advanced/data-time-partitioning)
Arc organizes Parquet files by the data's timestamp rather than ingestion time, enabling proper backfill of historical data and optimal query performance.
## Overview [#overview]
Data-time partitioning ensures that your data lands in the correct time-based partitions based on when the events actually occurred, not when they were ingested into Arc.
**Key Features:**
* **Historical backfill** - Past data lands in correct partitions (e.g., December 2024 data goes to `2024/12/` folders)
* **Sorted files** - Data is sorted by timestamp within each Parquet file
* **Automatic splitting** - Batches spanning multiple hours are split into separate files
* **Partition pruning** - Enables accurate time-range query optimization
Data-time partitioning is **enabled by default** and requires no configuration.
## Why it matters [#why-it-matters]
### The problem with ingestion-time partitioning [#the-problem-with-ingestion-time-partitioning]
Traditional ingestion-time partitioning creates problems when backfilling historical data:
```text
Scenario: Ingesting December 2024 sensor data on January 4, 2025
❌ Ingestion-time partitioning:
data/mydb/cpu/2025/01/04/... (wrong - today's partition)
✅ Data-time partitioning:
data/mydb/cpu/2024/12/01/14/... (correct - data's timestamp)
data/mydb/cpu/2024/12/01/15/...
```
**Impact:**
* **Broken queries** - Time-range queries can't find historical data
* **No partition pruning** - The query engine must scan all files, not just relevant partitions
* **Mixed data** - Historical and current data mixed in same partition
* **Poor compaction** - Files with mixed timestamps don't compact efficiently
### After data-time partitioning [#after-data-time-partitioning]
With data-time partitioning, your data is always organized correctly:
```bash
# Query for December 2024 data only scans December partitions
SELECT * FROM mydb.cpu
WHERE time >= '2024-12-01' AND time < '2025-01-01'
→ Arc scans only: data/mydb/cpu/2024/12/**/*.parquet
→ Skips all 2025 partitions entirely
```
**Benefits:**
* **Faster queries** - Partition pruning eliminates irrelevant files
* **Accurate historical analysis** - Data lives where it belongs
* **Efficient compaction** - Files with similar timestamps compact together
* **Predictable storage** - Easy to manage retention by date folders
## How it works [#how-it-works]
### Single-hour batches [#single-hour-batches]
When all records in a batch fall within the same hour:
```text
Incoming batch (all records from 2024-12-15 14:xx):
┌─────────────────────────┬────────┬───────┐
│ time │ host │ value │
├─────────────────────────┼────────┼───────┤
│ 2024-12-15T14:05:00.000 │ srv01 │ 45.2 │
│ 2024-12-15T14:32:00.000 │ srv01 │ 47.8 │
│ 2024-12-15T14:58:00.000 │ srv01 │ 44.1 │
└─────────────────────────┴────────┴───────┘
Result: Single sorted file
→ data/mydb/cpu/2024/12/15/14/abc123.parquet
(records sorted by timestamp)
```
### Multi-hour batches [#multi-hour-batches]
When a batch spans multiple hours, Arc automatically splits it:
```text
Incoming batch (records spanning 14:00-16:00):
┌─────────────────────────┬────────┬───────┐
│ time │ host │ value │
├─────────────────────────┼────────┼───────┤
│ 2024-12-15T14:30:00.000 │ srv01 │ 45.2 │
│ 2024-12-15T15:15:00.000 │ srv01 │ 47.8 │
│ 2024-12-15T15:45:00.000 │ srv01 │ 46.3 │
│ 2024-12-15T16:10:00.000 │ srv01 │ 44.1 │
└─────────────────────────┴────────┴───────┘
Result: Three separate sorted files
→ data/mydb/cpu/2024/12/15/14/abc123.parquet (1 record)
→ data/mydb/cpu/2024/12/15/15/def456.parquet (2 records)
→ data/mydb/cpu/2024/12/15/16/ghi789.parquet (1 record)
```
### Partition structure [#partition-structure]
Data is organized hierarchically by time:
```text
data/ # Storage root
├── default/ # Database
│ └── cpu/ # Measurement
│ ├── 2024/ # Year
│ │ └── 12/ # Month
│ │ ├── 01/ # Day
│ │ │ ├── 14/ # Hour (2 PM)
│ │ │ │ └── abc123.parquet
│ │ │ └── 15/ # Hour (3 PM)
│ │ │ └── def456.parquet
│ │ └── 15/ # Day 15
│ │ └── ...
│ └── 2025/ # Year
│ └── 01/ # Month
│ └── ...
```
## Sorting within files [#sorting-within-files]
Each Parquet file contains data sorted by timestamp in ascending order:
```sql
-- Data is pre-sorted, enabling efficient scans
-- Sorted file metadata enables:
-- - Early termination on LIMIT queries
-- - Efficient MIN/MAX aggregations
-- - Optimized range scans
SELECT * FROM mydb.cpu
WHERE time >= '2024-12-15T14:00:00'
AND time < '2024-12-15T14:30:00'
ORDER BY time
LIMIT 100
```
**Performance benefits:**
* **No runtime sorting** - Data already ordered
* **Efficient LIMIT** - Stop scanning after N rows
* **Fast aggregations** - MIN/MAX read file metadata
* **Optimal compression** - Similar timestamps compress better
## UTC consistency [#utc-consistency]
All partition paths use UTC time, regardless of server timezone:
```bash
Server in New York (UTC-5):
Local time: 2024-12-15 10:00 EST
UTC time: 2024-12-15 15:00 UTC
→ Data written to: data/mydb/cpu/2024/12/15/15/...
(UTC hour, not local hour)
```
Using UTC ensures consistent partitioning across servers in different timezones and prevents partition misalignment during timezone changes (DST).
## Query partition pruning [#query-partition-pruning]
Arc's query engine automatically prunes partitions based on time predicates:
```sql
-- This query only scans December 2024 partitions
SELECT host, AVG(value) as avg_value
FROM mydb.cpu
WHERE time >= '2024-12-01T00:00:00Z'
AND time < '2025-01-01T00:00:00Z'
GROUP BY host
```
**What happens:**
1. Arc parses the time range from the WHERE clause
2. Converts range to partition paths: `2024/12/**/*.parquet`
3. The query engine receives only the relevant file list
4. Files outside the range are never opened
**Performance impact:**
* Querying 1 month in a year of data → \~92% fewer files scanned
* Querying 1 day in a month of data → \~97% fewer files scanned
* Querying 1 hour in a day of data → \~96% fewer files scanned
## Backfilling historical data [#backfilling-historical-data]
Data-time partitioning makes historical backfill straightforward:
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Backfill sensor data from December 2024
# (even though we're ingesting in January 2025)
historical_data = {
"time": [
1701388800000000, # 2024-12-01T00:00:00Z
1701475200000000, # 2024-12-02T00:00:00Z
1701561600000000, # 2024-12-03T00:00:00Z
],
"sensor_id": ["temp-01", "temp-01", "temp-01"],
"value": [22.5, 23.1, 21.8],
}
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
client.write.write_columnar(
measurement="sensors",
columns=historical_data,
)
# Data lands in correct partitions:
# → data/default/sensors/2024/12/01/00/...
# → data/default/sensors/2024/12/02/00/...
# → data/default/sensors/2024/12/03/00/...
```
## Interaction with compaction [#interaction-with-compaction]
Data-time partitioning works seamlessly with [file compaction](/arc/advanced/compaction/):
1. **Ingestion** - Small files written to correct hourly partitions
2. **Compaction** - Files within each partition merged into larger files
3. **Result** - Each hour has one large, sorted, optimized file
```text
Before compaction:
data/mydb/cpu/2024/12/15/14/
├── file1.parquet (5 MB, 100K records)
├── file2.parquet (4 MB, 80K records)
├── file3.parquet (6 MB, 120K records)
└── ... (100 more small files)
After compaction:
data/mydb/cpu/2024/12/15/14/
└── compacted_abc123.parquet (450 MB, 10M records, sorted)
```
## Best practices [#best-practices]
### Timestamp requirements [#timestamp-requirements]
Ensure your timestamps are accurate:
```python
# ✅ Good: Microsecond Unix timestamps (UTC)
"time": [1701388800000000, 1701388801000000]
# ✅ Good: Nanosecond Unix timestamps (UTC)
"time": [1701388800000000000, 1701388801000000000]
# ❌ Bad: String timestamps (require parsing)
"time": ["2024-12-01T00:00:00Z", "2024-12-01T00:00:01Z"]
```
### Bulk imports [#bulk-imports]
When importing large historical datasets:
1. **Sort by time first** - Pre-sorted data writes faster
2. **Batch by hour** - Reduces file splitting overhead
3. **Use columnar format** - MessagePack columnar is fastest
4. **Trigger compaction after** - Consolidate small files
```bash
# After bulk import, trigger compaction
curl -X POST http://localhost:8000/api/v1/compaction/hourly \
-H "Authorization: Bearer $TOKEN"
```
### Monitoring partition distribution [#monitoring-partition-distribution]
Check that data is landing in expected partitions:
```sql
-- View partition distribution
SELECT
EXTRACT(YEAR FROM time) as year,
EXTRACT(MONTH FROM time) as month,
COUNT(*) as records
FROM mydb.sensors
GROUP BY year, month
ORDER BY year, month
```
## Next steps [#next-steps]
* [File Compaction](/arc/advanced/compaction/) - Optimize partitioned files
* [Retention Policies](/arc/data-lifecycle/retention-policies/) - Manage data by partition age
* [Performance Benchmarks](/arc/performance/benchmarks/) - Benchmark partition pruning benefits
# Edge Sync (/arc/advanced/edge-sync)
Both sides ship in Arc **v26.09.1**, over two transports: a **network** link (spoke pushes to hub) and an **air gap** (a signed bundle carried on removable media, with a receipt on the return leg). Passes are **manual** in this release — you decide when one runs. The scheduled agent that runs them automatically is Enterprise and lands in a later release. See the limitations below.
Arc runs at the edge: a single binary with local storage in a vehicle, a factory cell, a mine site, or a forward deployment. **Edge sync** ships the Parquet files it produces to a central Arc — the *hub* — so data collected somewhere with intermittent connectivity ends up somewhere you can query it.
## The model [#the-model]
* A **spoke** is an edge Arc instance. It initiates every transfer; the hub never reaches back into it. That matters because edges usually sit behind NAT with no inbound reachability.
* A **hub** is a central Arc with the receive endpoint enabled.
* The unit of sync is a **file**, not a row. Arc already writes immutable, content-addressed Parquet, so shipping whole files gives end-to-end integrity for free and costs the hub no re-ingestion.
Connectivity is treated as the exception rather than the norm. A transfer interrupted mid-file resumes from a byte offset rather than restarting, and re-delivering a file the hub already holds is a no-op.
## Configuration [#configuration]
```toml
[edge_sync]
enabled = true # default false
hub_id = "ground-station" # required when enabled
max_file_bytes = 536870912 # 512MiB default
max_reconcile_entries = 10000 # ~2MB per discovery batch
```
Environment variables follow the usual pattern: `ARC_EDGE_SYNC_ENABLED`, `ARC_EDGE_SYNC_HUB_ID`, `ARC_EDGE_SYNC_MAX_FILE_BYTES`, `ARC_EDGE_SYNC_MAX_RECONCILE_ENTRIES`.
### `enabled` [#enabled]
Off by default. Enabling it mounts `POST /api/v1/sync/file`, which accepts file writes from registered spokes — so it should be a deliberate decision, not something that appears on upgrade.
### `hub_id` [#hub_id]
Names this hub, and is bound into every request's HMAC. A request signed for one hub is rejected at another, even when the spoke legitimately syncs to both and shares a secret with each.
Arc refuses to start if this is empty while `enabled = true`, because an empty value would let a request captured at one hub be replayed at another. It also rejects path separators, NUL bytes, control characters, and values over 128 bytes.
### `max_file_bytes` [#max_file_bytes]
Caps a single upload. This is a denial-of-service control rather than a tuning knob: Arc buffers a request body before authentication runs, so without a bound, anyone who can reach the port could pin memory without holding any credential.
It must not exceed `server.max_payload_size` (1GB by default) — the server limit is enforced first, so a larger value would never take effect. Arc refuses to start rather than let you configure a bound that silently does nothing.
### `max_reconcile_entries` [#max_reconcile_entries]
Caps one batch-discovery request. A spoke with a larger backlog sends several requests.
The bound exists for the same reason as `max_file_bytes`: Arc buffers a request body before authentication, so an unbounded batch would be a memory claim by an unauthenticated caller. Capping does not cost the property that matters — discovery is still one request per batch rather than one per file, so 5,000 pending files is a handful of requests instead of 5,000.
A batch above the cap is refused with `413` and the limit, so a spoke knows what to page under — the spoke's agent reads the limit from the refusal and re-pages under it automatically.
### `staging_sweep_max_age_hours` [#staging_sweep_max_age_hours]
How old an abandoned partial upload must be before the hourly sweep reclaims its staging space. Default `72` — deliberately longer than a plausible contact gap, because a staged partial is also the spoke's resume checkpoint, and sweeping it early forces a full re-send on exactly the links resume exists for. `0` disables the sweep.
## Registering a spoke [#registering-a-spoke]
A hub only accepts files from a spoke it knows about. Registration generates the shared secret:
```bash
curl -X POST https://hub.example.com/api/v1/sync-spokes/ \
-H "Authorization: Bearer $ARC_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"spoke_id": "rocket-01", "name": "Rocket 07 Telemetry"}'
```
```json
{
"spoke_id": "rocket-01",
"name": "Rocket 07 Telemetry",
"secret": "5db508a4…",
"warning": "This secret is shown once and cannot be retrieved. Store it now; if it is lost, rotate."
}
```
**Capture the secret from this response.** It is the only time it is readable — the hub stores it encrypted and every later read returns metadata only. If it is lost, rotate rather than trying to recover it.
The `spoke_id` becomes the first path segment of everything that spoke writes, so it cannot contain a path separator, a dot prefix, or a NUL byte.
### Managing spokes [#managing-spokes]
| Endpoint | Effect |
| --------------------------------------- | ----------------------------------------------------------------------------- |
| `GET /api/v1/sync-spokes/` | List spokes with their file and byte counters, and when each last reported in |
| `GET /api/v1/sync-spokes/{id}` | One spoke's metadata — never its secret |
| `POST /api/v1/sync-spokes/{id}/rotate` | Issue a new secret, returned once. The old one stops working **immediately** |
| `POST /api/v1/sync-spokes/{id}/disable` | Cut the spoke off, reversibly. History and counters survive |
| `POST /api/v1/sync-spokes/{id}/enable` | Restore it, no re-provisioning needed |
| `DELETE /api/v1/sync-spokes/{id}` | Remove the registration. Files it already sent are **kept** |
All of these require an admin token, including the read paths: the spoke list is a map of which edge deployments exist and when each last checked in.
With Arc's authentication disabled there are no admin tokens, so **anyone who can reach the port can register a spoke and obtain working write credentials** for this hub. That is true of every Arc admin endpoint in that mode, but the consequence is sharper here. A hub logs a warning at startup when this applies. Enable authentication, or restrict network access to the hub.
Registering the same `spoke_id` twice is refused with `409` rather than treated as an update — silently reissuing a secret would lock out a live edge box with no signal.
### `ARC_ENCRYPTION_KEY` is required [#arc_encryption_key-is-required]
Spoke secrets are encrypted at rest using the same key MQTT uses for broker passwords. A hub with `edge_sync.enabled = true` and no `ARC_ENCRYPTION_KEY` **refuses to start**.
There is no plaintext fallback on purpose: the sync database also holds audit logs, and a silent downgrade would leave every spoke's write credential readable to anyone who copied the file.
Encrypted rather than hashed because the hub must *recompute* an HMAC from the secret — unlike an API token, which is only ever checked against a value the caller presents.
## Discovery: One round-trip [#discovery-one-round-trip]
`POST /api/v1/sync/reconcile` takes a spoke's pending set and answers which files the hub already holds:
```json
{
"missing": ["metrics/cpu/2026/08/07/14/a.parquet"],
"present": ["metrics/cpu/2026/08/07/14/b.parquet"],
"conflicts": [{"path": "…/c.parquet", "their_sha256": "…"}]
}
```
* **`missing`** — send these.
* **`present`** — the hub already has this exact content. This is the lost-acknowledgment path: a transfer that completed but whose acknowledgment never arrived is discovered here in bulk, and the spoke advances without re-sending a byte.
* **`conflicts`** — the hub holds that path with *different* content. Surfaced for the whole backlog at once rather than discovered one `409` at a time during transfer.
The answer comes from a hub-side index of received files, not from reading Parquet, so it costs the same whether the hub holds a thousand files or a million.
## How a transfer is handled [#how-a-transfer-is-handled]
1. The request is authenticated twice: an Arc API token, then a per-spoke HMAC binding the spoke, the hub, the path, and the content digest.
2. Bytes stream into a staging area while Arc hashes them.
3. The hash is compared against the digest the spoke declared.
4. **Only on a match** is the file promoted to its final location.
A mismatch is discarded at step 4, so corrupt bytes never appear where a reader would find them.
Files are stored under the spoke's namespace — `{spoke_id}/{original path}` — so two edges producing the same measurement for the same hour do not collide. The rewrite happens on the hub, so a spoke stays unaware of it and can sync to several hubs unmodified.
## Hub-side compaction of received data [#hub-side-compaction-of-received-data]
Since 26.09.1 the hub's own compaction processes spoke namespaces (`edge_sync.compact_received_namespaces`, default `true`): registered spoke IDs expand into per-database compaction targets, so the small raw files spokes deliver are folded into ordinary compacted Parquet on the hub's compaction schedule — no protocol involvement, and hub queries over spoke data stop paying a growing small-files cost.
The sync protocol stays truthful: receipts for compacted-away files are **marked, never forgotten**. A spoke that re-offers one — a pruned ledger being rediscovered, or a stale air-gap drive imported late — is answered *already present* with zero bytes moved; a different-content re-delivery still conflicts. Three operational rules:
* Keep spoke registrations for as long as their data sits in storage: an unregistered namespace is not expanded (and, on a dual-role node, loses its relay exclusion too).
* Set the key to `false` if you rely on the raw per-file layout of received data (forensics, per-file external tooling).
* On a multi-node cluster, enable this only where the spoke-facing endpoint runs on the compactor node — receipt bookkeeping is node-local in this release.
## Dual-role nodes (hub + spoke) [#dual-role-nodes-hub--spoke]
An Arc can be a hub and a spoke at once — receiving from its own edges while syncing its own telemetry upstream. Since 26.09.1 the node's own sync discovery **excludes received spoke namespaces** (the registered spoke IDs), so other edges' data is never forwarded upstream double-namespaced. Explicit relay topologies are not supported yet. Two rules for dual-role operators: keep spoke registrations for as long as their data sits in storage (an unregistered namespace loses its exclusion), and don't name a local database the same as a registered spoke ID.
## Querying spoke data on the hub [#querying-spoke-data-on-the-hub]
Each spoke's data appears on the hub as a database named after the spoke. Spoke IDs typically contain hyphens, so quote them:
```sql
SELECT count(*), min(temp), max(temp)
FROM "rocket-01".engine_temp
WHERE time > now() - INTERVAL 1 HOUR;
```
(Requires Arc 26.09.1+ — earlier versions had a query-layer bug where quoted identifiers resolved to nonexistent storage paths and returned zero rows.)
## Response codes [#response-codes]
| Code | Meaning | What the spoke does |
| ----- | -------------------------------------------------------------- | --------------------------------------------------------------- |
| `200` | Committed, or already present with identical content | Marks it synced |
| `206` | Partial — the body ended early | Resumes from the returned offset |
| `409` | Same path, **different** content | Stops and raises an alarm; never retries |
| `413` | Above `max_file_bytes` | Configuration problem, not transient |
| `422` | Checksum mismatch; the upload was discarded | Retries from its own copy |
| `401` | Authentication failed | Checks `ARC_EDGE_SYNC_HUB_TOKEN`, then its secret and its clock |
| `503` | Hub-side failure, e.g. a manifest write during a Raft election | Retries later |
A `409` is deliberately not retryable. It means either two spokes are writing the same namespaced path or one side's bytes are corrupt — both need a human, and overwriting would destroy whichever copy is correct.
## Storage backend support [#storage-backend-support]
| | Local | S3 / Azure |
| -------------------- | ----- | --------------------------- |
| Verify before commit | Yes | Yes |
| Byte-offset resume | Yes | **No** — restarts from zero |
Object storage cannot append to a block object, so a dropped transfer starts over. That is a throughput cost, not a correctness problem, and it only bites when a connectivity window is shorter than a single file transfer.
If you run a hub on object storage over intermittent links, lower [`compaction.max_files_per_batch`](/arc/advanced/compaction/#files-per-batch) on the **spokes** so individual files stay small enough to cross a window.
## Network-transport limitations [#network-transport-limitations]
These apply to the network path described above. The air-gap transport has its own, at the end of this page.
* **Passes are manual.** A pass runs when you trigger one, via `POST /api/v1/spoke-sync/run` or a scheduler of your own (cron, a systemd timer, a link-up hook). The built-in scheduled agent is Enterprise and not in this release.
* **Uploads are buffered, not streamed.** A transfer is bounded by `max_file_bytes` and held in memory for its duration.
* **Abandoned partial uploads are not swept automatically.** The mechanism exists but is not yet scheduled, so a spoke that abandons transfers leaves staging files behind.
* **Deleting a synced file from hub storage is reconciled lazily.** Reconcile confirms that files its index claims are still in storage, and forgets the ones that are gone, so a spoke re-sends them on the next pass. Note that a retention policy whose database matches a spoke's namespace **will** delete that spoke's files — the namespace is the spoke ID, and retention operates on whatever database name it is given.
* **Hub-side corruption after commit is not detected by reconcile.** Files are verified before they are committed, but reconcile then answers from the index rather than re-hashing storage — re-reading every received file on every pass is exactly the cost the index exists to avoid. Corruption at rest is a storage-integrity concern, not a sync one.
## The spoke side [#the-spoke-side]
A spoke is an edge Arc instance that pushes its files to a hub. Enable it in `arc.toml`:
```toml
[edge_sync.spoke]
enabled = true # default false
hub_url = "https://hub.example.com" # required when enabled
spoke_id = "rocket-01" # this spoke's ID, as registered on the hub
hub_id = "ground-station" # the REMOTE hub's edge_sync.hub_id
max_attempts = 5 # attempts before a file is marked failed
max_concurrent = 2 # simultaneous transfers
batch_size = 1000 # files per reconcile round-trip; 0 = whole backlog at once
ledger_retention_days = 90 # prune synced/skipped ledger rows; 0 = never
```
A reconcile page the hub refuses as too large (over its `max_reconcile_entries` or its byte limit) is split and retried within the same pass, so no `batch_size` value can leave a backlog undrainable.
Both credentials go in the environment, never the file:
```bash
export ARC_EDGE_SYNC_SPOKE_SECRET=""
export ARC_EDGE_SYNC_HUB_TOKEN=""
```
The secret drives the per-spoke HMAC; the token satisfies the hub's API-token middleware, which fronts the sync endpoints at write level. Without the token, every request against a hub running with authentication enabled (the default) fails with a `401` whose error text names the variable — only a hub with authentication disabled needs none. Arc **refuses to start** if the secret or token appears in the config file. One that is ignored still leaks, and leaving it in place makes the committed copy — the one that gets backed up and committed to a repo — look load-bearing.
`hub_id` must match the hub's own `edge_sync.hub_id` exactly. It is bound into every request MAC, so a mismatch fails *every* request with a `400` that looks like a hub problem; Arc validates it at startup instead of letting you discover it during a contact window.
### Running a pass [#running-a-pass]
| Endpoint | Purpose |
| ------------------------------- | ------------------------------------------ |
| `POST /api/v1/spoke-sync/run` | Run one pass and return what it did |
| `GET /api/v1/spoke-sync/status` | Pending/synced/failed counts, and sync lag |
| `GET /api/v1/spoke-sync/ledger` | Per-file state, attempts, and last error |
All three require an admin token: triggering a pass moves data off the box and spends whatever link budget it has.
```bash
curl -X POST https://edge.local:8000/api/v1/spoke-sync/run \
-H "Authorization: Bearer $ARC_TOKEN"
```
```json
{
"discovered": 20,
"recovered": 0,
"already_present": 0,
"sent": 20,
"bytes_sent": 22015,
"partial": 0,
"failed": 0,
"conflicts": [],
"duration_ms": 19
}
```
A pass recovers transfers interrupted by a crash, discovers new files, reconciles the backlog in one round-trip, then streams what the hub lacks — **newest first**, so a contact window that closes mid-backlog has already delivered the freshest telemetry. It **pages until the backlog drains**: one pass on a spoke returning from a long outage moves everything, not just the first `batch_size` files.
Files are hashed once at discovery and the ledger is on disk, so a spoke restarted mid-backlog neither re-hashes nor re-sends what already landed. **Nothing is deleted from the spoke** — sync is a copy, and local retention stays yours to configure.
### When a tracked file vanishes before delivery [#when-a-tracked-file-vanishes-before-delivery]
Compaction (on by default) rewrites raw Parquet and deletes the sources; retention deletes whole partitions. A file caught by either after discovery but before delivery has nothing left to send. The ledger marks it `skipped` — reported in `/status` and in each pass or export result — instead of retrying it into a permanent `failed` row, or (on the air-gap path) failing the whole export. Only a storage backend positively reporting the file gone triggers the skip; a transient storage error never does. Terminal rows (`synced`, `skipped`) are pruned after `ledger_retention_days`.
### Compaction waits for delivery [#compaction-waits-for-delivery]
Since 26.09.1, local compaction on a syncing spoke **defers until the data has been delivered** (`edge_sync.spoke.defer_compaction_until_synced`, default `true`): only files the sync ledger reports `synced` — over the network, or acked back on the air-gap path — are eligible compaction inputs, and compacted outputs are recorded as already-delivered content that never syncs. The hub receives every row exactly once, as raw files; hub-side duplication from compaction cannot happen, and neither can compaction destroying rows the hub never received.
What that means operationally:
* A spoke with a sync backlog logs a per-scan deferral line, broken out by ledger state. `exported` means files are on a drive awaiting its ack; `failed`/`conflicted` need operator attention (see below); the partition compacts on the first cycle after delivery.
### Fixing stuck entries [#fixing-stuck-entries]
A file that exhausts its retries — or hits a same-path-different-content conflict — lands in `failed` and stays there. Two admin endpoints resolve it:
```bash
# Transient cause fixed (hub reachable again, token rotated): retry with a fresh budget
curl -X POST https://edge.local:8000/api/v1/spoke-sync/ledger/requeue -H "Authorization: Bearer $ARC_TOKEN" -H "Content-Type: application/json" -d '{"path": "metrics/cpu/2026/08/07/14/cpu_001.parquet"}' # or {"all": true}
# Known-bad: stop surfacing it (pruned after ledger_retention_days; reversible via requeue)
curl -X POST https://edge.local:8000/api/v1/spoke-sync/ledger/dismiss -H "Authorization: Bearer $ARC_TOKEN" -H "Content-Type: application/json" -d '{"all": true}'
```
To find what to requeue, list by state: `GET /api/v1/spoke-sync/ledger?state=skipped` enumerates the entries the default view hides — vanished sources, compacted outputs, and operator-dismissed failures, told apart by `last_error`. Any single state works (`pending`, `in_flight`, `synced`, `exported`, `failed`, `skipped`).
A dismissal marks the entry `skipped` rather than deleting it — a deleted row whose file still exists would be re-discovered next pass and come right back. It is reversible via requeue **while the file survives**: dismissing also makes the file eligible for local compaction (you renounced delivery, so the partition must not stay wedged on it), and once compaction or retention consumes it there is nothing left to requeue. Requeue a *conflict* only after resolving the divergence on the hub; otherwise it conflicts again and returns to `failed`. Both endpoints also unblock delivery-deferred compaction for the affected partition.
* An air-gap spoke's compaction cadence is bounded by its drive round trips — plan local disk for the gap.
* Compacted files that predate the upgrade sync once (their rows may exist nowhere else); a one-time duplicate for old partitions is possible, a loss is not.
* Setting the key to `false` restores the pre-26.09.1 behavior: compaction runs freely and hub queries double-count any partition whose raws synced before compaction consumed them. Flipping it back to `true` later is safe: anything compacted during the ungated period is treated as legacy and syncs once — a bounded duplicate, never a silent loss.
### Reading the results [#reading-the-results]
`conflicts` is reported in full rather than counted, because each one needs a decision about which copy is right:
```json
{
"sent": 0,
"conflicts": [
{"path": "default/cpu/2026/08/07/18/cpu_001.parquet", "their_sha256": "04ed50c9…"}
],
"warning": "Some paths hold different content on the hub. These are not retried; investigate before re-syncing."
}
```
Conflicts are never retried and never overwrite: the same path holding different content means a spoke-ID collision or corruption, and re-sending would either be refused or destroy the evidence.
For anything stuck, `GET /api/v1/spoke-sync/ledger` shows attempt counts and the last error per file, so you do not have to open the SQLite database to answer "why is this not syncing?".
### Scheduling it yourself [#scheduling-it-yourself]
Until the Enterprise scheduled agent ships, a timer is enough:
```bash
# Every 15 minutes, and on link-up.
*/15 * * * * curl -fsS -X POST http://127.0.0.1:8000/api/v1/spoke-sync/run \
-H "Authorization: Bearer $ARC_TOKEN" >> /var/log/arc-sync.log 2>&1
```
A pass is safe to trigger concurrently with ingest and safe to re-run — one that finds nothing to do returns immediately.
## Air-gap bundles [#air-gap-bundles]
Some spokes have no network path at all — a submarine, a classified facility, a vehicle whose data comes off on a physical drive. For those, a spoke writes a **signed bundle** to removable media and someone carries it to the hub.
```toml
[edge_sync.spoke.bundle]
enabled = true # default false
allowed_dirs = ["/mnt/usb"] # REQUIRED; an empty list refuses every export
max_files = 10000 # per bundle
max_bytes = 68719476736 # 64 GiB per bundle
```
This is **independent of `edge_sync.spoke.enabled`**. A fully air-gapped spoke sets only the bundle block — it needs no `hub_url`, and the network endpoints return `503`. A spoke that has both intermittent connectivity and a drive courier enables both and uses whichever is available.
You still need `spoke_id` and `hub_id`: both are bound into the bundle's signature, and the hub rejects a bundle that names a different one.
### Writing a bundle [#writing-a-bundle]
```bash
curl -X POST https://edge.local:8000/api/v1/spoke-sync/export \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{"path": "/mnt/usb"}'
```
```json
{
"exported": true,
"bundle_id": "06FXVSQXJ2C0EBDFDQ9D24S1E8",
"dir": "/mnt/usb/bundle-submarine-01-06FXVSQXJ2C0EBDFDQ9D24S1E8",
"files": 3,
"bytes": 3285,
"note": "Files are marked exported, not synced. They advance to synced when the hub acknowledges the bundle."
}
```
Optional `"limit": N` caps one bundle below `max_files`. A spoke with nothing new returns `{"exported": false, "reason": "nothing to export"}` — not an error, so a scheduled export does not look broken when the backlog is drained.
### What a bundle looks like [#what-a-bundle-looks-like]
```text
bundle-submarine-01-06FXVSQXJ2C0EBDFDQ9D24S1E8/
manifest.json signed header: bundle ID, spoke, hub, entry digest, MAC
entries.jsonl one JSON object per file: path, sha256, size
data/ the Parquet files, under their original paths
ack.json on a RETURNED drive only: the hub's signed receipt
```
`ack.json` appears after the drive has been to the hub. It is not covered by the manifest's digest — it cannot be, since it is created after the manifest is signed — but it is independently signed with the same per-spoke secret, so a replaced one is refused when the spoke reads it.
A directory rather than an archive, for two reasons:
* **Resume is free.** An interrupted copy leaves whole files, and the manifest's per-file SHA says exactly which landed — so resuming re-copies the mismatches instead of restarting. A truncated tar offers no such granularity.
* **It is auditable.** Someone has to inspect what crosses an air gap. `ls` and `sha256sum entries.jsonl` answer that without special tooling:
```bash
sha256sum /mnt/usb/bundle-*/entries.jsonl
python3 -c "import json;print(json.load(open('/mnt/usb/bundle-.../manifest.json'))['entries_sha256'])"
```
Bundle IDs are time-prefixed, so a directory listing sorts into creation order. They use Crockford base32 (no `I`, `L`, `O`, or `U`) because operators read them off screens.
### Where bundles may be written [#where-bundles-may-be-written]
`allowed_dirs` is **required**, and an empty list refuses every export. Every other Arc write path is confined to the storage root by its storage backend, but a USB mount is outside that root by definition — so this is the only thing bounding where an operator-supplied path can land.
* Paths are resolved through symlinks **before** the check, so a link inside an allowed directory cannot point outside it.
* Containment is compared at a path-segment boundary, so `/mnt/usb-other` is not treated as inside `/mnt/usb`.
* Arc **refuses to export into its own storage root**. The next discovery pass would otherwise find the exported copies and queue them for sync, fanning out on every export.
A refused path returns `400` and names the reason.
### Exported is not synced [#exported-is-not-synced]
Exported files move to a distinct ledger state. They leave `pending`, so a later bundle or contact window does not re-send them, but they are **not** `synced` — no hub has confirmed anything yet. `GET /api/v1/spoke-sync/status` reports them separately:
```json
{"pending": 12, "exported": 3, "synced": 40, "pending_bytes": 15728640}
```
`pending_bytes` still includes exported bytes. A file on a drive in transit has not arrived, and excluding it would make the backlog appear to shrink at exactly the moment nothing was delivered.
They advance to `synced` when the hub's acknowledgment comes back — which ships with the import side.
### When a drive does not arrive [#when-a-drive-does-not-arrive]
```bash
curl -X POST https://edge.local:8000/api/v1/spoke-sync/export/06FXVSQXJ2C0EBDFDQ9D24S1E8/revert \
-H "Authorization: Bearer $ARC_TOKEN"
```
Returns just that bundle's files to `pending`, so the next bundle or contact window carries them. Other drives in transit are untouched. `GET /api/v1/spoke-sync/ledger` shows each file's `exported_bundle_id`, which is how you find the ID to revert.
### Importing a drive on the hub [#importing-a-drive-on-the-hub]
The other half. Enable it on the hub:
```toml
[edge_sync.import]
enabled = true # default false
allowed_dirs = ["/mnt/usb"] # REQUIRED; an empty list refuses every import
max_files = 10000 # refuses a manifest declaring more
```
Independent of `edge_sync.enabled`. A hub that only ever takes drives exposes **no** network-writable surface — `/api/v1/sync/file` returns 404. A hub that does both enables both.
```bash
curl -X POST https://hub.example.com/api/v1/bundle-import \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{"path": "/mnt/usb/bundle-submarine-01-06FXW4H1BHR3XHWK2J826G28JG"}'
```
```json
{
"imported": true,
"bundle_id": "06FXW4H1BHR3XHWK2J826G28JG",
"spoke_id": "submarine-01",
"committed": 5,
"already_present": 0,
"bytes_written": 5475,
"conflicts": []
}
```
**Nothing is committed until the whole bundle verifies** — the MAC, `entries.jsonl`'s hash, the canonical digest, every file's size and SHA-256, and the absence of any undeclared file (`ack.json` excepted on a returned drive; it carries its own signature). A tampered drive is refused and not one byte reaches storage.
| Situation | Response |
| --------------------------------------------------------- | ---------------------------------------------- |
| Already imported | `409`, with when it arrived and how many files |
| Tampered, truncated, wrong hub, unknown or disabled spoke | `422`, naming what failed |
| Path outside `allowed_dirs` | `400` |
A **refused** bundle is never recorded as imported, so a corrected drive still works.
Conflicts are reported in full and never overwritten, exactly as on the network path: the same path holding different content means a spoke-ID collision or corruption, and one of the two copies is wrong.
### Why a duplicate drive is refused [#why-a-duplicate-drive-is-refused]
Replay protection here is a **dedup ledger**, not a timestamp window. The online endpoints bind a nonce and a five-minute freshness check, which works because a request is in flight. A bundle legitimately sits on a drive for weeks, so the hub records every import keyed `(spoke_id, bundle_id)` instead — durable state that survives a restart, as a nonce cache does not.
Keyed by spoke as well as bundle, so a compromised spoke cannot burn IDs in another spoke's namespace and block its future drives.
There is no cleanup job: 200 spokes shipping weekly for five years is about 52,000 rows.
### Import history [#import-history]
```bash
curl https://hub.example.com/api/v1/bundle-import/history/submarine-01 \
-H "Authorization: Bearer $ARC_TOKEN"
```
Answers "did last month's drive ever arrive?" — which, on a link with no telemetry, nothing else can.
### Cluster mode [#cluster-mode]
Manifest registrations are **batched at 1000 operations per Raft proposal**. The network path is naturally rate-limited to one proposal per HTTP request, but an import is a tight loop: a 2,500-file bundle costs 3 proposals rather than 2,500.
If a batch fails, the import aborts rather than continuing — a Raft quorum loss is not transient. The bundle is not recorded, so re-importing re-registers everything.
### The return leg: Acknowledgments [#the-return-leg-acknowledgments]
On a successful import the hub writes a signed `ack.json` **into the bundle directory**, so the drive going back carries its own receipt and it cannot be separated from the bundle it answers.
Plug the drive back into the spoke:
```bash
curl -X POST https://edge.local:8000/api/v1/spoke-sync/ack \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{"path": "/mnt/usb/bundle-submarine-01-06FXWFA2NYJHJJFAJAXBDV4PKC"}'
```
```json
{
"applied": true,
"bundle_id": "06FXWFA2NYJHJJFAJAXBDV4PKC",
"hub_id": "shore-station",
"imported_at": "2026-08-07T21:59:56Z",
"synced": 4,
"conflicts": []
}
```
Those files move from `exported` to `synced`. **This is what makes them prunable** — without it `synced` is unreachable on an air-gapped spoke, so the ledger grows forever on the box least able to receive a site visit.
A full cycle looks like this:
```bash
after export: pending 0 exported 4 synced 0
after import: (hub holds 4 files, writes ack.json to the drive)
after ack: pending 0 exported 0 synced 4
```
### What the acknowledgment guarantees [#what-the-acknowledgment-guarantees]
The ack is signed with the **same per-spoke secret** the spoke signs bundles with. The secret is symmetric, so the key that lets a spoke prove authorship lets the hub prove receipt — no new key material to distribute.
The spoke **recomputes** the path digest rather than trusting the one in the file. The MAC binds the digest, so a tampered path list carrying a stale digest would otherwise validate and license marking files synced that the hub never received.
Refused, with a `400`:
* an ack signed by a different hub, or naming a different spoke
* any path added to or removed from the acknowledged list
* a changed MAC, bundle ID, or import time
* an acknowledged path that escapes the spoke's namespace
**Conflicted paths are not acknowledged.** A conflict means the hub holds *different* content at that path, so your copy was never delivered — those entries stay `exported` and are reported for you to look at.
Re-applying an ack is harmless: already-synced entries are a no-op, so a drive plugged in twice changes nothing. A bundle that has not yet been to the hub returns `{"applied": false, "reason": "this bundle carries no acknowledgment yet"}` rather than an error.
Like the bundle it answers, an ack carries **no freshness window** — it rides the same drive back, over the same weeks.
### Air-gap limitations [#air-gap-limitations]
* **Bundles are signed, not encrypted.** The manifest gives integrity and authenticity; the Parquet files themselves are readable by anyone holding the drive. Air-gap media is normally handled under physical controls, but if that does not hold for your deployment, encrypt the drive.
# Advanced (/arc/advanced)
These pages explain mechanisms rather than tasks — useful when tuning a deployment or working out why Arc behaved the way it did.
# Write-Ahead Log (WAL) (/arc/advanced/wal)
Arc's Write-Ahead Log (WAL) provides **zero data loss guarantees** on system crashes.
There are **two different WAL features** in Arc:
1. **SQLite WAL mode** (always enabled) - Internal mode for Arc's metadata database (`arc.db`). This enables concurrent access to connection settings, export jobs, and compaction locks. You'll see the log message `"SQLite WAL mode enabled for concurrent access"` on startup - this is expected and not related to data ingestion.
2. **Arc's WAL feature** (disabled by default) - Optional durability feature for **data ingestion** that provides zero data loss guarantees. This page documents the Arc WAL feature, controlled by the `WAL_ENABLED` environment variable.
**TL;DR**: The startup log `"SQLite WAL mode enabled"` is normal and does NOT mean Arc's data ingestion WAL is enabled.
## Overview [#overview]
WAL is an optional durability feature that persists all incoming data to disk **before** acknowledging writes. When enabled, Arc guarantees that data can be recovered even if the instance crashes.
WAL is **disabled by default** to maximize throughput. Enable it when zero data loss is required.
### When to enable WAL [#when-to-enable-wal]
Enable WAL if you need:
* **Zero data loss** on system crashes
* **Guaranteed durability** for regulatory compliance (finance, healthcare)
* **Recovery from unexpected failures** (power loss, OOM kills)
Keep WAL disabled if you:
* **Prioritize maximum throughput**
* **Can tolerate 0-5 seconds data loss** on rare crashes
* **Have client-side retry logic** or message queue upstream
### Performance vs durability tradeoff [#performance-vs-durability-tradeoff]
| Configuration | Throughput | Data Loss Risk |
| -------------------- | -------------- | ------------------------------------------------- |
| **No WAL (default)** | Highest | 0-5 seconds |
| **WAL enabled** | Somewhat lower | \<1 second (async) / near-zero (fdatasync, fsync) |
**Tradeoff**: a modest throughput reduction for near-zero data loss.
The cost is in enabling the WAL at all, not in the sync mode. Syncs are batched
on a 100 ms ticker rather than performed per write, so all three modes land
within measurement noise of each other on throughput — earlier figures showing
`fdatasync` as *slower* than `fsync` reflected exactly that noise. Choose the
sync mode for the durability semantics you need, not for speed.
## Architecture [#architecture]
### Data flow with WAL [#data-flow-with-wal]
```text
┌──────────────────────────────────────────────────────────┐
│ HTTP Request (MessagePack or Line Protocol) │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 1. WAL.append(records) │
│ - Serialize to MessagePack binary │
│ - Calculate CRC32 checksum │
│ - Write to disk │
│ - fdatasync() ← Force physical disk sync │
└──────────────────┬───────────────────────────────────────┘
│
▼ Data is DURABLE (on disk)
┌──────────────────────────────────────────────────────────┐
│ 2. HTTP 202 Accepted ← Response to client │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 3. Buffer.write(records) │
│ - Add to in-memory buffer │
│ - Flush when 50K records or 5 seconds │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 4. Parquet Writer │
│ - Convert to Arrow columnar format │
│ - Write Parquet file │
│ - Upload to S3/MinIO │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 5. WAL.mark_completed() ← Can now delete WAL entry │
└──────────────────────────────────────────────────────────┘
```
Once WAL confirms the write (step 1), the data is **guaranteed durable** even if Arc crashes before step 4 completes.
### WAL files [#wal-files]
Arc uses a single WAL writer with goroutines for concurrent access:
```text
./data/wal/
├── arc-20251008_140530.wal
└── arc-20251008_150530.wal
```
**Benefits:**
* Simple implementation
* Automatic rotation
* Parallel recovery on startup
## Configuration [#configuration]
### Enable WAL [#enable-wal]
Edit `arc.toml`:
```toml
[wal]
enabled = true
sync_mode = "fdatasync" # Recommended for production
directory = "./data/wal"
max_size_mb = 500 # Rotate at 500MB
max_age_seconds = 3600 # Rotate after 1 hour
```
Or via environment variables:
```ini
ARC_WAL_ENABLED=true
ARC_WAL_DIRECTORY=./data/wal
ARC_WAL_SYNC_MODE=fdatasync
ARC_WAL_MAX_SIZE_MB=500
ARC_WAL_MAX_AGE_SECONDS=3600
```
### Sync modes [#sync-modes]
Arc supports three sync modes with different durability/performance tradeoffs:
#### fdatasync (recommended) [#fdatasync-recommended]
```toml
[wal]
sync_mode = "fdatasync"
```
**How it works:**
* Syncs data to disk (file contents)
* Skips the metadata-only journal flush (timestamps)
* Marginally cheaper than `fsync` at equal durability for Arc's access pattern
**Guarantees:**
* Data is on physical disk
* Can recover all data on crash
**Platform support:** `fdatasync(2)` is used on **Linux**. macOS and Windows do
not expose it, so Arc falls back to a full `fsync` there and logs this once at
startup:
```bash
fdatasync is unavailable on this platform; using full fsync instead
```
The WAL startup log also reports `fdatasync_supported`, so the effective
behavior is visible without guessing.
**Do not expect a throughput change from this setting.** Arc does not sync per
write — a background writer batches appends and syncs on a 100 ms ticker (or
after `sync_bytes`), so there are at most \~10 syncs per second regardless of
ingest rate. The sync mode determines *durability semantics*, not throughput.
On Linux the saving over `fsync` is the inode's timestamp metadata; because the
WAL file grows on every append, the changed file size must still be persisted,
so the practical difference is small — larger on rotational or network-backed
storage than on NVMe.
On macOS, `Sync()` maps to `fsync(2)`, which does not force the drive's own
write cache to flush (that requires `F_FULLFSYNC`). Durability guarantees are
softer on macOS regardless of the selected sync mode — relevant for local
development, not for Linux production deployments.
**Use case**: Production deployments (recommended)
#### fsync (maximum safety) [#fsync-maximum-safety]
```toml
[wal]
sync_mode = "fsync"
```
**How it works:**
* Syncs both data AND metadata to disk
* Slowest, but absolute guarantee
**Use when:**
* Regulatory compliance requires it
* Zero tolerance for any data loss
* Performance is secondary
#### async (performance-first) [#async-performance-first]
```toml
[wal]
sync_mode = "async"
```
**How it works:**
* Writes to OS buffer cache
* No explicit sync (OS flushes periodically)
* Very fast, but small risk window
**Use when:**
* Need 90% of original throughput
* Can tolerate \~1 second data loss
* Have upstream retry mechanisms
### Rotation settings [#rotation-settings]
Control when WAL files rotate:
```toml
[wal]
max_size_mb = 100 # Rotate when file reaches 100MB
max_age_seconds = 3600 # Rotate after 1 hour (even if file is small)
```
**Why rotation matters:**
* Prevents unbounded growth
* Faster recovery (smaller files)
* Automatic cleanup of old WALs
## Operations [#operations]
### Recovery on startup [#recovery-on-startup]
Arc automatically recovers from WAL files on startup:
```text
2025-10-08 14:30:00 [INFO] WAL recovery started: 4 files
2025-10-08 14:30:01 [INFO] Recovering WAL: worker-1-20251008_143000.wal
2025-10-08 14:30:01 [INFO] WAL read complete: 1000 entries, 5242880 bytes, 0 corrupted
2025-10-08 14:30:02 [INFO] Recovering WAL: worker-2-20251008_143000.wal
...
2025-10-08 14:30:05 [INFO] WAL recovery complete: 4000 batches, 200000 entries, 0 corrupted
2025-10-08 14:30:05 [INFO] WAL archived: worker-1-20251008_143000.wal.recovered
```
**Process:**
1. Find all `*.wal` files in `WAL_DIR`
2. Read and validate each entry (checksum verification)
3. Replay records into buffer system
4. Archive recovered WAL as `*.wal.recovered`
5. Continue normal operations
**Recovery time:**
* \~5 seconds per 100MB WAL file
* Parallel recovery across workers
* Corrupted entries are skipped (logged)
## Monitoring [#monitoring]
### WAL status [#wal-status]
```bash
curl http://localhost:8000/api/wal/status \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response:**
```json
{
"enabled": true,
"configuration": {
"sync_mode": "fdatasync",
"worker_id": 1,
"current_file": "./data/wal/worker-1-20251008_143000.wal"
},
"stats": {
"current_size_mb": 45.2,
"current_age_seconds": 1850,
"total_entries": 5000,
"total_bytes": 47382528,
"total_syncs": 5000,
"total_rotations": 2
}
}
```
### WAL files [#wal-files-1]
```bash
curl http://localhost:8000/api/wal/files \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response:**
```json
{
"active": [
{
"name": "worker-1-20251008_143000.wal",
"size_mb": 45.2,
"modified": 1696775400
}
],
"recovered": [
{
"name": "worker-1-20251008_120000.wal.recovered",
"size_mb": 98.5,
"modified": 1696768800
}
],
"total_size_mb": 143.7
}
```
### Health check [#health-check]
```bash
curl http://localhost:8000/api/wal/health \
-H "Authorization: Bearer $ARC_TOKEN"
```
### Cleanup old WAL files [#cleanup-old-wal-files]
```bash
# Cleanup files older than 24 hours (default)
curl -X POST http://localhost:8000/api/wal/cleanup \
-H "Authorization: Bearer $ARC_TOKEN"
# Custom age (in hours)
curl -X POST "http://localhost:8000/api/wal/cleanup?max_age_hours=48" \
-H "Authorization: Bearer $ARC_TOKEN"
```
## Troubleshooting [#troubleshooting]
### WAL recovery taking too long [#wal-recovery-taking-too-long]
**Symptoms:**
```text
2025-10-08 14:30:00 [INFO] WAL recovery started: 50 files
... (minutes pass) ...
```
**Solutions:**
1. **Adjust rotation settings:**
```toml
[wal]
max_size_mb = 50 # Smaller files, faster recovery
max_age_seconds = 1800 # Rotate more frequently
```
2. **Use faster disks for WAL:**
```toml
[wal]
directory = "/mnt/nvme/arc-wal" # NVMe SSD
```
3. **Use faster storage:**
* NVMe SSD for WAL directory
* Separate disk from data storage
### WAL disk space growing [#wal-disk-space-growing]
**Symptoms:**
```bash
$ du -sh ./data/wal
5.2G ./data/wal
```
**Solutions:**
1. **Manual cleanup:**
```bash
rm -f ./data/wal/*.wal.recovered
```
2. **Reduce retention:**
```toml
[wal]
max_size_mb = 50 # Rotate sooner
max_age_seconds = 1800 # 30 minutes
```
3. **Add cron job for cleanup:**
```bash
# Cleanup recovered WALs older than 24 hours
0 2 * * * find /path/to/data/wal -name "*.wal.recovered" -mtime +1 -delete
```
### WAL write failures [#wal-write-failures]
**Symptoms:**
```text
2025-10-08 14:30:00 [ERROR] WAL append failed: [Errno 28] No space left on device
```
**Solutions:**
1. **Check disk space:**
```bash
df -h /path/to/WAL_DIR
```
2. **Check permissions:**
```bash
ls -ld ./data/wal
chmod 755 ./data/wal
```
3. **Move WAL to larger disk:**
```toml
[wal]
directory = "/mnt/large-disk/arc-wal"
```
### Performance degradation with WAL [#performance-degradation-with-wal]
**Symptoms:**
* Throughput dropped sharply after enabling WAL
* High CPU usage from fsync calls
**Solutions:**
1. **Do not expect the sync mode to be the cause.** Syncs are batched on a
100 ms ticker rather than performed per write, so `fsync`, `fdatasync` and
`async` land within noise of one another. Switching modes to chase
throughput will not help — look at disk I/O and WAL placement below.
2. **Check disk I/O wait:**
```bash
iostat -x 1
# Look for %iowait > 50%
```
3. **Move WAL to faster disk:**
```toml
[wal]
directory = "/mnt/nvme/arc-wal"
```
4. **Consider disabling WAL if durability isn't critical:**
```toml
[wal]
enabled = false
```
## Best practices [#best-practices]
### Production deployment [#production-deployment]
**Recommended configuration:**
```toml
[wal]
enabled = true
sync_mode = "fdatasync"
directory = "/mnt/fast-ssd/arc-wal"
max_size_mb = 100
max_age_seconds = 3600
```
**Monitoring setup:**
1. Monitor WAL disk usage
2. Alert on write failures
3. Track recovery time during restarts
4. Log rotation metrics
**Backup strategy:**
* WAL files are ephemeral (deleted after recovery)
* Don't backup WAL files directly
* Backup final Parquet files in S3/MinIO instead
### Development/testing [#developmenttesting]
**Recommended configuration:**
```toml
[wal]
enabled = false # WAL disabled for maximum speed
```
**Or if testing WAL:**
```toml
[wal]
enabled = true
sync_mode = "async"
max_size_mb = 10 # Small files for testing
```
## Summary [#summary]
**Enable WAL if:**
* Zero data loss is required
* Regulated industry (finance, healthcare)
* Can accept 19% throughput reduction
**Disable WAL if:**
* Maximum throughput is priority
* Can tolerate 0-5s data loss risk
* Have upstream retry/queue mechanisms
**Recommended settings:**
```toml
[wal]
enabled = true
sync_mode = "fdatasync" # Best balance
directory = "/mnt/nvme/arc-wal" # Fast disk
```
## Next steps [#next-steps]
* **[Configure Compaction](/arc/advanced/compaction/)** - Optimize query performance
* **[Monitor Arc](/arc/operations/telemetry/)** - Set up health checks
* **[Configuration Reference](/arc/configuration/overview/)** - Maximize throughput
# API Reference (/arc/api-reference)
Arc exposes one HTTP API for writes, queries, and administration. The reference below covers the endpoints, their payloads, and how to authenticate.
# REST API overview (/arc/api-reference/overview)
Arc provides a comprehensive REST API for data ingestion, querying, and management.
## Base URL [#base-url]
```text
http://localhost:8000
```
## Authentication [#authentication]
All endpoints (except public ones) require authentication. Arc supports multiple authentication methods for compatibility with various clients:
### Bearer token (standard) [#bearer-token-standard]
```bash
curl -H "Authorization: Bearer $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### Token header (InfluxDB 2.x style) [#token-header-influxdb-2x-style]
```bash
curl -H "Authorization: Token $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### API key header [#api-key-header]
```bash
curl -H "x-api-key: $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### Query parameter (InfluxDB 1.x style) [#query-parameter-influxdb-1x-style]
For InfluxDB 1.x client compatibility, tokens can be passed via the `p` query parameter:
```bash
curl "http://localhost:8000/write?db=mydb&p=$ARC_TOKEN" -d 'cpu,host=server01 usage=45.2'
```
### Public endpoints (no auth required) [#public-endpoints-no-auth-required]
* `GET /health` - Health check
* `GET /ready` - Readiness probe
* `GET /metrics` - Prometheus metrics
* `GET /api/v1/auth/verify` - Token verification
## Quick examples [#quick-examples]
### Write data (MessagePack) [#write-data-messagepack]
```python
import os
import msgpack
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
data = {
"m": "cpu",
"columns": {
"time": [1697472000000],
"host": ["server01"],
"usage": [45.2]
}
}
response = requests.post(
"http://localhost:8000/api/v1/write/msgpack",
headers={
"Authorization": f"Bearer {ARC_TOKEN}",
"Content-Type": "application/msgpack",
"x-arc-database": "default"
},
data=msgpack.packb(data)
)
```
### Query data (JSON) [#query-data-json]
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM default.cpu LIMIT 10", "format": "json"}'
```
### Query data (Apache Arrow) [#query-data-apache-arrow]
For large result sets, use Arrow format:
```python
import os
import requests
import pyarrow as pa
ARC_TOKEN = os.environ["ARC_TOKEN"]
response = requests.post(
"http://localhost:8000/api/v1/query/arrow",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"sql": "SELECT * FROM default.cpu LIMIT 100000"}
)
reader = pa.ipc.open_stream(response.content)
arrow_table = reader.read_all()
```
### Query data (MessagePack) [#query-data-messagepack]
MessagePack is the best general-purpose binary format for clients: it is
columnar, carries per-column type names, supports `SHOW` statements, and honors
`Accept-Encoding` — noticeably faster end-to-end than JSON on large result
sets. Arrow is faster still for raw throughput, but does not accept `SHOW` and
has no response compression.
```python
import os
import requests
import msgpack
ARC_TOKEN = os.environ["ARC_TOKEN"]
response = requests.post(
"http://localhost:8000/api/v1/query/msgpack",
headers={
"Authorization": f"Bearer {ARC_TOKEN}",
"Accept-Encoding": "zstd", # optional, typically ~40% smaller
},
json={"sql": "SELECT * FROM default.cpu LIMIT 100000"}
)
result = msgpack.unpackb(response.content, raw=False)
# NOTE: "data" is COLUMNAR — an array of columns, not an array of rows.
for name, type_name, column in zip(result["columns"], result["types"], result["data"]):
print(name, type_name, column[:5])
```
The response is a single MessagePack map:
| Field | Meaning |
| ------------------- | ------------------------------------------------ |
| `success` | bool |
| `columns` | column names |
| `types` | wire type name per column, parallel to `columns` |
| `data` | **array of columns**, each an array of values |
| `row_count` | number of rows |
| `execution_time_ms` | server-side execution time |
| `timestamp` | RFC3339, UTC |
| `profile` | present only when `x-arc-profile: true` |
#### Type vocabulary [#type-vocabulary]
The `types` values are a stable contract. Scalars are
`bool`, `int8`/`int16`/`int32`/`int64`, `uint8`/`uint16`/`uint32`/`uint64`,
`float32`/`float64`, `utf8`, `large_utf8`, `binary`, `large_binary`, `date32`,
and `null`. Timestamps carry their unit, e.g. `timestamp[us]`.
Three values tell you a column is **not** natively typed on the wire:
* `string_encoded` — a recognized type Arc transmits as text (`DATE64`, `TIME`,
`INTERVAL`, `DURATION`, `FLOAT16`, fixed-size binary).
* `list` / `struct` / `map` — nested values, transmitted as text.
* `unknown:` — a type with no published Arc name (an engine-level `ENUM`
arrives here). Never bind to the text after the prefix; it is diagnostic.
`SUM(int_col)` and other decimal-producing aggregates are normalized to
`int64` (scale 0) or `float64` (scaled), matching the Arrow endpoint — the
values are numbers, not strings.
### Health check [#health-check]
```bash
curl http://localhost:8000/health
```
***
## Health & monitoring [#health--monitoring]
### GET /health [#get-health]
Health check endpoint.
**Response:**
```json
{
"status": "ok",
"time": "2024-12-02T10:30:00Z",
"uptime": "1h 23m 45s",
"uptime_sec": 5025
}
```
### GET /ready [#get-ready]
Kubernetes readiness probe.
**Response:**
```json
{
"status": "ready",
"time": "2024-12-02T10:30:00Z",
"uptime_sec": 5025
}
```
### GET /metrics [#get-metrics]
Prometheus-format metrics.
**Response:** `text/plain` (Prometheus format)
Or request JSON:
```bash
curl -H "Accept: application/json" http://localhost:8000/metrics
```
### GET /api/v1/metrics [#get-apiv1metrics]
All metrics in JSON format.
### GET /api/v1/metrics/memory [#get-apiv1metricsmemory]
Detailed memory statistics including the Go runtime and the query engine.
### GET /api/v1/metrics/query-pool [#get-apiv1metricsquery-pool]
Query engine connection pool statistics.
### GET /api/v1/metrics/endpoints [#get-apiv1metricsendpoints]
Per-endpoint request statistics.
### GET /api/v1/metrics/timeseries/:type [#get-apiv1metricstimeseriestype]
Timeseries metrics data.
**Parameters:**
* `:type` - `system`, `application`, or `api`
* `?duration_minutes=30` - Time range (default: 30, max: 1440)
### GET /api/v1/logs [#get-apiv1logs]
Recent application logs. **Requires an admin token** (`Authorization: Bearer `) when authentication is enabled.
**Query Parameters:**
* `?limit=100` - Number of logs (default: 100, max: 1000)
* `?level=error` - Filter by level (error, warn, info, debug)
* `?since_minutes=60` - Time range (default: 60, max: 1440)
***
## Data ingestion [#data-ingestion]
### POST /api/v1/write/msgpack [#post-apiv1writemsgpack]
High-performance MessagePack binary writes (recommended).
**Headers:**
* `Authorization: Bearer TOKEN`
* `Content-Type: application/msgpack`
* `Content-Encoding: gzip` (optional)
* `x-arc-database: default` (optional)
**Body (MessagePack):**
```json
{
"m": "measurement_name",
"columns": {
"time": [1697472000000, 1697472001000],
"host": ["server01", "server02"],
"value": [45.2, 67.8]
}
}
```
**Response:** `204 No Content`
**Null values:** Columns may contain nulls, including a column whose values are
*all* null in a given batch. Arc keeps such a column and stores every value as
NULL, so it is queryable and returns NULLs rather than failing to resolve:
```python
# 'depth' is null for this entire batch — the column is still created
{"m": "sensors", "columns": {
"time": [1697472000000, 1697472001000],
"value": [1.5, 2.5],
"depth": [None, None],
}}
```
A later batch carrying real values for that column determines its type
normally; the all-null batch does not pin it. Column types only need to be
consistent within a single write, not across writes — reads union columns by
name across files.
The `time` column is the one exception: it must be a numeric epoch on every
row. A null, string, or otherwise non-numeric `time` is **rejected** with
`400 Bad Request` rather than stored, because a non-timestamp `time` column
makes the affected partition un-compactable.
### GET /api/v1/write/msgpack/stats [#get-apiv1writemsgpackstats]
MessagePack ingestion statistics.
### GET /api/v1/write/msgpack/spec [#get-apiv1writemsgpackspec]
MessagePack format specification.
### POST /write [#post-write]
InfluxDB 1.x Line Protocol compatible endpoint. This path matches InfluxDB's native API for drop-in client compatibility.
**Query Parameters:**
* `db` - Target database name (required)
* `rp` - Retention policy (optional, ignored)
* `precision` - Timestamp precision: `ns`, `us`, `ms`, `s` (default: `ns`)
* `p` - Authentication token (InfluxDB 1.x style)
**Headers:**
* `Content-Type: text/plain`
* `Authorization: Bearer TOKEN` (or use `p` query param)
**Body:**
```text
cpu,host=server01 usage=45.2 1697472000000000000
mem,host=server01 used=8.2,total=16.0 1697472000000000000
```
**Example:**
```bash
curl -X POST "http://localhost:8000/write?db=mydb&p=$ARC_TOKEN" \
-d 'cpu,host=server01 usage=45.2'
```
### POST /api/v2/write [#post-apiv2write]
InfluxDB 2.x compatible endpoint. This path matches InfluxDB's native API for drop-in client compatibility.
**Query Parameters:**
* `bucket` - Target database/bucket name (required)
* `org` - Organization (optional, ignored)
* `precision` - Timestamp precision: `ns`, `us`, `ms`, `s` (default: `ns`)
**Headers:**
* `Content-Type: text/plain`
* `Authorization: Token $ARC_TOKEN` (InfluxDB 2.x style)
**Example:**
```bash
curl -X POST "http://localhost:8000/api/v2/write?bucket=mydb&org=myorg" \
-H "Authorization: Token $ARC_TOKEN" \
-d 'cpu,host=server01 usage=45.2'
```
### POST /api/v1/write/line-protocol [#post-apiv1writeline-protocol]
Arc-native Line Protocol endpoint. Uses headers instead of query parameters.
**Headers:**
* `Content-Type: text/plain`
* `Authorization: Bearer TOKEN`
* `x-arc-database: default` - Target database
### POST /api/v1/write/line-protocol/flush [#post-apiv1writeline-protocolflush]
Force buffer flush to disk.
### GET /api/v1/write/line-protocol/stats [#get-apiv1writeline-protocolstats]
Line Protocol ingestion statistics.
### GET /api/v1/write/line-protocol/health [#get-apiv1writeline-protocolhealth]
Line Protocol handler health.
### POST /api/v1/write/tle [#post-apiv1writetle]
Stream TLE (Two-Line Element) satellite orbital data. Parses TLE entries into tags (NORAD ID, name, classification) and fields (orbital elements + derived metrics).
**Headers:**
* `Authorization: Bearer TOKEN`
* `X-Arc-Database: satellites` (default: `default`)
* `X-Arc-Measurement: satellite_tle` (default: `satellite_tle`)
```bash
curl -X POST "http://localhost:8000/api/v1/write/tle" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: satellites" \
--data-binary @stations.tle
```
Returns `204 No Content` on success.
See [TLE Integration](/arc/integrations/tle/) for full documentation including schema, format details, and example queries.
### GET /api/v1/write/tle/stats [#get-apiv1writetlestats]
TLE handler statistics.
***
## Data import [#data-import]
Bulk import endpoints for CSV, Parquet, Line Protocol, and TLE files. All endpoints use `multipart/form-data` with field name `file`, support gzip auto-detection, and enforce a 500 MB size limit.
### POST /api/v1/import/csv [#post-apiv1importcsv]
Bulk import a CSV file. See [CSV Import](/arc/data-import/csv/) for full documentation.
```bash
curl -X POST "http://localhost:8000/api/v1/import/csv?measurement=sensors" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: iot" \
-F "file=@data.csv"
```
### POST /api/v1/import/parquet [#post-apiv1importparquet]
Bulk import a Parquet file. See [Parquet Import](/arc/data-import/parquet/) for full documentation.
```bash
curl -X POST "http://localhost:8000/api/v1/import/parquet?measurement=metrics" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: production" \
-F "file=@data.parquet"
```
### POST /api/v1/import/lp [#post-apiv1importlp]
Bulk import a Line Protocol file. See [Line Protocol Bulk Import](/arc/data-import/line-protocol/) for full documentation.
```bash
curl -X POST "http://localhost:8000/api/v1/import/lp" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: mydb" \
-F "file=@export.lp"
```
### POST /api/v1/import/tle [#post-apiv1importtle]
Bulk import a TLE file. See [TLE Integration](/arc/integrations/tle/) for full documentation.
```bash
curl -X POST "http://localhost:8000/api/v1/import/tle" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: satellites" \
-F "file=@catalog.tle"
```
### GET /api/v1/import/stats [#get-apiv1importstats]
Import handler statistics (total requests, records imported, errors).
***
## Querying [#querying]
### POST /api/v1/query [#post-apiv1query]
Execute SQL queries with JSON response.
**Request:**
```json
{
"sql": "SELECT * FROM default.cpu LIMIT 10",
"format": "json"
}
```
**Response:**
```json
{
"columns": ["time", "host", "usage"],
"types": ["TIMESTAMP", "VARCHAR", "DOUBLE"],
"data": [
[1697472000000, "server01", 45.2],
[1697472001000, "server02", 67.8]
],
"row_count": 2,
"execution_time_ms": 12
}
```
### POST /api/v1/query/arrow [#post-apiv1queryarrow]
Execute SQL queries with Apache Arrow IPC response.
**Request:**
```json
{
"sql": "SELECT * FROM default.cpu LIMIT 10000"
}
```
**Response:** `application/vnd.apache.arrow.stream`
**Optional stream encodings (v26.09.1+)**, opted in per request — use them for
network-constrained clients pulling large result sets; leave them off for
same-host consumers (they trade CPU for wire bytes):
| Header | Effect |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `x-arc-arrow-dictionary: true` | Dictionary-encode low-cardinality string columns (adaptive, first-batch analysis). Columns arrive as standard Arrow dictionary arrays — pyarrow, polars, and pandas read them transparently. |
| `x-arc-arrow-compression: zstd` (or `lz4`) | Arrow IPC buffer compression, decompressed natively by Arrow clients. |
Combined, these halved wire size on a 500M-row benchmark (39 → 19.4 bytes/row).
### Response compression (v26.09.1+) [#response-compression-v26091]
`POST /api/v1/query` (JSON) and `POST /api/v1/query/msgpack` honor the
standard `Accept-Encoding` request header — send `Accept-Encoding: zstd, gzip`
and the response body is compressed (zstd preferred; `Content-Encoding` set
accordingly). curl, browsers, and HTTP libraries negotiate this
automatically. Typical savings: JSON −68%, msgpack −38% on large results.
Clients that omit the header get identical responses to previous versions.
### POST /api/v1/query/estimate [#post-apiv1queryestimate]
Estimate query cost before execution.
**Request:**
```json
{
"sql": "SELECT * FROM default.cpu WHERE time > now() - INTERVAL '1 hour'"
}
```
### GET /api/v1/measurements [#get-apiv1measurements]
List all measurements across databases.
### GET /api/v1/query/:measurement [#get-apiv1querymeasurement]
Query a specific measurement directly.
***
## Authentication [#authentication-1]
### GET /api/v1/auth/verify [#get-apiv1authverify]
Verify token validity (public endpoint).
**Response:**
```json
{
"valid": true,
"token_id": "abc123",
"name": "my-token",
"is_admin": false
}
```
### GET /api/v1/auth/tokens [#get-apiv1authtokens]
List all tokens (admin only).
### POST /api/v1/auth/tokens [#post-apiv1authtokens]
Create a new token (admin only).
**Request:**
```json
{
"name": "my-service",
"description": "Token for my service",
"is_admin": false
}
```
**Response:**
```json
{
"id": "abc123",
"name": "my-service",
"token": "arc_xxxxxxxxxxxxxxxxxxxxxxxx",
"is_admin": false,
"created_at": "2024-12-02T10:30:00Z"
}
```
### GET /api/v1/auth/tokens/:id [#get-apiv1authtokensid]
Get token details (admin only).
### DELETE /api/v1/auth/tokens/:id [#delete-apiv1authtokensid]
Delete/revoke a token (admin only).
### POST /api/v1/auth/tokens/:id/rotate [#post-apiv1authtokensidrotate]
Rotate a token (admin only).
### POST /api/v1/auth/tokens/:id/revoke [#post-apiv1authtokensidrevoke]
Revoke a token (admin only).
### GET /api/v1/auth/cache/stats [#get-apiv1authcachestats]
Token cache statistics (admin only).
### POST /api/v1/auth/cache/invalidate [#post-apiv1authcacheinvalidate]
Invalidate token cache (admin only).
***
## Compaction [#compaction]
### GET /api/v1/compaction/status [#get-apiv1compactionstatus]
Current compaction status.
**Response:**
```json
{
"enabled": true,
"running": false,
"last_run": "2024-12-02T10:00:00Z",
"next_run": "2024-12-02T11:00:00Z"
}
```
### GET /api/v1/compaction/stats [#get-apiv1compactionstats]
Compaction statistics.
### GET /api/v1/compaction/candidates [#get-apiv1compactioncandidates]
List files eligible for compaction.
### POST /api/v1/compaction/trigger [#post-apiv1compactiontrigger]
Manually trigger compaction.
**Request:**
```json
{
"database": "default",
"measurement": "cpu"
}
```
### GET /api/v1/compaction/jobs [#get-apiv1compactionjobs]
List active compaction jobs.
### GET /api/v1/compaction/history [#get-apiv1compactionhistory]
Compaction job history.
***
## Delete operations [#delete-operations]
### POST /api/v1/delete [#post-apiv1delete]
Delete data matching conditions.
**Request:**
```json
{
"database": "default",
"measurement": "cpu",
"where": "host = 'server01' AND time < '2024-01-01'",
"confirm": true
}
```
**Response:**
```json
{
"deleted_rows": 1523,
"deleted_files": 3
}
```
### GET /api/v1/delete/config [#get-apiv1deleteconfig]
Get delete operation configuration.
***
## Database management [#database-management]
Endpoints for managing databases programmatically.
### GET /api/v1/databases [#get-apiv1databases]
List all databases with measurement counts.
**Response:**
```json
{
"databases": [
{"name": "default", "measurement_count": 5},
{"name": "production", "measurement_count": 12}
],
"count": 2
}
```
### POST /api/v1/databases [#post-apiv1databases]
Create a new database.
**Request:**
```json
{
"name": "my_database"
}
```
**Response (201 Created):**
```json
{
"name": "my_database",
"measurement_count": 0,
"created_at": "2024-12-21T10:30:00Z"
}
```
**Validation rules:**
* Must start with a letter (a-z, A-Z)
* Can contain letters, numbers, underscores, and hyphens
* Maximum 64 characters
* Reserved names blocked: `system`, `internal`, `_internal`
**Error Response (400):**
```json
{
"error": "Invalid database name: must start with a letter and contain only alphanumeric characters, underscores, or hyphens"
}
```
### GET /api/v1/databases/:name [#get-apiv1databasesname]
Get information about a specific database.
**Response:**
```json
{
"name": "production",
"measurement_count": 12
}
```
**Error Response (404):**
```json
{
"error": "Database 'nonexistent' not found"
}
```
### GET /api/v1/databases/:name/measurements [#get-apiv1databasesnamemeasurements]
List all measurements in a database.
**Response:**
```json
{
"database": "production",
"measurements": [
{"name": "cpu"},
{"name": "memory"},
{"name": "disk"}
],
"count": 3
}
```
### DELETE /api/v1/databases/:name [#delete-apiv1databasesname]
Delete a database and all its data.
This operation is destructive and cannot be undone. Requires:
- `delete.enabled = true` in configuration
- `?confirm=true` query parameter
**Request:**
```bash
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/databases/old_data?confirm=true"
```
**Response:**
```json
{
"message": "Database 'old_data' deleted successfully",
"files_deleted": 47
}
```
**Error Responses:**
*Delete disabled (403):*
```json
{
"error": "Delete operations are disabled. Set delete.enabled=true in arc.toml to enable."
}
```
*Missing confirmation (400):*
```json
{
"error": "Confirmation required. Add ?confirm=true to delete the database."
}
```
***
## Retention policies [#retention-policies]
### POST /api/v1/retention [#post-apiv1retention]
Create a retention policy.
**Request:**
```json
{
"name": "30-day-retention",
"database": "default",
"measurement": "cpu",
"duration": "30d",
"schedule": "0 2 * * *"
}
```
### GET /api/v1/retention [#get-apiv1retention]
List all retention policies.
### GET /api/v1/retention/:id [#get-apiv1retentionid]
Get a specific policy.
### PUT /api/v1/retention/:id [#put-apiv1retentionid]
Update a retention policy.
### DELETE /api/v1/retention/:id [#delete-apiv1retentionid]
Delete a retention policy.
### POST /api/v1/retention/:id/execute [#post-apiv1retentionidexecute]
Execute a policy manually.
### GET /api/v1/retention/:id/executions [#get-apiv1retentionidexecutions]
Get policy execution history.
***
## Continuous queries [#continuous-queries]
### POST /api/v1/continuous\_queries [#post-apiv1continuous_queries]
Create a continuous query.
**Request:**
```json
{
"name": "hourly-rollup",
"source_database": "default",
"source_measurement": "cpu",
"destination_database": "default",
"destination_measurement": "cpu_hourly",
"query": "SELECT time_bucket('1 hour', time) as time, host, AVG(usage) as avg_usage FROM default.cpu GROUP BY 1, 2",
"schedule": "0 * * * *"
}
```
### GET /api/v1/continuous\_queries [#get-apiv1continuous_queries]
List all continuous queries.
### GET /api/v1/continuous\_queries/:id [#get-apiv1continuous_queriesid]
Get a specific continuous query.
### PUT /api/v1/continuous\_queries/:id [#put-apiv1continuous_queriesid]
Update a continuous query.
### DELETE /api/v1/continuous\_queries/:id [#delete-apiv1continuous_queriesid]
Delete a continuous query.
### POST /api/v1/continuous\_queries/:id/execute [#post-apiv1continuous_queriesidexecute]
Execute a continuous query manually.
### GET /api/v1/continuous\_queries/:id/executions [#get-apiv1continuous_queriesidexecutions]
Get execution history.
***
## MQTT subscriptions [#mqtt-subscriptions]
MQTT subscription management is available starting Arc v26.02.1.
Manage MQTT broker subscriptions for direct IoT data ingestion. See the [MQTT Integration Guide](/arc/integrations/mqtt/) for detailed usage.
### POST /api/v1/mqtt/subscriptions [#post-apiv1mqttsubscriptions]
Create a new MQTT subscription.
**Request:**
```json
{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"qos": 1,
"auto_start": true
}
```
**Response (201 Created):**
```json
{
"id": "sub_abc123",
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"status": "running",
"created_at": "2026-02-01T10:00:00Z"
}
```
**Full options:**
| Field | Type | Required | Default | Description |
| --------------- | --------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------- |
| `name` | string | Yes | - | Unique subscription name |
| `broker` | string | Yes | - | Broker URL (tcp\://, ssl://, ws\://) |
| `topics` | array | Yes | - | Topics to subscribe |
| `database` | string | Yes | - | Target Arc database |
| `qos` | int | No | 1 | QoS level: 0, 1, or 2 |
| `client_id` | string | No | auto | MQTT client ID |
| `username` | string | No | - | MQTT username |
| `password` | string | No | - | MQTT password (encrypted at rest) |
| `tls_enabled` | bool | No | false | Enable TLS/SSL |
| `tls_cert_path` | string | No | - | Client certificate path |
| `tls_key_path` | string | No | - | Client key path |
| `tls_ca_path` | string | No | - | CA certificate path |
| `topic_mapping` | object (`{string: string}`) | No | \{} | Per-topic target-database override (`{"": ""}`); does not configure measurements or tags |
| `auto_start` | bool | No | true | Start on creation and server restart |
### GET /api/v1/mqtt/subscriptions [#get-apiv1mqttsubscriptions]
List all MQTT subscriptions.
**Response:**
```json
{
"subscriptions": [
{
"id": "sub_abc123",
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"status": "running"
}
],
"count": 1
}
```
### GET /api/v1/mqtt/subscriptions/:id [#get-apiv1mqttsubscriptionsid]
Get subscription details.
### PUT /api/v1/mqtt/subscriptions/:id [#put-apiv1mqttsubscriptionsid]
Update a subscription. Subscription must be stopped first.
### DELETE /api/v1/mqtt/subscriptions/:id [#delete-apiv1mqttsubscriptionsid]
Delete a subscription. Subscription must be stopped first.
### POST /api/v1/mqtt/subscriptions/:id/start [#post-apiv1mqttsubscriptionsidstart]
Start a stopped subscription.
**Response:**
```json
{
"id": "sub_abc123",
"status": "running",
"message": "Subscription started"
}
```
### POST /api/v1/mqtt/subscriptions/:id/stop [#post-apiv1mqttsubscriptionsidstop]
Stop a running subscription.
### POST /api/v1/mqtt/subscriptions/:id/restart [#post-apiv1mqttsubscriptionsidrestart]
Restart a subscription (stop + start).
### GET /api/v1/mqtt/subscriptions/:id/stats [#get-apiv1mqttsubscriptionsidstats]
Get statistics for a specific subscription.
**Response:**
```json
{
"id": "sub_abc123",
"messages_received": 15420,
"bytes_received": 2458320,
"decode_errors": 0,
"last_message_at": "2026-02-01T10:30:15Z",
"topics": {
"sensors/temperature": 8500,
"sensors/humidity": 6920
}
}
```
### GET /api/v1/mqtt/stats [#get-apiv1mqttstats]
Aggregate statistics across all running subscriptions.
**Response:**
```json
{
"status": "success",
"running_count": 2,
"subscriptions_stats": {
"sub_abc123": { ... },
"sub_def456": { ... }
}
}
```
### GET /api/v1/mqtt/health [#get-apiv1mqtthealth]
MQTT service health check.
**Response:**
```json
{
"status": "healthy",
"healthy": true,
"running_count": 2,
"connected_count": 2,
"disconnected_count": 0,
"service": "mqtt_subscriptions"
}
```
***
## Backup & restore [#backup--restore]
Admin-only endpoints for backing up and restoring Arc data, metadata, and configuration. Operations run asynchronously with progress tracking.
See [Backup & Restore](/arc/operations/backup-restore/) for full documentation.
| Method | Endpoint | Description |
| -------- | ------------------------ | ------------------------------------------------------- |
| `POST` | `/api/v1/backup` | Trigger a full backup (async, returns 202) |
| `GET` | `/api/v1/backup` | List all available backups |
| `GET` | `/api/v1/backup/status` | Progress of active operation |
| `GET` | `/api/v1/backup/:id` | Get backup manifest |
| `DELETE` | `/api/v1/backup/:id` | Delete a backup |
| `POST` | `/api/v1/backup/restore` | Restore from a backup (async, requires `confirm: true`) |
```bash
# Create backup
curl -X POST "http://localhost:8000/api/v1/backup" \
-H "Authorization: Bearer $ARC_TOKEN"
# Poll progress
curl "http://localhost:8000/api/v1/backup/status" \
-H "Authorization: Bearer $ARC_TOKEN"
# Restore
curl -X POST "http://localhost:8000/api/v1/backup/restore" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"backup_id": "backup-20260211-143022-a1b2c3d4", "confirm": true}'
```
***
## Response formats [#response-formats]
### Success response [#success-response]
```json
{
"status": "success",
"data": [...],
"count": 10
}
```
### Error response [#error-response]
```json
{
"error": "Error message"
}
```
### HTTP status codes [#http-status-codes]
* `200` - Success
* `204` - No Content (successful write)
* `400` - Bad Request
* `401` - Unauthorized
* `403` - Forbidden (requires admin)
* `404` - Not Found
* `500` - Internal Server Error
***
## Rate limiting [#rate-limiting]
Arc does not enforce rate limiting by default. For production deployments, consider:
* Reverse proxy rate limiting (Nginx, Traefik)
* API Gateway (AWS API Gateway, Kong)
* Application-level throttling
## CORS [#cors]
CORS is enabled by default with permissive settings. Configure via reverse proxy for production.
## Best practices [#best-practices]
### 1. Use MessagePack for writes [#1-use-messagepack-for-writes]
MessagePack is considerably faster than Line Protocol:
```python
# Fast: MessagePack columnar
data = {"m": "cpu", "columns": {...}}
requests.post(url, data=msgpack.packb(data))
# Slower: Line Protocol text
data = "cpu,host=server01 usage=45.2"
requests.post(url, data=data)
```
### 2. Batch your writes [#2-batch-your-writes]
Send multiple records per request:
```python
# Good: Batch write
data = {
"m": "cpu",
"columns": {
"time": [t1, t2, t3, ...],
"host": [h1, h2, h3, ...],
"usage": [u1, u2, u3, ...]
}
}
```
### 3. Use Arrow for large queries [#3-use-arrow-for-large-queries]
For 10K+ rows, use the Arrow endpoint:
```python
response = requests.post(url + "/api/v1/query/arrow", ...)
table = pa.ipc.open_stream(response.content).read_all()
df = table.to_pandas() # Zero-copy conversion
```
### 4. Enable gzip compression [#4-enable-gzip-compression]
```python
import gzip
compressed = gzip.compress(msgpack.packb(data))
requests.post(
url,
data=compressed,
headers={"Content-Encoding": "gzip", ...}
)
```
## Client libraries [#client-libraries]
### Python (official SDK) [#python-official-sdk]
```bash
pip install arc-tsdb-client[all]
```
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
client.write.write_columnar(
measurement="cpu",
columns={"time": [...], "host": [...], "usage": [...]},
)
df = client.query.query_pandas("SELECT * FROM default.cpu LIMIT 10")
```
See [Python SDK Documentation](/arc/sdks/python/) for full details.
## Next steps [#next-steps]
* **[Python SDK](/arc/sdks/python/)** - Official Python client
* **[Getting Started](/arc/getting-started/)** - Quick start guide
* **[Configuration](/arc/configuration/overview/)** - Server configuration
# Connection Management (/arc/cli/connections)
`arcctl` stores connection profiles in `~/.arcctl/config.toml` (mode 0600 — plaintext tokens, same posture as `~/.aws/credentials`). One profile is marked active and used by default; you can override per-command via flags or env vars.
The model is deliberately the same as the InfluxDB v2 CLI's `influx config`, so operators coming from InfluxDB get the same UX.
## Adding a connection [#adding-a-connection]
```bash
arcctl config create \
--name local \
--endpoint http://localhost:8000 \
--token YOUR-TOKEN
```
Flags:
| Flag | Required | Description |
| -------------------- | -------- | ------------------------------------------------------ |
| `--name` | yes | Profile name (used by `--connection` and `set-active`) |
| `--endpoint` | yes | Arc HTTP base URL — no trailing slash |
| `--token` | yes | Bearer token from Arc's first-run banner |
| `--default-database` | no | Default database for query/write commands |
| `--insecure` | no | Skip TLS verification for this connection |
| `--activate` | no | Make this the active connection |
The first connection you create is auto-activated (saves you one command on first run). Subsequent ones require `--activate` to take over.
## Switching active connection [#switching-active-connection]
```bash
arcctl config set-active prod
```
Errors cleanly if the named connection does not exist.
## Listing connections [#listing-connections]
```text
$ arcctl config list
┌────────┬─────────┬─────────────────────────────┬─────────┬────────────┐
│ ACTIVE │ NAME │ ENDPOINT │ TOKEN │ DEFAULT_DB │
├────────┼─────────┼─────────────────────────────┼─────────┼────────────┤
│ * │ prod │ https://arc.prod.example… │ abc…xyz │ metrics │
│ │ local │ http://localhost:8000 │ dev…123 │ - │
└────────┴─────────┴─────────────────────────────┴─────────┴────────────┘
```
Tokens are redacted to `first4...last4`; tokens shorter than 12 chars are fully replaced with `*`.
## Inspecting the active connection [#inspecting-the-active-connection]
```bash
$ arcctl config current
name: prod
endpoint: https://arc.prod.example.com
token: abc…xyz
default_database: metrics
```
## Removing a connection [#removing-a-connection]
```bash
arcctl config delete staging
# Delete connection "staging"? [y/N] y
# Deleted connection "staging"
```
Pass `--yes` (or `-y`) to skip the confirmation prompt. If you delete the currently-active profile, the active pointer is cleared so the next command produces a clear "no active connection" error rather than silently falling back to an unrelated profile.
## Per-command overrides [#per-command-overrides]
Every command (`query`, `write`, future `db`, `import`, etc.) accepts the same connection overrides:
```bash
# Use a named profile other than the active one
arcctl --connection prod query "SELECT count(*) FROM cpu"
# Full ad-hoc — both flags must be set together
arcctl query --endpoint https://arc.x.example.com --token YOUR-TOKEN "SELECT 1"
# Env var, named profile lookup
ARC_CONNECTION=prod arcctl query "SELECT 1"
# Env var, full ad-hoc — CI-friendly, no config file needed
ARC_ENDPOINT=https://arc.x.example.com ARC_TOKEN=YOUR-TOKEN arcctl query "SELECT 1"
```
## Precedence [#precedence]
When `arcctl` needs to know which connection to use, it checks these sources in order and uses the first that matches:
1. `--connection NAME` flag
2. `--endpoint URL --token TOKEN` flags (both required together)
3. `ARC_CONNECTION` env var
4. `ARC_ENDPOINT` + `ARC_TOKEN` env vars (both required together)
5. The `active` connection in `~/.arcctl/config.toml`
If none of those is set, the command exits with a clear "no active connection" error rather than guessing.
## Config file location [#config-file-location]
By default `~/.arcctl/config.toml`. Override via `ARCCTL_CONFIG` env var — useful for tests, CI, or per-environment isolation:
```bash
ARCCTL_CONFIG=/etc/arcctl/prod.toml arcctl query "SELECT 1"
```
The file is written atomically (write to temp + rename) so a crash mid-`config create` never leaves a half-written file.
## TLS [#tls]
For HTTPS endpoints, certificate verification is on by default. To skip verification (lab or self-signed certs only) use either:
* `--insecure` on a single command, or
* `insecure_tls = true` in the connection profile (set once via `arcctl config create --insecure`)
When verification is skipped, a `WARNING:` line is printed to stderr. The flag is a no-op on `http://` endpoints and the warning is suppressed there.
**Never disable TLS verification against a production endpoint.** It exposes the bearer token to any on-path attacker.
## Security notes [#security-notes]
* The config file is mode 0600 (owner read/write only). The parent directory `~/.arcctl/` is mode 0700.
* Tokens are stored plaintext. Same posture as `~/.aws/credentials`.
* `arcctl` never logs the token. Help text, error messages, `config list`, and `config current` all use redaction.
* `arcctl` does not phone home. No telemetry. No update checks.
# Database Administration (/arc/cli/db)
`arcctl db` manages databases on an Arc cluster. The subcommands map directly onto Arc's `/api/v1/databases` endpoints.
## Quick reference [#quick-reference]
```bash
# List every database the active token can see
arcctl db list
# Inspect one database (info + its measurements)
arcctl db show production
# Create an empty database
arcctl db create metrics
# Drop a database and ALL its files (prompts y/N; use --yes to skip)
arcctl db drop old_metrics
```
## `db list` [#db-list]
```bash
arcctl db list
arcctl db list -o json | jq '.databases[] | select(.measurement_count > 0)'
arcctl db list -o csv > inventory.csv
```
Output formats: `table` (default), `json`, `csv`. Rows are sorted by name across every format so JSON / CSV / table agree on row order.
Empty server (no databases) renders `(no databases)` so you know the call ran.
## `db show ` [#db-show-name]
```text
$ arcctl db show production
Database: production
Measurements: 4
┌─────────────┬───────┐
│ MEASUREMENT │ FILES │
├─────────────┼───────┤
│ cpu │ │
│ disk │ │
│ mem │ │
│ net │ │
└─────────────┴───────┘
```
Combines `GET /api/v1/databases/:name` with `GET /api/v1/databases/:name/measurements`. JSON output (`-o json`) composes them into one object:
```json
{
"database": { "name": "production", "measurement_count": 4 },
"measurements": [{ "name": "cpu" }, { "name": "disk" }, ...],
"count": 4
}
```
`-o csv` emits only the measurements list — the database header isn't tabular-shaped.
If either API call fails, `arcctl` exits with that error and prints nothing — better to fail loud than render half a view.
## `db create ` [#db-create-name]
```bash
$ arcctl db create metrics
Created database "metrics" (created_at: 2026-05-31T23:39:58Z)
```
Server-side validation:
* Name must start with a letter and contain only alphanumeric, underscore (`_`), or hyphen (`-`) characters, max 64.
* Names `system`, `internal`, `_internal` are reserved.
* Creating a database that already exists returns HTTP 409.
The server's error messages surface verbatim, so a rejected create gives an actionable hint:
```bash
$ arcctl db create 1bad
Error: arc: Invalid database name: must start with a letter and contain only alphanumeric characters, underscores, or hyphens (max 64 characters) (HTTP 400)
```
## `db drop ` [#db-drop-name]
**Destructive.** Drops a database and **all** of its files.
```bash
# Interactive: prompts y/N
$ arcctl db drop old_metrics
Delete database "old_metrics" and ALL its files? [y/N] y
Deleted database "old_metrics"
# Scripted: --yes skips the prompt
$ arcctl db drop --yes ci_scratch
Deleted database "ci_scratch"
```
Layered safety. arcctl gates the call client-side; Arc gates it server-side:
1. **Client-side prompt** — default is N. Pass `--yes` (or `-y`) to skip; anything other than `y` / `yes` (case-insensitive) aborts.
2. **Server requires `delete.enabled=true`** in `arc.toml`. If not set, the server returns HTTP 403 with the verbatim message `"Delete operations are disabled. Set delete.enabled=true in arc.toml to enable."` — arcctl surfaces it as-is.
3. **Server requires admin token.** A read- or write-only token gets HTTP 403 from the server, surfaced verbatim.
4. **Server blocks reserved names** (`system`, `internal`, `_internal`). HTTP 403, surfaced verbatim.
5. **Server requires `?confirm=true`** on the request URL. arcctl always sends it; no operator-visible knob.
`arcctl` never bypasses any of these. If a drop fails, the message will tell you why.
## Connection overrides [#connection-overrides]
All `db` subcommands accept the same connection flags as `query` and `write`:
```bash
# Named profile
arcctl db list --connection prod
# Ad-hoc (no profile)
arcctl db list --endpoint https://arc.x.example.com --token YOUR-TOKEN
# CI-friendly env vars
ARC_CONNECTION=prod arcctl db list
ARC_ENDPOINT=https://... ARC_TOKEN=... arcctl db list
```
See [Connection management](/arc/cli/connections/#precedence) for the full precedence rules.
## Exit codes [#exit-codes]
| Code | Meaning |
| ---- | ----------------------------------------------------------------------- |
| 0 | Success (including `Aborted.` on a declined `db drop` prompt) |
| 1 | Any failure: bad flags, network error, server error, missing connection |
# Bulk Import (/arc/cli/import)
`arcctl import` is the migration entry point: load CSV, line protocol, Parquet, or TLE files into an Arc cluster in one shot. All four subcommands hit Arc's `/api/v1/import/*` endpoints and require an **admin** token.
The upload body is streamed via `io.Pipe` — even multi-GB files don't buffer in memory.
## Common shape [#common-shape]
Every subcommand:
| Flag | Required | Description |
| --------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `-f, --file PATH` | yes | Input file on the local filesystem |
| `--database NAME` | yes | Target database (also reads from the active connection's `default_database`) |
| `--measurement NAME` | depends | Required for `csv` and `parquet`; **optional filter** for `lp`; **optional override** for `tle` (server default: `satellite_tle`) |
| `-o, --output FORMAT` | no | `table` (default) or `json` — no `csv`/`arrow` since the result is a single summary record |
| `--timeout DURATION` | no | Per-request HTTP timeout (default 60s — bump it for large files) |
Plus the standard connection flags (`-c/--connection`, `--endpoint`, `--token`, `--insecure`) shared by every arcctl command.
## `import csv` [#import-csv]
```bash
arcctl import csv -f data.csv --database metrics --measurement cpu
```
Full options:
| Flag | Server default | Description |
| --------------- | -------------- | ------------------------------------------------------------------------------------------------------------ |
| `--time-column` | `time` | Column whose values become the row timestamp |
| `--time-format` | empty (auto) | `epoch_s`, `epoch_ms`, `epoch_us`, `epoch_ns`; empty means "let Arc infer", which works for ISO-8601 strings |
| `--delimiter` | `,` | Field separator |
| `--skip-rows` | `0` | Number of header rows to skip before parsing |
Example with everything:
```bash
arcctl import csv -f data.csv --database metrics --measurement cpu \
--time-column ts --time-format epoch_ms --delimiter ';' --skip-rows 1
```
## `import lp` [#import-lp]
Line protocol files self-declare their measurement(s) in the line syntax, so `--measurement` here acts as a **server-side filter** rather than a destination. Lines whose measurement doesn't match are dropped.
```bash
arcctl import lp -f telegraf-snapshot.lp --database metrics
arcctl import lp -f data.lp.gz --database metrics --precision ms
arcctl import lp -f data.lp --database metrics --measurement cpu # filter to cpu only
```
* **Gzip auto-detection.** The server inspects magic bytes (`0x1f 0x8b`); pass either `.lp` or `.lp.gz`.
* **Size cap.** 500 MB decompressed.
* **`--precision`** accepts `ns` / `us` / `ms` / `s` (default `ns`). Validated client-side before the upload starts.
A successful import shows the measurements that were ingested (LP can write to multiple in one file):
```bash
$ arcctl import lp -f mixed.lp --database metrics
OK
database: metrics
measurements: cpu, mem, disk
rows_imported: 12450
precision: ns
duration_ms: 137
```
## `import parquet` [#import-parquet]
Parquet preserves column types end-to-end — faster + lossless compared to CSV for the same data.
```bash
arcctl import parquet -f data.parquet --database metrics --measurement cpu
arcctl import parquet -f data.parquet --database metrics --measurement cpu --time-column ts
```
Only one option besides the common shape: `--time-column` (defaults to `time` server-side).
## `import tle` [#import-tle]
TLE (two-line element) is the standard NORAD/NASA format for orbital state vectors — each three-line record contains a satellite name, a "line 1", and a "line 2". The server parses every record and writes one row per satellite to the target measurement.
```bash
arcctl import tle -f starlink.tle --database satellites
arcctl import tle -f starlink.tle --database satellites --measurement starlink
```
* `--measurement` is **optional**; omitting it makes the server use `satellite_tle`.
* TLE checksums are enforced. Records with bad checksums show up in `parse_warnings` on the result and are skipped (the import doesn't fail unless every record is bad).
```bash
$ arcctl import tle -f mixed.tle --database satellites
OK
database: satellites
measurement: satellite_tle
satellite_count: 1247
rows_imported: 1247
duration_ms: 89
parse_warnings (3):
- entry 17 (UNKNOWN): line 2 checksum mismatch
- entry 312 (CUBESAT-X): line 1 checksum mismatch
- entry 489 (BEACON-9): name line missing
```
## Output formats [#output-formats]
```bash
# Table (default) — operator-friendly
arcctl import lp -f file.lp --database metrics
# JSON — pipe to jq, save for scripts
arcctl import lp -f file.lp --database metrics -o json
```
`csv` and `arrow` are not valid here — the "result" is a single summary record describing the outcome, not tabular data.
## Database selection [#database-selection]
Every subcommand picks the database from the usual precedence:
1. `--database NAME` flag
2. Active connection's `default_database`
If neither is set, the command exits with a clear "no database specified" error before any network call. See [Connection management](/arc/cli/connections/#precedence).
## Authentication [#authentication]
**All import endpoints require an admin token.** A read- or write-only token gets HTTP 403 from the server, surfaced verbatim:
```bash
$ arcctl import lp -f file.lp --database metrics
Error: arc: Admin permission required (HTTP 403)
```
Create or rotate admin tokens via the Arc API; arcctl's token-management subcommand ships in a later PR.
## Common errors [#common-errors]
| Error | Cause |
| ---------------------------------------------- | --------------------------------------------------------------------------------- |
| `arc: file is empty (HTTP 400)` | The uploaded file has zero bytes |
| `arc: no file uploaded ...` | Multipart field name mismatch (arcctl always uses `file`; this should not happen) |
| `arc: file exceeds 500MB limit` | LP files cap at 500 MB decompressed |
| `arc: invalid database name ...` | Database name doesn't match alphanumeric + `_-`, max 64 chars |
| `arc: measurement query parameter is required` | You forgot `--measurement` on a CSV or Parquet command |
| `Error: open /path: no such file or directory` | Client-side: the local file doesn't exist |
| `Error: --file is required` | Client-side: you didn't pass `-f` |
## Exit codes [#exit-codes]
| Code | Meaning |
| ---- | ---------------------------------------------------------------------- |
| 0 | Import succeeded (even if `parse_warnings` were emitted for some rows) |
| 1 | Any failure: missing flags, file errors, network errors, server errors |
Error messages go to stderr; results go to stdout. Standard Unix conventions apply:
```bash
arcctl import lp -f data.lp --database metrics -o json > result.json 2> err.log
```
# arcctl (/arc/cli)
`arcctl` is the operator-facing CLI for Arc. It replaces hand-crafted `curl` calls with a familiar workflow modeled on `influx`, `kubectl`, and `clickhouse-client`.
```bash
# Add a connection profile
arcctl config create --name local --endpoint http://localhost:8000 --token YOUR-TOKEN
# Run a query
arcctl query "SELECT count(*) FROM cpu"
# Write line protocol
echo "cpu,host=server-1 value=42.5 $(date +%s)000000000" | arcctl write
```
## Why arcctl [#why-arcctl]
Operating Arc without arcctl means:
* Reading the bootstrap token from a stderr banner once, then copying it into every `curl` (or losing it and forcing a restart with `ARC_AUTH_FORCE_BOOTSTRAP=true`).
* Building JSON query bodies by hand, setting `Authorization: Bearer`, remembering the `x-arc-database` header.
* Decoding `{"columns":[...],"data":[...]}` responses by eye.
* Juggling endpoints and tokens across dev / staging / prod via shell-var swaps.
`arcctl` handles all of that and adds named connection profiles, multiple output formats (table, JSON, CSV, Arrow IPC), file/stdin input for both query and write, and consistent error messages.
## Status [#status]
| Version | Surface |
| ------------ | ---------------------------------------------------------------------------- |
| v0.1.0 (PR1) | `config` subcommand tree, multi-connection store at `~/.arcctl/config.toml` |
| v0.2.0 (PR2) | `query`, `write` — table / JSON / CSV / Arrow IPC output, stdin / file input |
| v0.3.0 (PR3) | `db {list,show,create,drop}`, `measurement list` |
| v0.4.0 (PR4) | `import {csv,lp,parquet,tle}` — bulk file imports (admin-only) |
| v0.5.0+ | `auth`, `cluster`, msgpack write follow-up (in development) |
| v1.0.0 | release workflow + Homebrew tap + multi-arch Docker |
## Compatibility [#compatibility]
`arcctl` 0.x and 1.x talk to Arc 26.06 or newer. Arc \< 26.06 lacks the Phase A cluster auth replication that makes token admin behave consistently across nodes; an older Arc server may work for `query`/`write` but is not supported.
## Installation [#installation]
Pre-built binaries land in v1.0. For now, build from source:
```bash
git clone https://github.com/Basekick-Labs/arcctl
cd arcctl
go build -o arcctl ./cmd/arcctl
./arcctl --version
```
Requires Go 1.25+.
## Next [#next]
# Measurement Listing (/arc/cli/measurement)
`arcctl measurement list` shows the measurements inside one database. It's a thin wrapper over `GET /api/v1/databases/:name/measurements` — the same data as `arcctl db show `, but presented measurement-first.
Use `arcctl db show` when you want database metadata plus its measurements. Use `arcctl measurement list` when measurements are the primary thing you care about (scripts, CI, alerts).
## Quick reference [#quick-reference]
```bash
# Use the active connection's default_database
arcctl measurement list
# Explicit database
arcctl measurement list --database metrics
# Pipe to jq
arcctl measurement list --database logs -o json | jq '.measurements[].name'
# Save to CSV (includes the database column for downstream joins)
arcctl measurement list --database production -o csv > production-measurements.csv
```
## Database selection [#database-selection]
The database name comes from one of three places, in this precedence:
1. `--database NAME` flag
2. Active connection's `default_database` (set via `arcctl config create --default-database NAME`)
3. *(nothing else)*
If neither is set, the command exits with a clear error before any network call:
```bash
$ arcctl measurement list
Error: no database specified (pass --database or set default_database on the active connection)
```
## Output formats [#output-formats]
### Table (default) [#table-default]
```text
$ arcctl measurement list --database production
┌─────────────┬───────┐
│ MEASUREMENT │ FILES │
├─────────────┼───────┤
│ cpu │ │
│ disk │ │
│ mem │ │
│ net │ │
└─────────────┴───────┘
```
Rows sorted alphabetically. `FILES` is empty when the server omits the field (older Arc versions or when file counts haven't been computed).
Empty database: `(no measurements in database "X")` so you know the call ran.
### JSON [#json]
```bash
$ arcctl measurement list --database production -o json
{
"database": "production",
"measurements": [
{ "name": "cpu" },
{ "name": "disk" },
{ "name": "mem" },
{ "name": "net" }
],
"count": 4
}
```
Server-reported `count` is passed through verbatim — not re-derived from `len(measurements)` — so a future Arc server that paginates results will not silently disagree with itself.
### CSV [#csv]
```bash
$ arcctl measurement list --database production -o csv
database,measurement,file_count
production,cpu,
production,disk,
production,mem,
production,net,
```
The CSV includes a leading `database` column so you can concatenate output across multiple `arcctl measurement list` calls without losing context:
```bash
for db in production staging logs; do
arcctl measurement list --database "$db" -o csv --no-header
done > all-measurements.csv
```
## Connection overrides [#connection-overrides]
Same flags as every other arcctl command:
```bash
arcctl measurement list -c prod --database metrics
arcctl measurement list --endpoint https://arc.x.example.com --token YOUR-TOKEN --database logs
ARC_CONNECTION=prod arcctl measurement list --database metrics
```
See [Connection management](/arc/cli/connections/#precedence).
## Exit codes [#exit-codes]
| Code | Meaning |
| ---- | ------------------------------------------------------------------------------ |
| 0 | Success (including 0 measurements) |
| 1 | Any failure: missing database, network error, server error, missing connection |
# arcctl query (/arc/cli/query)
`arcctl query` runs SQL against an Arc cluster and renders the result in your chosen format. Defaults are operator-friendly: pretty table on stdout, errors on stderr, exit 0 on success, exit 1 on any failure.
## Quick reference [#quick-reference]
```bash
# Pretty table (default)
arcctl query "SELECT host, value FROM cpu ORDER BY value LIMIT 10"
# Different database for one call
arcctl query --database metrics "SELECT count(*) FROM cpu"
# SQL from a file
arcctl query -f reports/p99.sql
# SQL from stdin
echo "SELECT 1" | arcctl query
```
## SQL input [#sql-input]
`arcctl query` accepts SQL three ways, in this precedence:
1. **Positional argument** — `arcctl query "SELECT 1"`
2. **`-f file.sql` flag** — `arcctl query -f reports/p99.sql`
3. **Stdin** — used when neither arg nor `-f` is given and stdin is a pipe (not a TTY)
If you run `arcctl query` interactively with no arguments, it exits immediately with a clear error rather than hanging waiting for stdin.
## Output formats [#output-formats]
Pass `-o` (`--output`) to switch:
| Format | When to use |
| ----------------- | ---------------------------------------------------------------------------------------- |
| `table` (default) | Interactive use; pretty-printed with column headers |
| `json` | Pipe to `jq`, save as `.json`, parse from another script |
| `csv` | Save to a spreadsheet, load into pandas/R, RFC 4180 with header |
| `arrow` | Stream Arrow IPC bytes; pipe to pyarrow / duckdb / polars for analytical post-processing |
### Table [#table]
```text
$ arcctl query "SELECT host, value FROM cpu ORDER BY value"
┌──────────┬───────┐
│ HOST │ VALUE │
├──────────┼───────┤
│ server-1 │ 42.5 │
│ server-2 │ 43.2 │
│ server-3 │ 44.1 │
└──────────┴───────┘
```
Modifiers:
* `--no-header` — drop the column header row
* `--limit N` — cap output rows client-side (the server still computes the full result; use `LIMIT` in your SQL if you want to bound server work)
Empty result (e.g. measurement that has never been written): prints `(0 rows)` instead of nothing, so you know the query ran.
### JSON [#json]
```bash
$ arcctl query "SELECT host, value FROM cpu LIMIT 2" -o json
{
"columns": ["host", "value"],
"data": [
["server-1", 42.5],
["server-2", 43.2]
],
"row_count": 2,
"execution_time_ms": 1
}
```
The shape is row-major (`data[i]` is row i). This is the raw Arc JSON query response, indented for readability — pipe to `jq` for one-liner transformations:
```bash
arcctl query "SELECT * FROM cpu" -o json | jq '.data[] | {host: .[0], val: .[1]}'
```
### CSV [#csv]
```bash
$ arcctl query "SELECT host, value FROM cpu ORDER BY value" -o csv
host,value
server-1,42.5
server-2,43.2
server-3,44.1
```
RFC 4180 with a header row by default; `--no-header` drops it. Cell types are stringified — `true`/`false` for bools, `null` cells render as empty fields, integers print without a decimal tail, floats use compact `strconv` formatting.
### Arrow IPC [#arrow-ipc]
```bash
arcctl query "SELECT * FROM cpu" -o arrow > out.arrow
# arrow: 4096 bytes, server execution 12ms (stderr)
```
The Arrow IPC stream goes to stdout; the byte count and server-side execution time go to stderr. Stream the result into pyarrow, DuckDB, or polars:
```bash
# DuckDB
arcctl query "SELECT * FROM cpu" -o arrow | \
duckdb -c "SELECT count(*) FROM read_arrow('/dev/stdin')"
# pyarrow
arcctl query "SELECT * FROM cpu" -o arrow | python3 -c '
import pyarrow.ipc as ipc, sys
print(ipc.open_stream(sys.stdin.buffer).read_all())
'
```
If the stream is interrupted mid-flight (network drop, client kill, server reset), `arcctl` writes a clear `arrow: stream interrupted after N bytes` line to stderr along with the error. The partial bytes on stdout will not parse cleanly — that's a feature, not a bug; truncated IPC should fail loud.
## Database selection [#database-selection]
The default database for a query is taken from the active connection's `default_database` (set via `arcctl config create --default-database NAME`). Override per-call with `--database`:
```bash
arcctl query --database logs "SELECT count(*) FROM access"
```
If neither the connection default nor `--database` is set, Arc applies its own server-side default (`default`).
## Timeouts [#timeouts]
```bash
arcctl query --timeout 5m "SELECT count(*) FROM giant_table"
```
`--timeout` is the **per-request HTTP timeout** (default 60s). It must be `> 0`. For long-running queries override it explicitly; arcctl does not infer a longer timeout from the SQL.
## Exit codes [#exit-codes]
| Code | Meaning |
| ---- | --------------------------------------------------------------------- |
| 0 | Query succeeded (even if 0 rows returned) |
| 1 | Any failure: bad config, network error, server error, malformed flags |
Error messages go to stderr; output goes to stdout. Standard Unix conventions, so this works:
```bash
arcctl query "SELECT * FROM cpu" -o json > out.json 2> err.log
```
## Errors [#errors]
Server errors are surfaced with the original message and HTTP status:
```bash
$ arcctl query "SELECT FROM cpu"
Error: arc: Parser Error: syntax error at end of input (HTTP 500)
```
Client-side errors (bad flags, missing connection) are caught before any network call:
```bash
$ arcctl query --output yaml "SELECT 1"
Error: invalid --output "yaml" (valid: table, json, csv, arrow)
```
# Writing Line Protocol (/arc/cli/write)
`arcctl write` POSTs line-protocol records to Arc's `/api/v1/write/line-protocol` endpoint. Body is streamed — large files / pipes never buffer in memory — so `cat huge.lp | arcctl write` works at line-rate.
## Quick reference [#quick-reference]
```bash
# Stdin pipe — most common in CI / log forwarders / quick experiments
echo "cpu,host=server-1 value=42.5 $(date +%s)000000000" | arcctl write
# From a file
arcctl write -f payload.lp --database metrics
# Explicit precision (default is nanoseconds)
echo "cpu v=1 1700000000" | arcctl write --precision s
```
## Input [#input]
`arcctl write` reads its body from one of:
1. **`-f file.lp` flag** — opens the file with `os.Open` and streams it through to the POST body. The file handle is closed when the write completes.
2. **Stdin** — used when `-f` is not given. No buffering; bytes flow through as they arrive.
Unlike `arcctl query`, `arcctl write` does **not** error on an empty TTY stdin — typing line protocol interactively is a (rare) supported workflow. An empty body is accepted by the server as a no-op (`OK`, exit 0).
## Line protocol [#line-protocol]
Arc accepts the standard InfluxDB line protocol:
```text
,=,= =,=
```
* **Measurement** — required, the "table" name in Arc
* **Tags** — optional, comma-separated key=value pairs (string-only, schemas inferred)
* **Fields** — required, at least one numeric/string/bool value
* **Timestamp** — optional; if omitted, Arc applies wall-clock-at-receive
Example:
```text
cpu,host=server-1,region=us-east value=42.5,temp=68.0 1700000000000000000
mem,host=server-1 used=8.5,total=16.0 1700000000000000000
```
## Precision [#precision]
`--precision` tells the server how to interpret bare-integer timestamps:
| Value | Unit |
| -------------- | ----------------------- |
| `ns` (default) | nanoseconds since epoch |
| `us` | microseconds |
| `ms` | milliseconds |
| `s` | seconds |
```bash
echo "cpu v=1 1700000000" | arcctl write --precision s
echo "cpu v=1 1700000000000" | arcctl write --precision ms
```
`arcctl` validates the precision flag client-side before the request goes out, so a typo like `--precision furlong` fails fast:
```bash
$ arcctl write --precision furlong < x.lp
Error: invalid --precision "furlong" (must be one of ns, us, ms, s)
```
## Database selection [#database-selection]
The default database comes from the active connection's `default_database`. Override per-call:
```bash
arcctl write -f payload.lp --database metrics
```
If neither the connection default nor `--database` is set, Arc applies its own server-side default (`default`).
## Streaming behavior [#streaming-behavior]
`arcctl write` does NOT buffer the body. This matters for:
* **Large files** — `arcctl write -f /var/log/lp/all-day.lp` streams without loading the file into RAM.
* **Continuous pipes** — `tail -F app.log | parser | arcctl write` keeps memory flat while ingesting at line-rate.
* **HTTP timeout** — `--timeout` applies to the **whole request**. A 10 GB file + the default 60s timeout will time out before completion. For large writes, raise `--timeout`:
```bash
arcctl write -f huge.lp --timeout 30m
```
## Exit codes [#exit-codes]
| Code | Meaning |
| ---- | ------------------------------------------------------------- |
| 0 | Server returned 204 No Content (success) |
| 1 | Any failure: bad flags, network error, server error (4xx/5xx) |
On success, `arcctl write` prints `OK` to stdout. On failure, the server's error message is surfaced:
```bash
$ echo "garbage" | arcctl write
Error: arc: malformed line at offset 0 (HTTP 400)
```
## Common patterns [#common-patterns]
### Backfill from a file [#backfill-from-a-file]
```bash
arcctl write -f historical.lp --database backfill --precision ms
```
### Continuous ingestion from a log tail [#continuous-ingestion-from-a-log-tail]
```bash
tail -F /var/log/app.log | \
awk '{ printf("log,host=%s msg=\"%s\" %d000000000\n", "myhost", $0, systime()) }' | \
arcctl write --database logs
```
### Ad-hoc connection (no profile) [#ad-hoc-connection-no-profile]
```bash
echo "metric v=1" | arcctl write \
--endpoint https://arc.staging.example.com \
--token YOUR-TOKEN \
--database metrics
```
### CI-friendly with env vars [#ci-friendly-with-env-vars]
```bash
ARC_ENDPOINT=https://arc.x.example.com ARC_TOKEN=YOUR-TOKEN \
arcctl write -f payload.lp --database metrics
```
See [Connection management](/arc/cli/connections/#precedence) for the full precedence rules.
# Authentication (/arc/configuration/authentication)
Arc uses token-based authentication to secure API access. Tokens are stored in a SQLite database with an in-memory cache for high-performance validation.
Authentication is enabled by default since Arc v26.01.2. To disable it for local development, set `auth.enabled = false` in `arc.toml`.
## Configuration [#configuration]
```toml
[auth]
enabled = true # Enable/disable authentication
db_path = "./data/arc_auth.db" # SQLite database for token storage
cache_ttl = 30 # Token cache TTL in seconds
max_cache_size = 1000 # Maximum cached tokens
bootstrap_token = "" # Pre-set admin token value (v26.04.1+)
force_bootstrap = false # Add a recovery token without removing existing ones (v26.04.1+)
```
**Environment variables:**
```bash
export ARC_AUTH_ENABLED=true
export ARC_AUTH_DB_PATH="./data/arc_auth.db"
export ARC_AUTH_CACHE_TTL=30
export ARC_AUTH_MAX_CACHE_SIZE=1000
export ARC_AUTH_BOOTSTRAP_TOKEN="" # v26.04.1+
export ARC_AUTH_FORCE_BOOTSTRAP=false # v26.04.1+
```
## Authentication methods [#authentication-methods]
Arc supports multiple authentication methods for compatibility with various clients:
### Bearer token (standard) [#bearer-token-standard]
```bash
curl -H "Authorization: Bearer $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### Token header (InfluxDB 2.x style) [#token-header-influxdb-2x-style]
```bash
curl -H "Authorization: Token $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### API key header [#api-key-header]
```bash
curl -H "x-api-key: $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### Query parameter (InfluxDB 1.x style) [#query-parameter-influxdb-1x-style]
For InfluxDB 1.x client compatibility:
```bash
curl "http://localhost:8000/write?db=mydb&p=$ARC_TOKEN" -d 'cpu,host=server01 usage=45.2'
```
## Bootstrap & recovery [#bootstrap--recovery]
`ARC_AUTH_BOOTSTRAP_TOKEN` and `ARC_AUTH_FORCE_BOOTSTRAP` are available in Arc and Arc Enterprise v26.04.1 and later.
### Pre-configured bootstrap token [#pre-configured-bootstrap-token]
By default, Arc generates a random admin token on first start and prints it once to stderr. If you miss it, recovery requires deleting the auth database and redeploying.
`ARC_AUTH_BOOTSTRAP_TOKEN` lets you set a known token value at deploy time. On first run, Arc uses this value as the initial admin token instead of generating a random one. On subsequent restarts, it is a no-op — the existing token is preserved.
```bash
export ARC_AUTH_BOOTSTRAP_TOKEN="your-secret-token-value-at-least-32-chars"
```
This is especially useful for:
* **Automated deployments** — bake the token into your secrets manager (Vault, AWS Secrets Manager, Kubernetes Secrets) and have it ready without catching a log line
* **Reproducible environments** — staging and production can use different known tokens set consistently at deploy time
Token values must be at least 32 characters. Values are stored as bcrypt hashes — the plaintext never persists to disk.
### Recovery when the admin token is lost [#recovery-when-the-admin-token-is-lost]
If you no longer have access to any admin token, set both `ARC_AUTH_BOOTSTRAP_TOKEN` and `ARC_AUTH_FORCE_BOOTSTRAP=true` before restarting Arc. Arc will add a new admin token named `arc-recovery` **without removing any existing tokens**.
```bash
export ARC_AUTH_BOOTSTRAP_TOKEN="your-new-recovery-token-at-least-32-chars"
export ARC_AUTH_FORCE_BOOTSTRAP=true
```
Existing tokens are preserved so that if the recovery token was injected by a bad actor, any legitimate admin still has their token and can revoke it immediately via the API.
After recovering access:
1. Use the API to review and revoke any tokens you no longer need
2. Remove `ARC_AUTH_FORCE_BOOTSTRAP` from your deployment configuration
If Arc restarts with `ARC_AUTH_FORCE_BOOTSTRAP=true` and the `arc-recovery` token already exists, it is a no-op. You still hold the token value you provided.
## Cluster auth replication (Enterprise) [#cluster-auth-replication-enterprise]
Cluster-wide token replication is available in Arc Enterprise v26.06.1 and later. OSS / standalone deployments are unaffected — tokens stay in the local SQLite as they always have.
Before v26.06.1, every Arc Enterprise cluster node carried its own SQLite auth DB. A token created via `POST /api/v1/auth/tokens` on the writer was **not** valid on the reader — the reader's local SQLite never saw the row. Operators worked around this by pre-seeding `ARC_AUTH_BOOTSTRAP_TOKEN` with the same value on every node, but API-created tokens and revocations did not propagate. A revocation on the writer left the same token still valid on every reader for the lifetime of those reader processes.
v26.06.1 routes auth **writes** through the cluster's Raft consensus. Auth **reads** still hit the local cache — there's no Raft round trip on every API call.
### What replicates [#what-replicates]
| Operation | Replicates cluster-wide? |
| ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Create token | Yes |
| Update token (rename, change permissions, change expiry) | Yes |
| Revoke token | Yes |
| Delete token | Yes |
| Rotate token (new value, same metadata) | Yes |
| `EnsureInitialToken` (first-run bootstrap) | Yes — only the Raft leader's proposal lands; other nodes get an "already exists" no-op |
| RBAC tables (organizations, teams, roles, measurement permissions, token memberships) | Yes — landed in v26.06.1 alongside Phase A. See [RBAC replication](#rbac-replication-enterprise) below. |
| Audit log entries | **No — intentionally per-node** (high-volume append-only, no consensus needed) |
| SSO / OIDC / LDAP | **No — Phase B, separate roadmap item** |
### Convergence semantics [#convergence-semantics]
Eventual consistency, typically under 50 ms via Raft apply on local loopback or LAN. Customer SDKs already retry on transient 401, so the brief window between leader commit and follower materialise is invisible in normal usage.
A read-after-write barrier is **not** included in v26.06.1. If a real customer report surfaces the window, a follow-up will add a per-query `LastApplied()` barrier on the query path.
### Bootstrap banner now prints on the leader only [#bootstrap-banner-now-prints-on-the-leader-only]
Before v26.06.1, every cluster node printed its own randomly-generated admin token at first start, so a 4-node boot produced 4 banners and 4 different admin tokens (each valid only on its own node).
From v26.06.1, every node still calls `EnsureInitialToken` on boot with its own random plaintext, but only the Raft leader's proposal lands cluster-wide. Followers receive the FSM's `"token name already exists"` rejection from the leader and return an empty plaintext to the caller, so **no banner is emitted on losers**. The losing node's local SQLite still gets the winner's bcrypt hash + prefix via the FSM materialise callback — every node converges on the same admin token.
If you watch a 3-writer + 1-reader cluster boot, expect:
* 1 `Admin API token:` banner on the Raft leader's stderr
* 3 `INFO Deferring initial token bootstrap until cluster Raft proposer is wired (Phase A)` lines during startup (one per node)
* 1 `INFO Cluster auth state replication enabled — token writes now propagate via Raft` line per node after Raft elects a leader
### Security posture [#security-posture]
Plaintext token values are **never** written to the Raft log. The proposer generates the token, hashes it with bcrypt locally, and only the hash + prefix go into the replicated payload. The plaintext is returned to the API caller out-of-band before any Raft work begins. Snapshot dumps don't contain plaintext either — verified by a snapshot-grep test in the test suite.
Applier-side validation runs on every node before a token command lands in the FSM. Empty name, missing bcrypt hash, missing prefix, malformed permission string, or zero `created_at` all cause the entry to be rejected on every node. The cluster-wide rejection counter (`arc_cluster_auth_rejected_total`) increments and the rejection is logged at `Error`.
### Prometheus counters [#prometheus-counters]
Per node, on the `/metrics` endpoint:
```text
arc_cluster_auth_apply_create_total
arc_cluster_auth_apply_update_total
arc_cluster_auth_apply_revoke_total
arc_cluster_auth_apply_delete_total
arc_cluster_auth_apply_rotate_total
arc_cluster_auth_rejected_total
```
In a healthy cluster every node sees the same monotonic count for each `apply_*` counter — they all apply the same Raft log. Divergence across nodes is the load-bearing signal that one of them is missing applies (network partition, FSM stall).
`arc_cluster_auth_rejected_total` is the **security alerting signal** — non-zero growth means somebody is proposing tokens that fail applier-side validation (malformed payload, fuzz attempt, or a buggy client). Alert on growth, not on absolute value.
### Pre-existing tokens DO NOT auto-migrate [#pre-existing-tokens-do-not-auto-migrate]
Tokens created on a pre-v26.06.1 node by the local-only API path remain valid **only on that node** after upgrade. The expected migration path is:
1. Upgrade every cluster node to v26.06.1.
2. Re-issue API tokens via `POST /api/v1/auth/tokens` after restart. New tokens are cluster-wide automatically.
3. Revoke the old per-node tokens via `POST /api/v1/auth/tokens/:id/revoke` (the revoke also propagates cluster-wide).
Bootstrap tokens set via `ARC_AUTH_BOOTSTRAP_TOKEN` are unaffected if the same value was used on every node (which the pre-v26.06.1 workaround required) — the bytes match, so all nodes effectively share the same admin token already.
### Divergence detection [#divergence-detection]
If a pre-v26.06.1 AUTOINCREMENT row in your local `auth.db` happens to share an ID with a new cluster-replicated token (Raft log indices land in the same `INTEGER` space), the cluster apply on that node will detect the collision and **refuse to overwrite** the pre-existing row. The cluster's in-memory FSM map remains authoritative; the local SQLite cache stays divergent until the operator resolves it.
You'll see an `Error`-level log line on the affected node:
```text
ApplyCreateToken: id already exists locally with different token (cluster<->local divergence; see upgrade notes for pre-26.06.1 tokens)
```
And the `arc_cluster_auth_rejected_total` counter increments on that node only.
**Remediation**: drop the diverging rows from the local `auth.db`, or drop the whole local auth DB and let the FSM repopulate from the cluster's snapshot. Stop Arc on the affected node, run:
```sql
sqlite3 /app/data/arc.db "DELETE FROM api_tokens WHERE id = "
```
Then restart Arc — it'll re-apply the cluster's authoritative state on the affected ID range.
Identical hash + name is treated as idempotent log replay (no-op, no error), so a normal cluster restart never surfaces this.
### `arcx`-style upgrade path [#arcx-style-upgrade-path]
Operators of pre-v26.06.1 Enterprise clusters with many AUTOINCREMENT tokens may prefer to **drain the local auth DB** before re-joining:
1. Note down the names of any service tokens currently in active use.
2. Stop Arc on the node.
3. Move `auth.db`, `auth.db-shm`, `auth.db-wal` aside (don't delete — keep as backup).
4. Restart Arc with `ARC_AUTH_BOOTSTRAP_TOKEN` matching the cluster's admin token.
5. Re-issue the service tokens cluster-wide via the API on any leader-eligible node.
The cluster's FSM is the source of truth, so this drain-and-rejoin is non-destructive — the only state at risk is per-node tokens that weren't intended to be cluster-wide, and the release notes already document that they don't carry over.
### Required configuration [#required-configuration]
In addition to standard cluster mode (`cluster.enabled = true`, `cluster.raft_data_dir`, etc.), token replication requires:
```toml
[cluster]
shared_secret = "..." # min 32 chars; same value on every node
```
The shared secret authenticates leader-forward HMAC for non-leader nodes proposing token writes. Without it, follower nodes refuse to forward auth proposals and the cluster falls back to OSS-mode bootstrap on every node (you'll see 4 banners again).
```bash
export ARC_CLUSTER_SHARED_SECRET="$(openssl rand -hex 32)"
```
If nodes have different secrets, follower-to-leader forward-apply fails HMAC validation and token writes that originate on a non-leader silently fail. There is no graceful fallback — operators must ensure the secret is identical across the cluster (e.g. via Kubernetes Secrets or environment-variable injection from a single source).
## RBAC replication (Enterprise) [#rbac-replication-enterprise]
Phase A.1 of Cluster Auth Convergence — every RBAC write (organizations, teams, roles, measurement permissions, token memberships) propagates cluster-wide via the same Raft FSM seam used for tokens. Lands in the same v26.06.1 release as Phase A token replication.
Before v26.06.1, RBAC writes hit only the local node's SQLite — same shape as the token gap Phase A closes. An organization created on the writer was invisible to RBAC checks on every reader; a role grant on one node didn't grant the corresponding permission anywhere else. v26.06.1 routes all 13 RBAC writes through Raft so the cluster converges on a single RBAC state.
### What replicates [#what-replicates-1]
| RBAC operation | Replicates cluster-wide? |
| ---------------------------------------------------------------------------------------------------------------- | ------------------------ |
| `POST /api/v1/rbac/organizations` (create) | Yes |
| `PATCH /api/v1/rbac/organizations/:id` (update) | Yes |
| `DELETE /api/v1/rbac/organizations/:id` (delete + cascade to teams, roles, measurement permissions, memberships) | Yes |
| `POST /api/v1/rbac/organizations/:org_id/teams` (create) | Yes |
| `PATCH /api/v1/rbac/teams/:id` (update) | Yes |
| `DELETE /api/v1/rbac/teams/:id` (delete + cascade to roles, measurement permissions, memberships) | Yes |
| `POST /api/v1/rbac/teams/:team_id/roles` (create) | Yes |
| `PATCH /api/v1/rbac/roles/:id` (update) | Yes |
| `DELETE /api/v1/rbac/roles/:id` (delete + cascade to measurement permissions) | Yes |
| `POST /api/v1/rbac/roles/:role_id/measurements` (create) | Yes |
| `DELETE /api/v1/rbac/measurement-permissions/:id` (leaf delete) | Yes |
| `POST /api/v1/auth/tokens/:id/teams` (add membership) | Yes |
| `DELETE /api/v1/auth/tokens/:id/teams/:team_id` (remove membership) | Yes |
### Cascade-on-delete [#cascade-on-delete]
Deleting an organization removes every descendant — teams, roles, measurement permissions, and token memberships — under a single Raft log entry. Same for `DeleteTeam` (cascades to roles + measurement permissions + memberships) and `DeleteRole` (cascades to measurement permissions). Concurrent writes targeting a being-cascaded entity are serialised and see the post-cascade state.
When a token is hard-deleted via `DELETE /api/v1/auth/tokens/:id` its memberships are also removed cluster-wide — mirroring the SQLite FK cascade `rbac_token_memberships.token_id REFERENCES api_tokens(id) ON DELETE CASCADE` at the FSM layer so the in-memory state stays consistent across nodes.
### Cascade-on-delete soft cap [#cascade-on-delete-soft-cap]
Phase A.2 Item 2 — a configurable cap on the number of descendants `DeleteOrganization` / `DeleteTeam` will cascade through in cluster mode.
The FSM cascade-on-delete runs under `f.mu.Lock()` on the single-threaded Raft apply goroutine. hashicorp/raft runs `runFSM` async of heartbeats, so a long apply does **not** directly cost the leader its lease — but for a pathologically large tenant (\~100k+ descendants under one organization), the cascade can hold the apply goroutine long enough to blow past the proposer-side 5 s `proposeTimeout`. The originating client sees an opaque timeout while the apply still completes in the background; meanwhile later commands queue behind the slow apply and risk failing their own timeout budgets. Operators see unclear "propose timeout" diagnostics on a delete that "should have worked," instead of a clear "you tried to delete too much at once."
v26.06.1 ships a configurable proposer-side cap. Before proposing `CommandDeleteOrganization` or `CommandDeleteTeam`, the proposer counts the descendants in local SQLite (`teams + roles + measurement_permissions + token_memberships`). If the total exceeds the cap, the API returns **HTTP 409 Conflict** without spending a Raft log entry on a cascade that would block the apply path.
| Setting | Value |
| ---------------------- | ---------------------------------------------------------------- |
| Config key (TOML) | `cluster.rbac.max_cascade_descendants` |
| Env var | `ARC_CLUSTER_RBAC_MAX_CASCADE_DESCENDANTS` |
| Default | `50000` |
| Disable | `0` (no cap; escape hatch for operators who know their workload) |
| HTTP code on rejection | `409 Conflict` |
| Metric | `arc_cluster_rbac_cascade_rejected_total` |
The error body includes the actual descendant count, the configured cap, and the operator workaround:
```json
{
"success": false,
"error": "cascade exceeds configured limit: 73214 descendants under organization 42 (max 50000); delete child entities (teams, roles, measurement_permissions, token_memberships) first"
}
```
Operator workaround when 409 lands: `DELETE` the affected children first (roles → teams → re-attempt the org delete), or raise the cap if your tenant size justifies it. `DeleteRole`'s cascade is 1-level (only measurement\_permissions) and is not capped — it can't plausibly blow up the apply path.
The pre-check costs four small `COUNT(*)` queries against indexed columns — sub-millisecond at realistic cap values, well under the 5-second Raft proposal timeout.
### Pre-existing RBAC rows: Auto-seed for orgs, manual re-issue for the rest [#pre-existing-rbac-rows-auto-seed-for-orgs-manual-re-issue-for-the-rest]
On the first leader boot after upgrading to v26.06.1, Arc runs a one-time **upgrade seed** that walks the leader's local `rbac_organizations` table and proposes a `CommandCreateOrganization` for every pre-existing row. The cluster's in-memory FSM learns the org under a fresh log-index ID; the leader's local SQLite keeps the row under its pre-v26.06.1 AUTOINCREMENT ID; both IDs map to the same logical org because the `UNIQUE(name)` constraint is enforced cluster-wide. Followers see the new org via Raft replication and store it under the cluster ID.
**Teams, roles, measurement permissions, and token memberships are NOT auto-seeded.** They reference parent entities by surrogate ID, and the pre-v26.06.1 local AUTOINCREMENT IDs don't generally match the cluster's log-index-stamped IDs after the org seed runs. The seed logs a `Warn` at startup listing the unseeded counts per table:
```bash
WARN child RBAC rows present in local SQLite are NOT auto-seeded (FK-ID rebase ambiguity);
re-issue them via the API post-upgrade for cluster-wide replication
teams_local=3 roles_local=7 measurement_permissions_local=12 token_memberships_local=4
```
Re-issue each affected team, role, measurement permission, and token membership via the API after upgrade. Existing local rows stay readable on the leader (the FK chain in local SQLite is intact) — they just won't replicate to followers until re-created.
The seed runs only on the Raft leader, gated by `WaitForLeader(30s)`. It is idempotent — re-running on the same leader is a no-op (each proposal is rejected as `"organization name already exists"` and the seed counts it as skipped rather than retrying).
### Prometheus counters [#prometheus-counters-1]
Per node, alongside the Phase A token counters:
```text
arc_cluster_rbac_apply_create_organization_total
arc_cluster_rbac_apply_update_organization_total
arc_cluster_rbac_apply_delete_organization_total
arc_cluster_rbac_apply_create_team_total
arc_cluster_rbac_apply_update_team_total
arc_cluster_rbac_apply_delete_team_total
arc_cluster_rbac_apply_create_role_total
arc_cluster_rbac_apply_update_role_total
arc_cluster_rbac_apply_delete_role_total
arc_cluster_rbac_apply_create_measurement_permission_total
arc_cluster_rbac_apply_delete_measurement_permission_total
arc_cluster_rbac_apply_add_token_to_team_total
arc_cluster_rbac_apply_remove_token_from_team_total
arc_cluster_rbac_rejected_total
arc_cluster_rbac_cascade_rejected_total
```
`arc_cluster_rbac_rejected_total` is a **single counter aggregating applier-side validation failures across all 13 RBAC command types** — empty names, missing parent IDs, malformed permission strings, UNIQUE collisions. Same security-alerting semantics as `arc_cluster_auth_rejected_total`: non-zero growth means somebody is proposing malformed RBAC commands; alert on growth.
`arc_cluster_rbac_cascade_rejected_total` counts proposer-side cascade-cap refusals (see [Cascade-on-delete soft cap](#cascade-on-delete-soft-cap) above). Non-zero growth means operators are issuing cascades larger than `cluster.rbac.max_cascade_descendants`. Alert if you'd rather raise the cap than have operators retry after manual cleanup.
In a healthy cluster every node sees the same monotonic count for each `apply_*` counter. Per-node divergence indicates a node missing applies (network partition, FSM stall).
### Required configuration [#required-configuration-1]
RBAC replication itself requires no new env vars — it is gated by the same `cluster.enabled = true` + Enterprise license + `cluster.shared_secret` that gate Phase A token replication.
One **optional** knob is documented above: `cluster.rbac.max_cascade_descendants` (env `ARC_CLUSTER_RBAC_MAX_CASCADE_DESCENDANTS`, default `50000`) caps cluster-mode `DeleteOrganization` / `DeleteTeam` cascades. See [Cascade-on-delete soft cap](#cascade-on-delete-soft-cap).
## Token management [#token-management]
All token management endpoints require **admin** authentication.
### Creating tokens [#creating-tokens]
```bash
curl -X POST "http://localhost:8000/api/v1/auth/tokens" \
-H "Authorization: Bearer ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "my-service",
"description": "Token for production service",
"is_admin": false
}'
```
```json
{
"id": "abc123",
"name": "my-service",
"token": "arc_xxxxxxxxxxxxxxxxxxxxxxxx",
"is_admin": false,
"created_at": "2026-01-15T10:30:00Z"
}
```
The token value is only returned once at creation time. Store it securely -- it cannot be retrieved later.
### Listing tokens [#listing-tokens]
```bash
curl "http://localhost:8000/api/v1/auth/tokens" \
-H "Authorization: Bearer ADMIN_TOKEN"
```
### Rotating a token [#rotating-a-token]
Generate a new token value while keeping the same token ID and permissions:
```bash
curl -X POST "http://localhost:8000/api/v1/auth/tokens/abc123/rotate" \
-H "Authorization: Bearer ADMIN_TOKEN"
```
### Revoking a token [#revoking-a-token]
Immediately invalidate a token:
```bash
curl -X POST "http://localhost:8000/api/v1/auth/tokens/abc123/revoke" \
-H "Authorization: Bearer ADMIN_TOKEN"
```
### Deleting a token [#deleting-a-token]
Permanently remove a token:
```bash
curl -X DELETE "http://localhost:8000/api/v1/auth/tokens/abc123" \
-H "Authorization: Bearer ADMIN_TOKEN"
```
## Verifying a token [#verifying-a-token]
The verify endpoint is public (no authentication required) and checks if a token is valid:
```bash
curl -H "Authorization: Bearer $ARC_TOKEN" \
"http://localhost:8000/api/v1/auth/verify"
```
```json
{
"valid": true,
"token_info": {
"id": "abc123",
"name": "my-service",
"is_admin": false
},
"permissions": []
}
```
## Token cache [#token-cache]
Arc caches validated tokens in memory to avoid SQLite lookups on every request. This is critical for high-throughput ingestion.
### Cache statistics [#cache-statistics]
```bash
curl "http://localhost:8000/api/v1/auth/cache/stats" \
-H "Authorization: Bearer ADMIN_TOKEN"
```
### Invalidating the cache [#invalidating-the-cache]
Force all cached tokens to be re-validated against SQLite:
```bash
curl -X POST "http://localhost:8000/api/v1/auth/cache/invalidate" \
-H "Authorization: Bearer ADMIN_TOKEN"
```
Cache invalidation is automatic for most operations (revoke, delete, rotate). Manual invalidation is only needed if you modify the SQLite database directly.
## Public endpoints [#public-endpoints]
These endpoints do not require authentication:
* `GET /health` -- Health check
* `GET /ready` -- Readiness probe
* `GET /metrics` -- Prometheus metrics
* `GET /api/v1/auth/verify` -- Token verification
## API endpoints reference [#api-endpoints-reference]
| Method | Endpoint | Auth | Description |
| -------- | -------------------------------- | ------ | ---------------------- |
| `GET` | `/api/v1/auth/verify` | Public | Verify token validity |
| `GET` | `/api/v1/auth/tokens` | Admin | List all tokens |
| `POST` | `/api/v1/auth/tokens` | Admin | Create a new token |
| `GET` | `/api/v1/auth/tokens/:id` | Admin | Get token details |
| `PATCH` | `/api/v1/auth/tokens/:id` | Admin | Update token metadata |
| `DELETE` | `/api/v1/auth/tokens/:id` | Admin | Delete a token |
| `POST` | `/api/v1/auth/tokens/:id/rotate` | Admin | Rotate token value |
| `POST` | `/api/v1/auth/tokens/:id/revoke` | Admin | Revoke a token |
| `GET` | `/api/v1/auth/cache/stats` | Admin | Cache statistics |
| `POST` | `/api/v1/auth/cache/invalidate` | Admin | Invalidate token cache |
# FIPS 140-3 Mode (/arc/configuration/fips)
Arc ships an optional **FIPS build variant** (`arc-fips`) for US federal, defense,
aerospace, and other regulated environments that require validated cryptography.
The `arc-fips` build variant is available **since Arc 26.06.2**. Every release from
26.06.2 onward publishes `arc-fips` artifacts alongside the standard ones.
## Why it exists [#why-it-exists]
Many US government, defense, and regulated deployments are contractually required
(under FIPS 140-2/3, FedRAMP, NIST SP 800-171 / CMMC, and similar) to run software
whose cryptography comes from a **validated cryptographic module**. The standard
Arc build uses Go's cryptography, which is correct and modern but is not, by
default, operated as a FIPS-validated module. The `arc-fips` variant closes that
gap: it is compiled against the **CMVP-certified Go Cryptographic Module** and runs
it in FIPS-only mode, so an auditor can trace every cryptographic operation to a
validated module. If you have no FIPS/compliance requirement, use the standard
build — it is the right default for everyone else.
The FIPS variant is the **same Arc, at the same version, from the same commit** as
the standard build — it is not a separate product, a different feature set, or a
separate version line. Only the build is different.
`arc-fips` reports the same version as the standard build (for example `26.06.2`).
You identify it by its artifact name (`arc-fips-…`, image tag `:VERSION-fips`) and
by `"fips_mode":true` in its startup log — never by a different version number.
## Standard build vs `arc-fips` [#standard-build-vs-arc-fips]
| | Standard `arc` | `arc-fips` |
| ---------------------------- | ------------------------------------- | ------------------------------------------------------ |
| Features / API / wire format | identical | identical |
| Version string | `26.06.2` | `26.06.2` (same) |
| Performance build tag | `duckdb_arrow` | `duckdb_arrow` (same) |
| Cryptographic module | Go crypto (not operated as validated) | **CMVP-certified Go Cryptographic Module**, FIPS-only |
| TLS cipher suites / curves | Go defaults (incl. X25519) | FIPS-approved only (AES-GCM, NIST P-curves; no X25519) |
| API-token hashing | bcrypt (and legacy SHA-256) | **PBKDF2-HMAC-SHA256** only |
| Startup if not in FIPS mode | n/a | **refuses to start** (fails closed) |
| Artifact name | `arc-linux-*`, image `:26.06.2` | `arc-fips-linux-*`, image `:26.06.2-fips` |
| Who it's for | everyone (default) | FIPS / regulated deployments only |
Both are signed (cosign) with SLSA provenance, and both ship `.deb`/`.rpm`
packages, container images, and SBOMs.
## What the FIPS build enforces [#what-the-fips-build-enforces]
* **Validated cryptographic module.** Built with `GOFIPS140=v1.0.0`, the
CMVP-certified Go Cryptographic Module snapshot. The module runs its power-on
self-tests at process start.
* **FIPS-only runtime.** `GODEBUG=fips140=only` is baked into the binary, so
non-approved standard-library crypto fails closed. The binary **refuses to
start** if it is not actually running in FIPS mode.
* **Approved TLS only.** The API, cluster, and MQTT TLS paths are restricted to
FIPS-approved cipher suites and elliptic curves (TLS 1.2+; AES-GCM; P-256/384/521).
* **Approved password hashing.** API-token hashing uses PBKDF2-HMAC-SHA256
(FIPS-approved) instead of bcrypt.
* **No non-approved crypto linked.** The FIPS binary contains no bcrypt/Blowfish
code; this is verified in CI by an import-graph check.
## Cryptographic boundary [#cryptographic-boundary]
All cryptography in Arc is provided by the Go Cryptographic Module. The query
engine and SQLite perform **no** cryptography — SQLite is used for token
*storage*, not encryption — and both are outside the module boundary.
## Installing the FIPS build [#installing-the-fips-build]
The FIPS variant ships alongside the standard build in every release. Pick the
`-fips` artifact instead of the standard one.
### Packages (.deb / .rpm) [#packages-deb--rpm]
The FIPS variant ships the same package formats as the standard build, so an
existing install can be swapped in place — same service name, same config, same
data directory.
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
# arc-fips_${LATEST_VERSION}_arm64.deb on ARM
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-fips_${LATEST_VERSION}_amd64.deb
sudo dpkg -i arc-fips_${LATEST_VERSION}_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
# arc-fips-${LATEST_VERSION}-1.aarch64.rpm on ARM
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-fips-${LATEST_VERSION}-1.x86_64.rpm
sudo rpm -i arc-fips-${LATEST_VERSION}-1.x86_64.rpm
sudo systemctl enable arc && sudo systemctl start arc
```
The standard build ships an Arch package, but the FIPS variant does not — the
release publishes `.deb`, `.rpm`, container images and standalone binaries only.
On Arch, use the binary below.
### Binary [#binary]
```bash
# Download arc-fips-linux-amd64 (or -arm64) from the GitHub release, then verify:
cosign verify-blob arc-fips-linux-amd64 \
--bundle arc-fips-linux-amd64.bundle \
--certificate-identity-regexp "^https://github.com/Basekick-Labs/arc/" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com
```
### Container [#container]
```bash
# Same repo as the standard image, with a -fips tag suffix:
docker pull ghcr.io/basekick-labs/arc:VERSION-fips
# or: docker pull basekicklabs/arc:VERSION-fips
cosign verify ghcr.io/basekick-labs/arc:VERSION-fips \
--certificate-identity-regexp "^https://github.com/Basekick-Labs/arc/" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com
```
### Kubernetes (Helm) [#kubernetes-helm]
Set the image tag to the `-fips` variant:
```yaml
image:
repository: ghcr.io/basekick-labs/arc
tag: VERSION-fips
```
### Confirming FIPS mode [#confirming-fips-mode]
On startup the FIPS build logs:
```json
{"level":"info","version":"26.06.2","fips_mode":true,"message":"Starting Arc..."}
```
If `fips_mode` is `false` on an `arc-fips` binary, the process exits — it will not
run outside FIPS mode.
The `arc-fips` binary bakes in `GODEBUG=fips140=only` (the strict mode — calls to
non-approved algorithms fail). The startup check (`fips_mode:true`) confirms the
module is *active*, but Go's runtime exposes no API to distinguish `only` from the
weaker `on` mode, so an operator who explicitly sets `GODEBUG=fips140=on` in the
process environment would override the baked-in `only` without the log changing.
For an auditable deployment, do **not** set `GODEBUG=fips140=*` in the
environment — leave the binary's compiled-in `only` default in place — and verify
no such override exists in your unit files / container env. (Setting
`fips140=off` is still caught: the process refuses to start.)
## Upgrading from standard `arc` to `arc-fips` [#upgrading-from-standard-arc-to-arc-fips]
This is the one operational gotcha when moving an existing deployment from the
standard build to `arc-fips`. Plan for it before you cut over.
API tokens created by the **standard** build are stored as bcrypt hashes (and
pre-v26 tokens as SHA-256). The FIPS build **refuses to verify** those hashes — it
denies the request (and logs the reason at debug level), because verifying them
would use a non-FIPS-approved algorithm. **Tokens that worked on the standard
build will be rejected on `arc-fips` until they are rotated.**
To cut over:
1. Bring up the `arc-fips` build pointed at the same data.
2. **Rotate (recreate) every API token.** New tokens are stored as PBKDF2 hashes
automatically and work immediately.
3. Update any clients/automation with the new token values.
Tokens are random 256-bit values that Arc never stores in plaintext, so existing
bcrypt hashes **cannot** be migrated in place — rotation is the only path. The
`/usr/bin/arc` path, `arc.service`, config, and data directory are otherwise
unchanged, so the rest of the upgrade is a normal binary/image swap.
Going the other direction (`arc-fips` → standard) needs no token rotation — the
standard build verifies PBKDF2 hashes too.
## Arc Enterprise [#arc-enterprise]
Arc Enterprise customers get FIPS by running the `arc-fips` build with their
license key (`ARC_ENTERPRISE_LICENSE`). There is no separate enterprise FIPS
binary — Enterprise FIPS is `arc-fips` + license.
## CMVP status [#cmvp-status]
Arc's FIPS build is compiled against the **CMVP-certified** Go Cryptographic
Module v1.0.0. This means the *cryptographic module* is validated. **Arc itself is
not a CMVP-listed module**, and this is **not** a statement that "Arc is FIPS 140-3
validated." Confirm the live certificate number on the
[NIST CMVP Validated Modules list](https://csrc.nist.gov/projects/cryptographic-module-validation-program)
before relying on it in an accreditation package.
# Configuration (/arc/configuration)
Arc reads `arc.toml` at startup, and every key in it can be overridden by an environment variable. These pages cover what each section controls.
# Configuration Overview (/arc/configuration/overview)
Arc uses a TOML configuration file (`arc.toml`) with environment variable overrides for flexibility.
## Configuration files [#configuration-files]
### Primary: arc.toml [#primary-arctoml]
The main configuration file with production-ready defaults:
```toml
# Server Configuration
[server]
# host = "" # bind address (default: all interfaces, dual-stack)
port = 8000
# Logging
[log]
level = "info" # debug, info, warn, error
format = "console" # json or console
# Database (query engine)
[database]
# Auto-detected if not set (recommended)
# max_connections = 28 # 2x CPU cores
# memory_limit = "8GB" # ~50% system RAM
# thread_count = 14 # CPU cores
enable_wal = false
# Storage Backend
[storage]
backend = "local" # local, s3, minio, azure, azblob
local_path = "./data/arc"
# Ingestion
[ingest]
max_buffer_size = 50000 # records before flush
max_buffer_age_ms = 5000 # ms before force flush
# flush_workers = 16 # async flush workers (auto-detected)
# flush_queue_size = 64 # pending flush queue (auto-detected)
# shard_count = 32 # buffer shards
# Compaction
[compaction]
enabled = true
hourly_enabled = true
hourly_min_age_hours = 0
hourly_min_files = 5
# Authentication
[auth]
enabled = true
# Delete Operations
[delete]
enabled = true
confirmation_threshold = 10000
max_rows_per_delete = 1000000
# Retention Policies
[retention]
enabled = true
# Continuous Queries
[continuous_query]
enabled = true
```
### Environment variables [#environment-variables]
Override any setting via environment variables with the `ARC_` prefix:
```bash
# Server
ARC_SERVER_HOST= # bind address (empty = all interfaces, v26.06.1+)
ARC_SERVER_PORT=8000
ARC_SERVER_TLS_ENABLED=false
ARC_SERVER_TLS_CERT_FILE=/path/to/cert.pem
ARC_SERVER_TLS_KEY_FILE=/path/to/key.pem
ARC_SERVER_MAX_PAYLOAD_SIZE=1GB # v26.01.1+
# Logging
ARC_LOG_LEVEL=info
ARC_LOG_FORMAT=json
# Database
ARC_DATABASE_MAX_CONNECTIONS=28
ARC_DATABASE_MEMORY_LIMIT=8GB
ARC_DATABASE_THREAD_COUNT=14
# Features
ARC_AUTH_ENABLED=true
ARC_COMPACTION_ENABLED=true
ARC_DELETE_ENABLED=true
ARC_RETENTION_ENABLED=true
ARC_CONTINUOUS_QUERY_ENABLED=true
ARC_MQTT_ENABLED=true # Enables the MQTT subsystem and API routes
# Ingestion Concurrency (v26.01.1+)
ARC_INGEST_FLUSH_WORKERS=32
ARC_INGEST_FLUSH_QUEUE_SIZE=200
ARC_INGEST_SHARD_COUNT=64
```
## Configuration priority [#configuration-priority]
Settings are applied in this order (highest to lowest):
1. **Environment variables** (e.g., `ARC_SERVER_PORT=8000`)
2. **arc.toml file**
3. **Built-in defaults**
## Storage backends [#storage-backends]
**Local Filesystem** - Default, simplest option for single-node deployments.
```toml
[storage]
backend = "local"
local_path = "./data/arc"
```
Environment variables:
```ini
ARC_STORAGE_BACKEND=local
ARC_STORAGE_LOCAL_PATH=./data/arc
```
**AWS S3** - Recommended for production cloud deployments.
```toml
[storage]
backend = "s3"
s3_bucket = "arc-production"
s3_region = "us-east-1"
# Credentials via env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
# Or use IAM roles (recommended)
```
Environment variables:
```ini
ARC_STORAGE_BACKEND=s3
ARC_STORAGE_S3_BUCKET=arc-data
ARC_STORAGE_S3_REGION=us-east-1
ARC_STORAGE_S3_ENDPOINT=s3.amazonaws.com
ARC_STORAGE_S3_ACCESS_KEY=your_key
ARC_STORAGE_S3_SECRET_KEY=your_secret
ARC_STORAGE_S3_USE_SSL=true
ARC_STORAGE_S3_PATH_STYLE=false
```
For EC2/EKS, omit `s3_access_key`/`s3_secret_key` entirely (leave them unset). Arc
then authenticates via the AWS credential chain — EC2 instance profile, **EKS IRSA
(IAM Roles for Service Accounts)**, **EKS Pod Identity**, or environment
credentials. This applies to both writes and query reads. Temporary
credentials (IRSA, instance roles, Pod Identity) are refreshed automatically
before each session expiry; this requires **Arc 26.09.1+** — on 26.06.2–26.06.3,
query reads stop working roughly one hour after each process start
([#600](https://github.com/Basekick-Labs/arc/issues/600)). Setting exactly one
of the two keys is rejected at startup; set both or neither.
**MinIO** - Self-hosted S3-compatible storage.
```toml
[storage]
backend = "minio"
s3_bucket = "arc"
s3_endpoint = "minio:9000"
s3_access_key = "minioadmin"
s3_secret_key = "minioadmin123"
s3_use_ssl = false
s3_path_style = true # Required for MinIO
```
Environment variables:
```ini
ARC_STORAGE_BACKEND=minio
ARC_STORAGE_S3_ENDPOINT=minio:9000
ARC_STORAGE_S3_BUCKET=arc
ARC_STORAGE_S3_ACCESS_KEY=minioadmin
ARC_STORAGE_S3_SECRET_KEY=minioadmin123
ARC_STORAGE_S3_USE_SSL=false
ARC_STORAGE_S3_PATH_STYLE=true
```
**Azure Blob Storage** - For Azure cloud deployments.
Azure Blob Storage support is available starting Arc v26.01.1.
```toml
[storage]
backend = "azure" # or "azblob"
azure_container = "arc-data"
azure_account_name = "your_account"
azure_account_key = "your_key"
# Or use managed identity:
# azure_use_managed_identity = true
```
Environment variables:
```ini
ARC_STORAGE_BACKEND=azure
ARC_STORAGE_AZURE_CONTAINER=arc-data
ARC_STORAGE_AZURE_ACCOUNT_NAME=your_account
ARC_STORAGE_AZURE_ACCOUNT_KEY=your_key
```
For Azure VMs/AKS, use managed identity for keyless authentication:
```toml
azure_use_managed_identity = true
```
Managed-identity (and service-principal environment) tokens are refreshed
automatically before each expiry on **both** writes and query reads —
this requires **Arc 26.09.1+**; on earlier versions query reads stop working
roughly an hour after each process start
([#605](https://github.com/Basekick-Labs/arc/issues/605)). A SAS token, when
configured, is deliberately never widened to a managed identity; `/health`
reports it as `sas / unknown`. `storage.azure_endpoint` now applies to query
reads as well (sovereign clouds); path-style endpoints such as Azurite apply
to writes only.
## Enterprise licensing [#enterprise-licensing]
Licensing (the `[license]` section — license keys, boot resilience, and the
air-gapped offline license file) is an Arc Enterprise feature and is documented
in the [Arc Enterprise configuration docs](/arc-enterprise/configuration/overview/#enterprise-license).
## Key configuration areas [#key-configuration-areas]
### Server [#server]
Basic HTTP server settings:
```toml
[server]
host = "" # bind address (default: empty = all interfaces, dual-stack IPv4 + IPv6)
port = 8000 # HTTP/HTTPS port to listen on
```
#### Bind address (`server.host`) [#bind-address-serverhost]
The `server.host` setting is available starting from Arc v26.06.1.
Controls which network interface Arc binds to. The default (empty string) preserves the historical behavior of binding to all interfaces with Linux dual-stack semantics (IPv4 + IPv6-mapped addresses).
Common values:
| Value | Effect |
| -------------- | ---------------------------------------------------------- |
| `""` (empty) | All interfaces, dual-stack (default — same as pre-26.06.1) |
| `"0.0.0.0"` | All interfaces, **IPv4 only** |
| `"::"` | All interfaces, IPv6 (with IPv4-mapped on Linux) |
| `"127.0.0.1"` | Loopback only (IPv4) — useful behind a reverse proxy |
| `"::1"` | Loopback only (IPv6) |
| `"192.0.2.10"` | Specific interface address |
Environment variable:
```ini
ARC_SERVER_HOST=127.0.0.1
```
If Arc sits behind nginx / Traefik / a Kubernetes Ingress on the same host, set `host = "127.0.0.1"` so Arc is not directly reachable from the network.
Setting `host = "0.0.0.0"` binds IPv4 only — IPv6 clients will be unable to connect. Leave the value empty (default) to keep dual-stack behavior.
### TLS/SSL (HTTPS) [#tlsssl-https]
Arc supports native HTTPS/TLS without requiring a reverse proxy:
```toml
[server]
port = 443
tls_enabled = true
tls_cert_file = "/etc/letsencrypt/live/example.com/fullchain.pem"
tls_key_file = "/etc/letsencrypt/live/example.com/privkey.pem"
```
Environment variables:
```ini
ARC_SERVER_TLS_ENABLED=true
ARC_SERVER_TLS_CERT_FILE=/path/to/cert.pem
ARC_SERVER_TLS_KEY_FILE=/path/to/key.pem
```
* **Native packages** (deb/rpm): Use native TLS for simple deployments
* **Docker/Kubernetes**: Use a reverse proxy (Traefik, nginx, Ingress) for TLS termination
* **Development**: Use self-signed certificates for local HTTPS testing
When TLS is enabled, Arc automatically:
* Adds the `Strict-Transport-Security` (HSTS) header
* Validates certificate and key files on startup
### Max payload size [#max-payload-size]
This configuration option is available starting from Arc v26.01.1.
Configure the maximum request payload size for write endpoints (msgpack, line protocol):
```toml
[server]
# Maximum payload size (applies to both compressed and decompressed)
# Supports units: B, KB, MB, GB
# Default: 1GB
max_payload_size = "1GB"
```
Environment variable:
```ini
ARC_SERVER_MAX_PAYLOAD_SIZE=2GB
```
If you're importing large datasets and encounter 413 errors, you can:
1. Increase `max_payload_size` (e.g., `"2GB"`)
2. Batch your imports into smaller chunks (recommended for reliability)
### Database (query engine) [#database-query-engine]
Query engine connection pool and resource settings:
```toml
[database]
# AUTO-DETECTION: If not set, Arc automatically configures:
# - max_connections: 2x CPU cores (min 4, max 64)
# - memory_limit: ~50% of system memory
# - thread_count: Number of CPU cores
# Manual override examples:
max_connections = 28 # Connection pool size
memory_limit = "8GB" # Query engine memory limit
thread_count = 14 # Query execution threads
enable_wal = false # Query engine WAL (not Arc WAL)
# Allow the engine to reorder results of queries WITHOUT an ORDER BY (SQL-standard
# semantics); can reduce memory usage on large un-ordered scans and exports.
# Queries with an explicit ORDER BY are unaffected. Set true to make un-ordered
# SELECTs return rows in file/insertion order (pre-26.09.1 behavior).
preserve_insertion_order = false
```
### Ingestion [#ingestion]
Buffer and concurrency settings for write performance:
```toml
[ingest]
# Maximum records to buffer before flushing to Parquet
max_buffer_size = 50000
# Maximum age (ms) before forcing a flush
max_buffer_age_ms = 5000
# Concurrency settings (auto-detected if not set)
# flush_workers = 16 # async flush workers (2x CPU cores, min 8, max 64)
# flush_queue_size = 64 # pending flush queue (4x workers, min 100)
# shard_count = 32 # buffer shards for lock distribution
# Decimal128 precision columns (v26.04.1+)
# decimal_columns = ["trades:price=18,8;amount=18,8", "balances:balance=38,18"]
# default_decimal_columns = ""
# Parquet dictionary encoding at ingest (v26.09.1+). Off by default: ingest
# files are transient — hourly/daily compaction rewrites them,
# which re-encodes every column adaptively — and skipping ingest-time
# dictionaries improves sustained write throughput ~26%. The tradeoff is a
# temporarily larger uncompacted hot partition until the next compaction.
# Set use_dictionary = true to dictionary-encode string columns at ingest
# (numeric columns stay plain); add numeric_dictionary = true to restore
# the full pre-26.09.1 encoding.
# use_dictionary = false
# numeric_dictionary = false
```
Environment variables: `ARC_INGEST_USE_DICTIONARY`, `ARC_INGEST_NUMERIC_DICTIONARY` (v26.09.1+).
Data flushes when **either** condition is met:
1. Buffer reaches `max_buffer_size` records
2. Buffer age exceeds `max_buffer_age_ms`
For deployments with many concurrent clients (50+), increase `flush_workers` and `flush_queue_size`:
```toml
[ingest]
flush_workers = 32
flush_queue_size = 200
shard_count = 64
```
For financial or scientific data requiring exact decimal precision, configure decimal columns to store values as native Parquet DECIMAL instead of float64. See the [Decimal Precision guide](/arc/guides/decimal-precision/) for details.
### Compaction [#compaction]
Automatic file optimization:
```toml
[compaction]
enabled = true
hourly_enabled = true
hourly_min_age_hours = 0 # Files must be this old
hourly_min_files = 5 # Minimum files to trigger
daily_enabled = false # Daily tier (optional)
daily_min_age_hours = 24
daily_min_files = 3
```
### Authentication [#authentication]
Token-based API authentication:
```toml
[auth]
enabled = true # Enable/disable auth
db_path = "./data/arc_auth.db" # Token database
cache_ttl = 30 # Token cache TTL (seconds)
max_cache_size = 1000 # Max cached tokens
```
### Delete operations [#delete-operations]
Safe deletion with confirmation:
```toml
[delete]
enabled = true
confirmation_threshold = 10000 # Require confirmation above this
max_rows_per_delete = 1000000 # Hard limit per operation
```
### Query [#query]
Query execution limits:
```toml
[query]
timeout = 300 # Query execution timeout in seconds (0 = no timeout)
```
### Retention policies [#retention-policies]
Automatic data expiration:
```toml
[retention]
enabled = true
db_path = "./data/arc_retention.db"
```
### Continuous queries [#continuous-queries]
Scheduled automated queries:
```toml
[continuous_query]
enabled = true
db_path = "./data/arc_cq.db"
```
### Write-Ahead Log (WAL) [#write-ahead-log-wal]
Optional durability guarantee:
```toml
[wal]
enabled = false # Enable for zero data loss
directory = "./data/wal"
sync_mode = "fdatasync" # none, fdatasync, fsync
max_size_mb = 500
max_age_seconds = 3600
```
### Metrics [#metrics]
Timeseries metrics collection:
```toml
[metrics]
timeseries_retention_minutes = 60
timeseries_interval_seconds = 10
```
## Quick configuration examples [#quick-configuration-examples]
```toml
[server]
port = 8000
[log]
level = "debug"
format = "console"
[storage]
backend = "local"
local_path = "./dev_data"
[auth]
enabled = false
[compaction]
enabled = false
```
```toml
[server]
port = 8000
[log]
level = "info"
format = "json"
[database]
max_connections = 32
memory_limit = "16GB"
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
[ingest]
max_buffer_size = 100000
max_buffer_age_ms = 10000
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
daily_enabled = true
[wal]
enabled = true
sync_mode = "fdatasync"
```
```toml
[server]
port = 8000
[log]
level = "info"
format = "json"
[storage]
backend = "s3"
s3_bucket = "arc-production"
s3_region = "us-east-1"
# Use IAM roles for credentials
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
```
```toml
[server]
port = 443
tls_enabled = true
tls_cert_file = "/etc/letsencrypt/live/example.com/fullchain.pem"
tls_key_file = "/etc/letsencrypt/live/example.com/privkey.pem"
[log]
level = "info"
format = "json"
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
```
```toml
[server]
port = 8000
[storage]
backend = "minio"
s3_bucket = "arc"
s3_endpoint = "minio:9000"
[wal]
enabled = true
sync_mode = "fdatasync"
directory = "/var/lib/arc/wal"
max_size_mb = 1000
max_age_seconds = 3600
[compaction]
enabled = true
```
```toml
# Optimized for 50+ concurrent clients (e.g., many Telegraf agents)
[server]
port = 8000
[log]
level = "info"
format = "json"
[database]
max_connections = 64
memory_limit = "16GB"
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
[ingest]
max_buffer_size = 100000
max_buffer_age_ms = 10000
# Scale concurrency for many clients
flush_workers = 32 # More workers for parallel I/O
flush_queue_size = 200 # Larger queue for burst handling
shard_count = 64 # More shards to reduce lock contention
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
```
## Best practices [#best-practices]
### 1. Use arc.toml for permanent settings [#1-use-arctoml-for-permanent-settings]
Store configuration in `arc.toml` and version control it (without secrets):
```toml
[storage]
backend = "s3"
s3_bucket = "arc"
s3_region = "us-east-1"
# Credentials via environment variables
```
### 2. Use environment variables for secrets [#2-use-environment-variables-for-secrets]
```bash
export ARC_STORAGE_S3_ACCESS_KEY="your_access_key"
export ARC_STORAGE_S3_SECRET_KEY="your_secret_key"
```
### 3. Let Arc auto-detect resources [#3-let-arc-auto-detect-resources]
Arc automatically detects optimal query engine settings based on your system. Only override if you have specific requirements:
```toml
[database]
# Leave commented for auto-detection
# max_connections = 28
# memory_limit = "8GB"
# thread_count = 14
```
### 4. Enable features progressively [#4-enable-features-progressively]
Start simple, add features as needed:
1. Basic configuration (storage + auth)
2. Compaction (for query optimization)
3. Retention policies (for data management)
4. WAL (for zero data loss guarantee)
### 5. Monitor configuration impact [#5-monitor-configuration-impact]
Check metrics after configuration changes:
```bash
# Memory usage
curl http://localhost:8000/api/v1/metrics/memory
# Query performance
curl http://localhost:8000/api/v1/metrics/query-pool
# Compaction status
curl http://localhost:8000/api/v1/compaction/status
```
## Troubleshooting [#troubleshooting]
### Configuration not loading [#configuration-not-loading]
```bash
# Verify TOML syntax (use any TOML validator)
# Check file exists in expected location
ls -la arc.toml
# Arc looks for arc.toml in:
# 1. Current directory
# 2. /etc/arc/arc.toml (native install)
```
### Environment variables not working [#environment-variables-not-working]
```bash
# Verify they're set
env | grep ARC_
# Use correct prefix and format
export ARC_SERVER_PORT=8000 # Correct
export SERVER_PORT=8000 # Wrong - missing ARC_ prefix
```
### Resource issues [#resource-issues]
```bash
# Check current settings via metrics
curl http://localhost:8000/api/v1/metrics/memory
# Adjust in arc.toml:
[database]
memory_limit = "4GB"
max_connections = 16
```
## Next steps [#next-steps]
* **[Authentication](/arc/configuration/authentication/)** - Token management
* **[Advanced Features](/arc/advanced/compaction/)** - Compaction and WAL
# Storage File Format (/arc/configuration/storage-file-format)
Arc stores measurement data as columnar files on disk. The format is **Apache Parquet**
for every deployment and every storage backend — there is no setting to change it.
## Parquet everywhere [#parquet-everywhere]
Parquet is used across all storage backends (local, S3, MinIO, Azure) and by every Arc
feature: queries, compaction, retention, delete, backup and restore, and tiered storage.
Because the files are ordinary Parquet, they are also readable directly by external
tools — DuckDB, Spark, Polars, pandas — without going through Arc.
There is no `storage.file_format` key, and no pluggable on-disk format. A `file_format`
setting in an `arc.toml` is ignored.
## Compression [#compression]
Compression differs between freshly-ingested files and compacted files:
| Stage | Codec | Configurable |
| ------------------------ | ----------------- | -------------------------- |
| Ingest (flushed buffers) | Snappy by default | Yes — `ingest.compression` |
| After compaction | ZSTD | No — fixed |
Snappy keeps the ingest hot path cheap. Compaction rewrites those files with ZSTD, which
is the main reason compacted files are substantially smaller than the files they replace.
See the [configuration overview](/arc/configuration/overview/) for `ingest.compression`,
and [file compaction](/arc/advanced/compaction/) for how and when files are rewritten.
## Layout on disk [#layout-on-disk]
Files are laid out by database, measurement, and hour:
```text
arc/ # Bucket or local path
└── default/ # Database
└── cpu/ # Measurement
└── 2025/10/08/ # Date
└── 14/ # Hour
├── file1.parquet
└── file2.parquet
```
This is what makes partition pruning effective: a query with a time bound can skip whole
directories without opening the files inside them. See
[data time partitioning](/arc/advanced/data-time-partitioning/) for details.
# CSV Import (/arc/data-import/csv)
Import CSV files into Arc via the REST API. Arc parses the file in-process, infers column types, partitions data by hour, and writes optimized Parquet files to storage through the same streaming ingestion pipeline used for Line Protocol writes.
CSV bulk import is available starting Arc v26.02.1 (February 2026).
CSV import now parses rows in-process instead of reading the uploaded file through the query engine. The request and response are unchanged, with stricter up-front validation: empty files, duplicate column names, and a `time_column` rename that would collide with an existing `time` column are now rejected with `400` before any data is ingested.
## Endpoint [#endpoint]
```bash
POST /api/v1/import/csv
```
## Headers [#headers]
| Header | Required | Default | Description |
| ---------------- | -------- | ------- | ---------------------------------------------- |
| `Authorization` | Yes | - | `Bearer $ARC_TOKEN` |
| `X-Arc-Database` | Yes | - | Target database name (or use `db` query param) |
## Query parameters [#query-parameters]
| Parameter | Required | Default | Description |
| ------------- | -------- | ----------- | -------------------------------------------------------------------------------------------------- |
| `measurement` | Yes | - | Target measurement name |
| `time_column` | No | `time` | Name of the timestamp column in the CSV |
| `time_format` | No | auto-detect | Timestamp format: `epoch_s`, `epoch_ms`, `epoch_us`, `epoch_ns`, or leave empty for auto-detection |
| `delimiter` | No | `,` | Column delimiter character |
| `skip_rows` | No | `0` | Number of header/metadata rows to skip before the CSV header |
## Basic example [#basic-example]
```bash
curl -X POST "http://localhost:8000/api/v1/import/csv?measurement=sensors" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: iot" \
-F "file=@sensor_data.csv"
```
## Example with options [#example-with-options]
```bash
# TSV file with epoch seconds and 2 metadata rows to skip
curl -X POST "http://localhost:8000/api/v1/import/csv?measurement=telemetry&time_column=ts&time_format=epoch_s&delimiter=%09&skip_rows=2" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: satellites" \
-F "file=@telemetry_export.tsv"
```
## Response [#response]
```json
{
"status": "ok",
"result": {
"database": "iot",
"measurement": "sensors",
"rows_imported": 50000,
"partitions_created": 3,
"time_range_min": "2026-01-15T00:00:00Z",
"time_range_max": "2026-01-15T02:30:00Z",
"columns": ["time", "temperature", "humidity", "device_id"],
"duration_ms": 245
}
}
```
## Notes [#notes]
* The `measurement` parameter is **required** -- unlike Line Protocol import where measurements are embedded in the data.
* The time column is renamed to `time` in the output Parquet files.
* Data is automatically partitioned by hour for optimal query performance.
* Maximum file size: **500 MB**.
* RBAC: write permissions are checked for the target measurement.
* Column types are inferred per column from the values: a column is `BIGINT` if every value parses as an integer, otherwise `DOUBLE` if every value parses as a number, otherwise `BOOLEAN` if every value is `true`/`false`, otherwise `VARCHAR`. Empty cells in a numeric/boolean column are stored as null.
* The time column accepts integer epochs, **fractional epochs** (e.g. `1609459200.123`, sub-second precision preserved), or timestamp strings (RFC 3339, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, or `YYYY-MM-DD`). With `time_format` empty, the unit of a numeric epoch is auto-detected by magnitude (s/ms/µs/ns).
## Error responses [#error-responses]
| Status | Description |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400` | Missing database/measurement/file; empty file or no data rows; `time_column` not found; empty, blank, or duplicate column names; or a `time_column` rename that collides with an existing `time` column |
| `403` | Insufficient write permissions |
| `413` | File exceeds 500 MB size limit |
| `422` | Malformed CSV rows, or an unparseable value in the time column |
| `500` | Import execution error |
# Data Import (/arc/data-import)
Bulk import is for loading data you already have. For continuous writes, use the ingestion endpoints in the [API reference](/arc/api-reference/overview/) instead.
# Line Protocol Bulk Import (/arc/data-import/line-protocol)
Import InfluxDB Line Protocol files into Arc. Enables one-command migration from InfluxDB by uploading `.lp` or `.txt` files (plain or gzip-compressed).
Line Protocol bulk import is available starting Arc v26.02.1 (February 2026).
This page covers **bulk file import** via `POST /api/v1/import/lp`. For streaming ingestion of Line Protocol data (real-time writes), see the [Line Protocol write endpoints](/arc/api-reference/overview/#data-ingestion) in the API Reference.
## Endpoint [#endpoint]
```bash
POST /api/v1/import/lp
```
## Headers [#headers]
| Header | Required | Default | Description |
| ---------------- | -------- | ------- | ---------------------------------------------- |
| `Authorization` | Yes | - | `Bearer $ARC_TOKEN` |
| `X-Arc-Database` | Yes | - | Target database name (or use `db` query param) |
## Query parameters [#query-parameters]
| Parameter | Required | Default | Description |
| ------------- | -------- | ------- | ----------------------------------------------- |
| `measurement` | No | *(all)* | Filter to a single measurement from the LP file |
| `precision` | No | `ns` | Timestamp precision: `ns`, `us`, `ms`, `s` |
## Basic example [#basic-example]
```bash
curl -X POST "http://localhost:8000/api/v1/import/lp" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: mydb" \
-F "file=@export.lp"
```
## Example with precision [#example-with-precision]
```bash
# Import LP file with second-precision timestamps
curl -X POST "http://localhost:8000/api/v1/import/lp?precision=s" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: mydb" \
-F "file=@export_seconds.lp"
```
## Response [#response]
```json
{
"status": "ok",
"result": {
"database": "mydb",
"measurements": ["cpu", "mem", "disk"],
"rows_imported": 150000,
"precision": "ns",
"duration_ms": 342
}
}
```
## InfluxDB migration [#influxdb-migration]
Export from InfluxDB and import directly into Arc. For InfluxDB 1.x, use `influx_inspect export` (the `influx` CLI does not support Line Protocol output):
```bash
# Export from InfluxDB 1.x (reads TSM and WAL directly)
influx_inspect export \
-datadir /var/lib/influxdb/data \
-waldir /var/lib/influxdb/wal \
-database mydb -lponly -out export.lp
# Import to Arc
curl -X POST "http://localhost:8000/api/v1/import/lp" \
-H "X-Arc-Database: mydb" \
-H "Authorization: Bearer $ARC_TOKEN" \
-F "file=@export.lp"
```
`influx_inspect export` writes one Line Protocol line per field, so a multi-field
point becomes several rows in Arc, each with one field populated. For multi-field
measurements, use [tsm2arc](https://github.com/Basekick-Labs/tsm2arc), which rejoins
fields into a single point. See the full [InfluxDB migration guide](/arc/migration/influxdb/).
## How it works [#how-it-works]
Data flows through Arc's high-performance columnar ingest pipeline (ArrowBuffer -> ArrowWriter -> Parquet -> storage) -- the same path used by streaming LP ingestion. This means bulk imports benefit from the same throughput, sort optimization, and hourly partitioning.
## Notes [#notes]
* **Multi-measurement** -- a single LP file can contain multiple measurements; all are imported in one request.
* **Precision-aware** -- timestamps are losslessly converted from the specified precision to Arc's internal microsecond format.
* **Gzip support** -- compressed files (`.lp.gz`) are automatically detected and decompressed via magic bytes.
* **RBAC** -- write permissions are checked for every measurement in the file.
* Maximum file size: **500 MB** (after decompression).
## Error responses [#error-responses]
| Status | Description |
| ------ | ----------------------------------------------------------- |
| `400` | Missing database, invalid precision, or no file uploaded |
| `403` | Insufficient write permissions for one or more measurements |
| `413` | File exceeds 500 MB size limit |
| `500` | Import execution error |
# Parquet Import (/arc/data-import/parquet)
Import existing Parquet files directly into Arc. Useful for data lake integration, analytics pipeline output, or migrating from other columnar stores.
Parquet bulk import is available starting Arc v26.02.1 (February 2026).
Parquet import now reads the file in-process (via Apache Arrow) instead of through the query engine. The request and response are unchanged. Two things to be aware of: `DECIMAL` columns are imported as `DOUBLE`, and empty files / duplicate column names / a `time_column` rename that collides with an existing `time` column are rejected with `400`.
## Endpoint [#endpoint]
```bash
POST /api/v1/import/parquet
```
## Headers [#headers]
| Header | Required | Default | Description |
| ---------------- | -------- | ------- | ---------------------------------------------- |
| `Authorization` | Yes | - | `Bearer $ARC_TOKEN` |
| `X-Arc-Database` | Yes | - | Target database name (or use `db` query param) |
## Query parameters [#query-parameters]
| Parameter | Required | Default | Description |
| ------------- | -------- | ------- | ------------------------------------------------ |
| `measurement` | Yes | - | Target measurement name |
| `time_column` | No | `time` | Name of the timestamp column in the Parquet file |
## Example [#example]
```bash
curl -X POST "http://localhost:8000/api/v1/import/parquet?measurement=metrics" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: production" \
-F "file=@data_export.parquet"
```
## Response [#response]
```json
{
"status": "ok",
"result": {
"database": "production",
"measurement": "metrics",
"rows_imported": 1200000,
"partitions_created": 8,
"time_range_min": "2026-01-01T00:00:00Z",
"time_range_max": "2026-01-01T07:45:00Z",
"columns": ["time", "host", "region", "cpu_usage", "mem_usage"],
"duration_ms": 890
}
}
```
## Notes [#notes]
* The Parquet file must contain a timestamp column (default name: `time`). Use the `time_column` parameter if your column has a different name. The time column may be an Arrow `TIMESTAMP` (any unit), an integer epoch column (any width), a floating-point epoch column (use `time_format`, or auto-detect by magnitude), or a timestamp string column.
* Arc reads the Parquet file in-process via Apache Arrow and repartitions the data into Arc's hourly partition layout regardless of the source file's structure. Supported column types: integer (all widths, signed and unsigned), floating point, boolean, string, binary/byte-array, decimal (imported as `DOUBLE`), and timestamp.
* `DECIMAL` columns are imported as `DOUBLE`. If you need exact decimal precision, use Line Protocol ingestion with a configured decimal column.
* Maximum file size: **500 MB**.
* RBAC: write permissions are checked for the target measurement.
## Error responses [#error-responses]
| Status | Description |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400` | Missing database/measurement/file; empty file or no rows; `time_column` not found; empty or duplicate column names; a `time_column` rename that collides with an existing `time` column; or a `NaN`/`Inf` value in a floating-point time column |
| `403` | Insufficient write permissions |
| `413` | File exceeds 500 MB size limit |
| `422` | Unreadable Parquet file, or an unsupported column type |
| `500` | Import execution error |
# Continuous Queries (/arc/data-lifecycle/continuous-queries)
Continuous queries enable automatic downsampling and aggregation of data into materialized views, reducing storage requirements while maintaining queryable historical data.
Automatic execution is an [Arc Enterprise](/arc-enterprise/operations/automated-scheduling/) feature. In Arc OSS, continuous queries must be manually triggered via the API.
## Overview [#overview]
Continuous queries in Arc help you:
* **Downsample Data**: Aggregate high-frequency data into lower-frequency summaries
* **Reduce Storage**: Store aggregated data instead of raw metrics
* **Maintain History**: Keep long-term trends without full granularity
* **Improve Query Performance**: Query pre-aggregated data for faster results
* **Create Materialized Views**: Automatically maintain aggregated datasets
## How it works [#how-it-works]
Continuous queries use standard analytical SQL to aggregate data from source measurements into destination measurements:
1. **Define Query**: Specify aggregation logic using SQL
2. **Set Schedule**: Configure time intervals for grouping (e.g., hourly, daily)
3. **Execute Manually**: Trigger execution via API with start/end times
4. **Store Results**: Write aggregated data to a new measurement
5. **Apply Retention**: Optionally set custom retention for aggregated data
### Architecture [#architecture]
```bash
Source Measurement (cpu)
↓
Continuous Query (AVG, MAX, MIN, etc.)
↓
Destination Measurement (cpu_hourly)
↓
Optional: Retention Policy
```
### Execution model and semantics [#execution-model-and-semantics]
Continuous queries are **recomputed once per interval**, not maintained as an
incremental/streaming aggregate. There is no running state carried between runs.
On each scheduled run, Arc:
1. Computes the time window `[last_processed_time, now)` — the slice since the
previous successful run (on the very first run it defaults to the last hour).
2. Substitutes that range into your query's `{start_time}` / `{end_time}`
placeholders and runs the aggregation **fresh over the full set of source rows
in that window** (read from Parquet).
3. Writes the results to the destination measurement, stamped with the window
start time and tagged so duplicate windows can be deduped (see
[Idempotency and `tag_columns`](#idempotency-and-tag_columns)).
4. Advances `last_processed_time` to the window's end.
Because the watermark advances to the end of each processed window, **windows are
tumbling and non-overlapping** — the next run starts where the previous one
ended; earlier windows are not revisited.
- **Late / out-of-order data is not reprocessed.** A window is computed once and
the watermark then moves past it. Rows that arrive *after* their window has
already been processed (late events, corrections, backfills) are **not folded
back** into that window's aggregate. There is no lookback/grace window today.
- **Window boundaries are processing-time based** (`now` at run time), not
event-time. A window's correctness assumes its data has been ingested by the
time the interval fires.
- **Output is idempotent at compaction, not atomically exactly-once.** A retry,
an overlapping manual re-run, or a crash between the write and the watermark
update re-emits a window's rows, so duplicates can appear transiently. They are
collapsed to one row per `(dimensions, time)` the next time the destination
partition compacts — provided you declared the grouping dimensions in
[`tag_columns`](#idempotency-and-tag_columns) (a query with no grouping is
deduped automatically). Until compaction runs, a query over the destination may
see the duplicate rows.
This model fits **periodic roll-ups and downsampling of in-order, timely data**
(for example building 1m/5m bars from a clean feed). If your workload involves
late corrections, out-of-order events, or strict synchronous exactly-once
aggregation, design around these semantics (e.g. reprocess on a delay, or do the
final roll-up as a query-time aggregation).
## API endpoints [#api-endpoints]
### Create continuous query [#create-continuous-query]
Define a new continuous query:
```bash
POST /api/v1/continuous_queries
```
**Request Body**:
```json
{
"name": "cpu_hourly_avg",
"database": "telegraf",
"source_measurement": "cpu",
"destination_measurement": "cpu_hourly",
"query": "SELECT time_bucket('1 hour', time) AS time, host, AVG(usage_idle) AS avg_usage_idle, AVG(usage_user) AS avg_usage_user, COUNT(*) AS sample_count FROM telegraf.cpu GROUP BY time_bucket('1 hour', time), host",
"interval": "1h",
"tag_columns": ["host"],
"retention_policy": "90d",
"is_active": true
}
```
**Parameters**:
* `name` (string, required): Unique query identifier
* `database` (string, required): Target database name
* `source_measurement` (string, required): Source measurement to aggregate
* `destination_measurement` (string, required): Where to store results
* `query` (string, required): SQL aggregation query
* `interval` (string, required): Time bucket interval (`1m`, `5m`, `1h`, `1d`, etc.)
* `tag_columns` (array of strings, optional): The **grouping dimension** columns in the query's output (e.g. `["host"]` for `GROUP BY host`). See [Idempotency and `tag_columns`](#idempotency-and-tag_columns) below — set this for any `GROUP BY` query so re-runs don't produce duplicate rows.
* `retention_policy` (string, optional): Retention for aggregated data (e.g., `90d`, `365d`)
* `is_active` (boolean, required): Enable/disable the query
#### Idempotency and `tag_columns` [#idempotency-and-tag_columns]
Continuous-query output is made idempotent by Arc's compaction step: duplicate emissions of the same window (from a retry, an overlapping manual run, or a crash between the write and the watermark advance) are collapsed to one row per `(grouping dimensions, time)` when the destination partition compacts.
For this to work, Arc must know which output columns are the grouping dimensions:
* **A query with `GROUP BY `** (e.g. `GROUP BY host`) must list those dimensions in `tag_columns` (e.g. `"tag_columns": ["host"]`). Arc writes them as Parquet tag metadata and dedups on `(tags, time)`.
* **A query with no grouping** (one row per window, e.g. `SELECT AVG(x) …`) needs no `tag_columns` — Arc detects it produces one row per timestamp and dedups on time automatically.
* **If you group but forget to declare `tag_columns`**, Arc detects the multiple-rows-per-timestamp output and does **not** dedup it (to avoid deleting distinct series). The query still runs and its rows are correct, but duplicate windows will accumulate; a warning is logged asking you to add `tag_columns`.
`tag_columns` may not include `time` (time is always part of the dedup key). Names must be plain identifiers (letters, digits, `_`, `-`). Output-row timestamps are stamped with the window's start time; a query that does not select a `time` column now gets the correct window timestamp instead of the ingestion wall-clock.
This is not a breaking change. Existing continuous queries keep running with no action — the CQ database is migrated automatically and `tag_columns` is optional. Add `tag_columns` to a grouped CQ when you want its duplicate windows to collapse; until you do, it behaves exactly as before (append-only). Note that a CQ which does **not** select a `time` column now stamps output with the window start rather than the ingestion wall-clock, so its destination has a one-time timestamp discontinuity at the upgrade.
### List continuous queries [#list-continuous-queries]
Retrieve all continuous queries:
```bash
GET /api/v1/continuous_queries
```
**Response**:
```json
[
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"name": "cpu_hourly_avg",
"database": "telegraf",
"source_measurement": "cpu",
"destination_measurement": "cpu_hourly",
"interval": "1h",
"tag_columns": ["host"],
"retention_policy": "90d",
"is_active": true,
"created_at": "2024-01-15T10:30:00Z",
"last_executed_at": "2024-01-20T02:00:00Z"
}
]
```
### Get single query [#get-single-query]
Retrieve a specific continuous query:
```bash
GET /api/v1/continuous_queries/{query_id}
```
### Update continuous query [#update-continuous-query]
Update an existing continuous query:
```bash
PUT /api/v1/continuous_queries/{query_id}
```
**Request Body**: Same as create query
### Delete continuous query [#delete-continuous-query]
Remove a continuous query:
```bash
DELETE /api/v1/continuous_queries/{query_id}
```
Deleting a continuous query does not delete the destination measurement or its data. The aggregated data remains queryable.
### Execute continuous query [#execute-continuous-query]
Manually trigger a continuous query:
```bash
POST /api/v1/continuous_queries/{query_id}/execute
```
**Request Body**:
```json
{
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-31T23:59:59Z",
"dry_run": false
}
```
**Parameters**:
* `start_time` (string, required): Start timestamp (ISO 8601 format)
* `end_time` (string, required): End timestamp (ISO 8601 format)
* `dry_run` (boolean, optional): Test without writing data (default: `false`)
**Response**:
```json
{
"query_id": "550e8400-e29b-41d4-a716-446655440000",
"rows_processed": 1000000,
"rows_written": 720,
"execution_time_ms": 2500,
"time_range": {
"start": "2024-01-01T00:00:00Z",
"end": "2024-01-31T23:59:59Z"
},
"dry_run": false
}
```
### View execution history [#view-execution-history]
View past executions of a continuous query:
```bash
GET /api/v1/continuous_queries/{query_id}/executions?limit=50
```
**Response**:
```json
[
{
"execution_id": "abc123",
"executed_at": "2024-01-20T02:00:00Z",
"start_time": "2024-01-19T00:00:00Z",
"end_time": "2024-01-20T00:00:00Z",
"rows_processed": 86400,
"rows_written": 24,
"execution_time_ms": 1200,
"status": "success"
}
]
```
## Query syntax [#query-syntax]
Continuous queries use standard analytical SQL with temporal optimizations.
### Recommended approach [#recommended-approach]
Use `epoch_us()` for timestamp conversion and `date_trunc()` for time bucketing:
```sql
SELECT
date_trunc('hour', epoch_us(time)) AS time,
host,
AVG(usage_idle) AS avg_usage_idle,
MAX(usage_user) AS max_usage_user,
MIN(usage_system) AS min_usage_system,
COUNT(*) AS sample_count
FROM telegraf.cpu
GROUP BY date_trunc('hour', epoch_us(time)), host
```
### Common aggregations [#common-aggregations]
These are common examples. Arc supports the **full analytical SQL aggregate set** — `MEDIAN`, `MODE`, `QUANTILE_CONT`, `APPROX_QUANTILE`, `ARG_MAX`, `HISTOGRAM`, `CORR`, `REGR_*` and the rest all work. See the [Querying guide](/arc/guides/querying/#useful-sql-functions) for more.
* `AVG()` - Average values
* `SUM()` - Sum of values
* `MIN()` - Minimum value
* `MAX()` - Maximum value
* `COUNT()` - Row count
* `STDDEV()` - Standard deviation
* `PERCENTILE_CONT()` - Percentile calculations
### Time bucketing [#time-bucketing]
**Using `date_trunc()`**:
```sql
-- Hourly buckets
date_trunc('hour', epoch_us(time))
-- Daily buckets
date_trunc('day', epoch_us(time))
-- 5-minute buckets (requires rounding)
date_trunc('hour', epoch_us(time)) + INTERVAL '5 minutes' * floor(extract(minute from epoch_us(time)) / 5)
```
### Including sample counts [#including-sample-counts]
Always include `COUNT(*)` to track how many raw samples each aggregate represents:
```sql
SELECT
date_trunc('hour', epoch_us(time)) AS time,
host,
AVG(usage_idle) AS avg_usage_idle,
COUNT(*) AS sample_count -- Important for data quality
FROM telegraf.cpu
GROUP BY date_trunc('hour', epoch_us(time)), host
```
## Usage examples [#usage-examples]
### Example 1: hourly CPU metrics [#example-1-hourly-cpu-metrics]
Aggregate per-second CPU metrics into hourly averages:
```python
import os
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create continuous query
response = requests.post(
"http://localhost:8000/api/v1/continuous_queries",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "cpu_hourly",
"database": "telegraf",
"source_measurement": "cpu",
"destination_measurement": "cpu_hourly",
"query": """
SELECT
date_trunc('hour', epoch_us(time)) AS time,
host,
AVG(usage_idle) AS avg_usage_idle,
AVG(usage_user) AS avg_usage_user,
AVG(usage_system) AS avg_usage_system,
MAX(usage_user) AS max_usage_user,
COUNT(*) AS sample_count
FROM telegraf.cpu
GROUP BY date_trunc('hour', epoch_us(time)), host
""",
"interval": "1h",
"tag_columns": ["host"], # grouping dimension → idempotent output
"retention_policy": "365d",
"is_active": True
}
)
query_id = response.json()["id"]
# Execute for the last 30 days
from datetime import datetime, timedelta
end_time = datetime.utcnow()
start_time = end_time - timedelta(days=30)
result = requests.post(
f"http://localhost:8000/api/v1/continuous_queries/{query_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"start_time": start_time.isoformat() + "Z",
"end_time": end_time.isoformat() + "Z"
}
)
print(f"Processed {result.json()['rows_processed']} rows")
print(f"Generated {result.json()['rows_written']} aggregated rows")
```
### Example 2: daily request summary [#example-2-daily-request-summary]
Aggregate API request logs into daily summaries:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create daily request summary
response = requests.post(
"http://localhost:8000/api/v1/continuous_queries",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "requests_daily",
"database": "logs",
"source_measurement": "api_requests",
"destination_measurement": "api_requests_daily",
"query": """
SELECT
date_trunc('day', epoch_us(time)) AS time,
endpoint,
status_code,
COUNT(*) AS total_requests,
AVG(response_time_ms) AS avg_response_time,
MAX(response_time_ms) AS max_response_time,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY response_time_ms) AS p95_response_time
FROM api_requests
GROUP BY date_trunc('day', epoch_us(time)), endpoint, status_code
""",
"interval": "1d",
"tag_columns": ["endpoint", "status_code"], # grouping dimensions → idempotent output
"retention_policy": "730d", # 2 years
"is_active": True
}
)
```
### Example 3: 5-minute sensor readings [#example-3-5-minute-sensor-readings]
Downsample IoT sensor data to 5-minute intervals:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create 5-minute sensor aggregation
response = requests.post(
"http://localhost:8000/api/v1/continuous_queries",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "sensors_5min",
"database": "iot",
"source_measurement": "temperature",
"destination_measurement": "temperature_5min",
"query": """
SELECT
date_trunc('hour', epoch_us(time)) +
INTERVAL '5 minutes' * floor(extract(minute from epoch_us(time)) / 5) AS time,
sensor_id,
location,
AVG(temperature) AS avg_temperature,
MIN(temperature) AS min_temperature,
MAX(temperature) AS max_temperature,
COUNT(*) AS sample_count
FROM temperature
GROUP BY
date_trunc('hour', epoch_us(time)) +
INTERVAL '5 minutes' * floor(extract(minute from epoch_us(time)) / 5),
sensor_id,
location
""",
"interval": "5m",
"tag_columns": ["sensor_id", "location"], # grouping dimensions → idempotent output
"retention_policy": "90d",
"is_active": True
}
)
```
### Example 4: dry run testing [#example-4-dry-run-testing]
Test a continuous query before execution:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create the query
response = requests.post(
"http://localhost:8000/api/v1/continuous_queries",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={...}
)
query_id = response.json()["id"]
# Test with dry run
dry_run = requests.post(
f"http://localhost:8000/api/v1/continuous_queries/{query_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-02T00:00:00Z",
"dry_run": True
}
)
print(f"Would process {dry_run.json()['rows_processed']} rows")
print(f"Would generate {dry_run.json()['rows_written']} aggregated rows")
# If satisfied, execute for real
if dry_run.json()['rows_written'] > 0:
result = requests.post(
f"http://localhost:8000/api/v1/continuous_queries/{query_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-02T00:00:00Z",
"dry_run": False
}
)
```
## Storage benefits [#storage-benefits]
Continuous queries significantly reduce storage requirements:
### Before downsampling [#before-downsampling]
**Raw CPU metrics** (1-second intervals):
* 1 year = 31,536,000 rows per host
* 10 hosts = 315,360,000 rows
* Storage: \~20GB
### After downsampling to hourly [#after-downsampling-to-hourly]
**Hourly aggregates**:
* 1 year = 8,760 rows per host
* 10 hosts = 87,600 rows
* Storage: \~50MB
**Reduction**: \~400x smaller while maintaining hourly trend visibility.
### Multi-tier strategy [#multi-tier-strategy]
Combine different granularities for optimal storage:
```python
# Tier 1: Keep raw data for 7 days
# Tier 2: Hourly aggregates for 90 days
requests.post("/api/v1/continuous_queries", json={
"name": "cpu_hourly",
"interval": "1h",
"retention_policy": "90d"
})
# Tier 3: Daily aggregates for 2 years
requests.post("/api/v1/continuous_queries", json={
"name": "cpu_daily",
"source_measurement": "cpu_hourly", # Aggregate the hourly data
"destination_measurement": "cpu_daily",
"interval": "1d",
"retention_policy": "730d"
})
# Use retention policy to delete raw data after 7 days
requests.post("/api/v1/retention", json={
"database": "telegraf",
"measurement": "cpu",
"retention_days": 7
})
```
## Best practices [#best-practices]
### 1. Start conservative [#1-start-conservative]
Begin with longer intervals and adjust based on actual needs:
```python
# Start with hourly
{"interval": "1h"}
# If too coarse, reduce to 15 minutes
{"interval": "15m"}
```
### 2. Preserve source data initially [#2-preserve-source-data-initially]
Keep raw data while testing aggregations:
```python
# Create continuous query
create_query(...)
# Test aggregations thoroughly
execute_dry_run(...)
execute_for_real(...)
# Only after validation, apply retention to raw data
requests.post("/api/v1/retention", json={
"measurement": "cpu",
"retention_days": 30 # Keep raw for 30 days
})
```
### 3. Use dry run extensively [#3-use-dry-run-extensively]
Always test queries with dry run before full execution:
```python
# Test on small time range first
dry_run(start="2024-01-01", end="2024-01-02")
# Gradually expand
dry_run(start="2024-01-01", end="2024-01-07")
# Finally, full execution
execute(start="2024-01-01", end="2024-12-31")
```
### 4. Include sample counts [#4-include-sample-counts]
Track the number of raw samples in each aggregate:
```sql
SELECT
date_trunc('hour', epoch_us(time)) AS time,
COUNT(*) AS sample_count, -- Essential for data quality
AVG(value) AS avg_value
FROM measurement
GROUP BY date_trunc('hour', epoch_us(time))
```
This helps identify:
* Missing data (low sample counts)
* Data quality issues
* Unexpected patterns
### 5. Monitor execution performance [#5-monitor-execution-performance]
Track continuous query execution times:
```python
result = execute_query(...)
print(f"Execution time: {result['execution_time_ms']}ms")
print(f"Throughput: {result['rows_processed'] / (result['execution_time_ms'] / 1000):.0f} rows/sec")
# Alert if execution takes too long
if result['execution_time_ms'] > 60000: # 1 minute
print("Warning: Slow execution!")
```
### 6. Use appropriate intervals [#6-use-appropriate-intervals]
Match intervals to data characteristics:
**High-Frequency Data** (IoT sensors at 1-second intervals):
* 5-minute aggregates for recent analysis
* Hourly aggregates for medium-term
* Daily aggregates for long-term trends
**Medium-Frequency Data** (Application metrics at 1-minute intervals):
* Hourly aggregates for recent analysis
* Daily aggregates for long-term
**Low-Frequency Data** (Business metrics at hourly intervals):
* Daily aggregates
* Monthly aggregates for multi-year analysis
## Troubleshooting [#troubleshooting]
### No rows written [#no-rows-written]
**Problem**: Execution returns `rows_written: 0`.
**Solutions**:
* Verify source measurement contains data in the specified time range
* Check that the query syntax is correct
* Ensure `GROUP BY` clause matches aggregation columns
* Use dry run to inspect query results
### Query syntax errors [#query-syntax-errors]
**Problem**: Execution fails with SQL error.
**Solutions**:
* Test the query directly using the `/query` endpoint
* Verify column names exist in source measurement
* Check for dialect-specific syntax requirements
* Use `epoch_us()` for timestamp conversion
### Slow execution [#slow-execution]
**Problem**: Continuous query takes longer than expected.
**Solutions**:
* Reduce the time range per execution
* Ensure source measurement is properly compacted
* Consider creating indexes on frequently grouped columns
* Monitor query engine performance
### Duplicate data [#duplicate-data]
**Problem**: Re-running the query creates duplicate aggregates.
**Solutions**:
* Delete destination measurement data before re-execution:
```python
requests.post("/api/v1/delete", json={
"database": "telegraf",
"measurement": "cpu_hourly",
"where": f"time >= '{start_time}' AND time <= '{end_time}'"
})
```
* Or use `UPSERT` semantics if supported (future feature)
## Related topics [#related-topics]
* [Retention Policies](/arc/data-lifecycle/retention-policies/) - Automatically delete old raw data after downsampling
* [Delete Operations](/arc/data-lifecycle/delete-operations/) - Manually remove data ranges
* [Compaction](/arc/advanced/compaction/) - Optimize file structure for better query performance
# Delete Operations (/arc/data-lifecycle/delete-operations)
Arc supports deleting data using a rewrite-based approach that provides precise deletion with zero overhead on write and query operations.
Delete operations must be explicitly enabled in configuration for safety.
## Overview [#overview]
Arc's delete operations provide:
* **Precise Control**: Delete specific rows using WHERE clauses
* **Zero Runtime Overhead**: No performance impact on writes or queries
* **Physical Removal**: Data is permanently removed by rewriting Parquet files
* **Safety Mechanisms**: Multiple safeguards prevent accidental deletion
* **Dry Run Mode**: Test operations before execution
## How it works [#how-it-works]
Arc uses a rewrite-based deletion approach:
### 1. Find affected files [#1-find-affected-files]
Scan the measurement directory to identify Parquet files containing rows that match the WHERE clause.
### 2. Rewrite files [#2-rewrite-files]
For each affected file:
1. Load the file into an Arrow table
2. Filter out matching rows: `SELECT * FROM table WHERE NOT (delete_clause)`
3. Write filtered data to a temporary file
4. Atomically replace the original file using `os.replace()`
### 3. Cleanup [#3-cleanup]
* Files that become empty after filtering are deleted entirely
* Files with remaining data are replaced with their rewritten versions
* All operations use atomic file replacement to ensure data integrity
### Atomic safety [#atomic-safety]
System crashes during deletion result in either the old file or the new file being present, never corruption or partial writes.
## Configuration [#configuration]
Delete operations must be explicitly enabled and configured.
### Configuration file [#configuration-file]
Edit `arc.toml`:
```toml
[delete]
enabled = true
confirmation_threshold = 10000
max_rows_per_delete = 1000000
```
### Environment variables [#environment-variables]
```bash
export DELETE_ENABLED=true
export DELETE_CONFIRMATION_THRESHOLD=10000
export DELETE_MAX_ROWS=1000000
```
### Configuration parameters [#configuration-parameters]
* `enabled` (boolean): Enable/disable delete functionality (default: `false`)
* `confirmation_threshold` (integer): Row count requiring explicit confirmation (default: `10000`)
* `max_rows_per_delete` (integer): Maximum rows allowed per operation (default: `1000000`)
## API endpoints [#api-endpoints]
### Delete data [#delete-data]
Execute a delete operation:
```bash
POST /api/v1/delete
```
**Request Body**:
```json
{
"database": "telegraf",
"measurement": "cpu",
"where": "host = 'server01' AND time < '2024-01-01'",
"dry_run": false,
"confirm": false
}
```
**Parameters**:
* `database` (string, required): Target database name
* `measurement` (string, required): Target measurement name
* `where` (string, required): SQL WHERE clause for deletion
* `dry_run` (boolean, optional): Test without deleting (default: `false`)
* `confirm` (boolean, optional): Confirm large operations (default: `false`)
**Response**:
```json
{
"deleted_count": 15000,
"affected_files": 3,
"rewritten_files": 2,
"deleted_files": 1,
"execution_time_ms": 1250,
"files": [
{
"path": "/data/telegraf/cpu/2023-12-15.parquet",
"action": "rewritten",
"rows_before": 10000,
"rows_after": 5000
},
{
"path": "/data/telegraf/cpu/2023-12-20.parquet",
"action": "deleted",
"rows_before": 5000,
"rows_after": 0
}
]
}
```
### Get configuration [#get-configuration]
Retrieve current delete configuration:
```bash
GET /api/v1/delete/config
```
**Response**:
```json
{
"enabled": true,
"confirmation_threshold": 10000,
"max_rows_per_delete": 1000000
}
```
## Safety mechanisms [#safety-mechanisms]
### 1. WHERE clause required [#1-where-clause-required]
Delete operations **must** include a WHERE clause to prevent accidental full-table deletion.
**Intentional Full Delete**:
```json
{
"where": "1=1" // Explicitly delete all rows
}
```
### 2. Confirmation threshold [#2-confirmation-threshold]
Operations exceeding the configured threshold require explicit confirmation:
```json
{
"where": "time < '2024-01-01'",
"confirm": true // Required if deleted_count > threshold
}
```
**Without Confirmation**:
```json
{
"error": "Operation would delete 15000 rows, exceeding threshold of 10000. Set confirm=true to proceed."
}
```
### 3. Maximum rows limit [#3-maximum-rows-limit]
Hard cap prevents extremely large operations that could exhaust resources:
```json
{
"error": "Operation would delete 2000000 rows, exceeding maximum of 1000000"
}
```
### 4. Atomic file replacement [#4-atomic-file-replacement]
Files are replaced atomically using `os.replace()`, ensuring:
* No partial writes
* No data corruption
* Recovery from crashes (either old or new file exists)
## Usage examples [#usage-examples]
### Example 1: delete old data [#example-1-delete-old-data]
```python
import os
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Delete data older than a specific date
response = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "time < '2024-01-01'"
}
)
print(f"Deleted {response.json()['deleted_count']} rows")
print(f"Execution time: {response.json()['execution_time_ms']}ms")
```
### Example 2: delete specific host data [#example-2-delete-specific-host-data]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Delete data from a specific host
response = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "host = 'server01' OR host = 'server02'"
}
)
```
### Example 3: dry run first [#example-3-dry-run-first]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Always test with dry run before deleting
dry_run = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "host = 'server01'",
"dry_run": True
}
)
print(f"Would delete {dry_run.json()['deleted_count']} rows")
print(f"Affected files: {dry_run.json()['affected_files']}")
# Review the files that would be affected
for file in dry_run.json()['files']:
print(f" {file['path']}: {file['rows_before']} -> {file['rows_after']} rows")
# If satisfied, execute for real
if input("Proceed? (yes/no): ") == "yes":
result = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "host = 'server01'",
"dry_run": False
}
)
print(f"Deleted {result.json()['deleted_count']} rows")
```
### Example 4: delete with confirmation [#example-4-delete-with-confirmation]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Large delete requiring confirmation
response = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "time < '2023-01-01'",
"confirm": True # Explicitly confirm large operation
}
)
```
### Example 5: complex WHERE clause [#example-5-complex-where-clause]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Delete based on multiple conditions
response = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": """
host IN ('server01', 'server02', 'server03')
AND time BETWEEN '2023-01-01' AND '2023-06-30'
AND usage_idle < 10
"""
}
)
```
## Performance characteristics [#performance-characteristics]
Delete operations are computationally expensive but designed for infrequent use:
### Execution times [#execution-times]
**Small Files** (10MB):
* Read + Filter + Write: \~50-100ms per file
**Medium Files** (100MB):
* Read + Filter + Write: \~500ms-1s per file
**Large Files** (1GB):
* Read + Filter + Write: \~2-5s per file
### Performance factors [#performance-factors]
* **File Size**: Larger files take longer to rewrite
* **Selectivity**: Fewer deleted rows = faster (less data movement)
* **Storage I/O**: Disk speed affects read/write performance
* **Concurrent Load**: Other operations may slow deletion
## Best practices [#best-practices]
### 1. Keep disabled by default [#1-keep-disabled-by-default]
Only enable delete operations when needed:
```toml
[delete]
enabled = false # Enable only when necessary
```
### 2. Always use dry run [#2-always-use-dry-run]
Test operations before execution to verify scope:
```python
# Step 1: Dry run
result = requests.post(..., json={"dry_run": True})
print(f"Would delete {result['deleted_count']} rows")
# Step 2: Review affected files
for file in result['files']:
print(f"{file['path']}: {file['action']}")
# Step 3: Execute if satisfied
result = requests.post(..., json={"dry_run": False, "confirm": True})
```
### 3. Consider retention policies [#3-consider-retention-policies]
For time-based deletion, use [retention policies](/arc/data-lifecycle/retention-policies/) instead:
```python
# Instead of manual deletes:
# requests.post("/api/v1/delete", json={"where": "time < '2024-01-01'"})
# Use retention policies:
requests.post("/api/v1/retention", json={
"retention_days": 90,
"buffer_days": 7
})
```
### 4. Monitor execution times [#4-monitor-execution-times]
Track deletion performance for capacity planning:
```python
import time
start = time.time()
result = requests.post("/api/v1/delete", json={...})
elapsed = time.time() - start
print(f"Deleted {result['deleted_count']} rows in {elapsed:.2f}s")
```
### 5. Batch large deletes [#5-batch-large-deletes]
Break large deletions into smaller batches by time range:
```python
from datetime import datetime, timedelta
# Instead of one large delete:
# WHERE time < '2023-01-01'
# Batch by month:
start = datetime(2022, 1, 1)
while start < datetime(2023, 1, 1):
end = start + timedelta(days=30)
requests.post("/api/v1/delete", json={
"where": f"time >= '{start.isoformat()}' AND time < '{end.isoformat()}'"
})
start = end
```
### 6. Understand storage impact [#6-understand-storage-impact]
Deletion rewrites files, which may temporarily increase storage usage:
```python
# Before deletion: 100MB original file
# During deletion: 100MB original + 60MB temp file = 160MB
# After deletion: 60MB rewritten file
```
Ensure sufficient disk space for temporary files during operations.
## Limitations [#limitations]
### Not for frequent operations [#not-for-frequent-operations]
Delete operations rewrite entire Parquet files, making them expensive. They are designed for **infrequent, manual operations** only.
**Use Cases**:
* Removing test data
* Deleting specific hosts/sensors
* One-time cleanup operations
**Not Suitable For**:
* Automated recurring deletions (use retention policies)
* High-frequency data cleanup
* Real-time data removal
### Explicit WHERE required [#explicit-where-required]
Full-table deletion requires explicit `WHERE 1=1`:
```python
# This will fail:
{"where": ""} # Error: WHERE clause required
# Explicit full delete:
{"where": "1=1", "confirm": True}
```
### Maximum row limits [#maximum-row-limits]
Large deletions are subject to `max_rows_per_delete` configuration:
```python
# Will fail if exceeds limit:
{"where": "time < '2020-01-01'"} # May exceed max_rows
# Solution: Batch by time range
{"where": "time >= '2023-01-01' AND time < '2023-02-01'"}
```
### File-level locking [#file-level-locking]
During deletion, affected files are locked. Concurrent writes may be delayed.
## Troubleshooting [#troubleshooting]
### Delete not enabled [#delete-not-enabled]
**Problem**: `DELETE_ENABLED=false` or not configured.
**Solution**:
```toml
[delete]
enabled = true
```
### Confirmation required [#confirmation-required]
**Problem**: Operation exceeds confirmation threshold.
**Solution**: Add `confirm: true`:
```json
{"confirm": true}
```
### Exceeds maximum rows [#exceeds-maximum-rows]
**Problem**: Deletion would affect more rows than `max_rows_per_delete`.
**Solutions**:
1. Batch the operation by time range
2. Increase `max_rows_per_delete` (carefully)
3. Use retention policies for large-scale cleanup
### No rows deleted [#no-rows-deleted]
**Problem**: `deleted_count: 0` but expected deletions.
**Solutions**:
* Verify WHERE clause syntax matches data
* Check that data exists in the specified database/measurement
* Use dry run to inspect affected files
### Slow execution [#slow-execution]
**Problem**: Delete operations take longer than expected.
**Solutions**:
* Check file sizes (large files take longer)
* Monitor disk I/O performance
* Batch operations during low-traffic periods
* Consider using retention policies for time-based cleanup
## Related topics [#related-topics]
* [Retention Policies](/arc/data-lifecycle/retention-policies/) - Automated time-based deletion
* [Continuous Queries](/arc/data-lifecycle/continuous-queries/) - Downsample before deletion
* [Compaction](/arc/advanced/compaction/) - File optimization for better performance
# Data Lifecycle (/arc/data-lifecycle)
These features control what happens to data after it lands. In Arc OSS each one is triggered through the API; scheduled execution is an [Arc Enterprise](/arc-enterprise/operations/automated-scheduling/) feature.
# Retention Policies (/arc/data-lifecycle/retention-policies)
Retention policies allow you to automatically manage data lifecycle by defining how long data should be kept.
Automatic execution is reserved for [Arc Enterprise](/arc-enterprise/operations/automated-scheduling/). In Arc OSS, retention policies must be manually triggered via the API.
## Overview [#overview]
Retention policies in Arc help you:
* Define data retention periods at database or measurement level
* Automatically clean up old data through manual execution
* Reduce storage costs by removing unnecessary historical data
* Maintain compliance with data retention requirements
* Test deletion operations safely with dry-run mode
## How it works [#how-it-works]
Arc implements retention through physical file deletion:
1. **Scanning**: Examines Parquet files in measurement directories
2. **Metadata Analysis**: Reads file metadata to find maximum timestamps
3. **Identification**: Locates files where all rows are older than the cutoff date
4. **Deletion**: Physically removes entire files from disk
**Cutoff Calculation**: `cutoff_date = today - retention_days - buffer_days`
## API endpoints [#api-endpoints]
### Create policy [#create-policy]
Create a new retention policy:
```bash
POST /api/v1/retention
```
**Request Body**:
```json
{
"name": "delete_old_metrics",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": true
}
```
**Parameters**:
* `name` (string, required): Unique policy identifier
* `database` (string, required): Target database name
* `measurement` (string, optional): Target measurement (null for database-wide)
* `retention_days` (integer, required): Number of days to retain data
* `buffer_days` (integer, required): Safety margin in days
* `is_active` (boolean, required): Enable/disable the policy
### List policies [#list-policies]
Retrieve all retention policies:
```bash
GET /api/v1/retention
```
**Response**:
```json
[
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"name": "delete_old_metrics",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": true,
"created_at": "2024-01-15T10:30:00Z",
"last_executed_at": "2024-01-20T02:00:00Z",
"last_deleted_count": 1500
}
]
```
### Get single policy [#get-single-policy]
Retrieve a specific retention policy:
```bash
GET /api/v1/retention/{policy_id}
```
### Update policy [#update-policy]
Update an existing retention policy:
```bash
PUT /api/v1/retention/{policy_id}
```
**Request Body**: Same as create policy
### Delete policy [#delete-policy]
Remove a retention policy:
```bash
DELETE /api/v1/retention/{policy_id}
```
### Execute policy [#execute-policy]
Manually trigger a retention policy:
```bash
POST /api/v1/retention/{policy_id}/execute
```
**Request Body**:
```json
{
"dry_run": false,
"confirm": true
}
```
**Dry Run Example**:
```json
{
"dry_run": true,
"confirm": false
}
```
**Response**:
```json
{
"policy_id": "550e8400-e29b-41d4-a716-446655440000",
"cutoff_date": "2023-10-22T00:00:00Z",
"files_to_delete": [
"/data/telegraf/cpu/2023-10-15.parquet",
"/data/telegraf/cpu/2023-10-20.parquet"
],
"total_files": 2,
"dry_run": true
}
```
### View execution history [#view-execution-history]
View past executions of a retention policy:
```bash
GET /api/v1/retention/{policy_id}/executions?limit=50
```
**Response**:
```json
[
{
"execution_id": "abc123",
"executed_at": "2024-01-20T02:00:00Z",
"deleted_count": 1500,
"execution_time_ms": 2500,
"status": "success"
}
]
```
## Configuration parameters [#configuration-parameters]
### Retention days [#retention-days]
The number of days to keep data before it becomes eligible for deletion. Choose based on:
* Business requirements
* Compliance regulations
* Storage capacity
* Query patterns
**Example**: `retention_days: 90` keeps data for 90 days.
### Buffer days [#buffer-days]
A safety margin added to the retention period to prevent accidental deletion of recent data.
**Recommended Values**:
* Development: 7 days
* Production: 14-30 days
**Example**: With `retention_days: 90` and `buffer_days: 7`, data older than 97 days will be deleted.
### Database vs measurement level [#database-vs-measurement-level]
**Database-wide policy**:
```json
{
"database": "telegraf",
"measurement": null,
"retention_days": 365
}
```
**Measurement-specific policy**:
```json
{
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90
}
```
Use measurement-specific policies for granular control over different data types.
## Usage examples [#usage-examples]
### Example 1: clean old metrics [#example-1-clean-old-metrics]
```python
import os
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create a retention policy for old CPU metrics
response = requests.post(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "cpu_cleanup",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": True
}
)
policy_id = response.json()["id"]
# Test with dry run first
dry_run = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"dry_run": True, "confirm": False}
)
print(f"Would delete {dry_run.json()['total_files']} files")
# Execute for real
if input("Proceed? (yes/no): ") == "yes":
result = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"dry_run": False, "confirm": True}
)
print(f"Deleted {result.json()['total_files']} files")
```
### Example 2: database-wide retention [#example-2-database-wide-retention]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Apply retention to all measurements in a database
response = requests.post(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "database_cleanup",
"database": "telegraf",
"measurement": None, # Apply to all measurements
"retention_days": 180,
"buffer_days": 14,
"is_active": True
}
)
```
### Example 3: list and monitor policies [#example-3-list-and-monitor-policies]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# List all policies
policies = requests.get(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"}
)
for policy in policies.json():
print(f"Policy: {policy['name']}")
print(f" Last executed: {policy['last_executed_at']}")
print(f" Last deleted: {policy['last_deleted_count']} rows")
# Get execution history
history = requests.get(
f"http://localhost:8000/api/v1/retention/{policy['id']}/executions?limit=10",
headers={"Authorization": f"Bearer {ARC_TOKEN}"}
)
print(f" Recent executions: {len(history.json())}")
```
## Best practices [#best-practices]
### 1. Always test first [#1-always-test-first]
Use dry-run mode before executing retention policies:
```python
# Always start with dry run
result = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
json={"dry_run": True, "confirm": False}
)
# Review what will be deleted
print(f"Files to delete: {result.json()['files_to_delete']}")
```
### 2. Use buffer days [#2-use-buffer-days]
Implement a safety buffer to prevent accidental deletion:
```json
{
"retention_days": 90,
"buffer_days": 14 // 14-day safety margin
}
```
### 3. Start conservative [#3-start-conservative]
Begin with longer retention periods and gradually shorten:
```json
// Start here
{"retention_days": 365, "buffer_days": 30}
// After monitoring, reduce if needed
{"retention_days": 180, "buffer_days": 14}
```
### 4. Test in non-production [#4-test-in-non-production]
Create and test policies in a development environment first:
```bash
# Development environment
export ARC_ENV=dev
# Test policies thoroughly before production
```
### 5. Monitor execution history [#5-monitor-execution-history]
Regularly check the `last_deleted_count` field:
```python
# Check if deletion counts are as expected
policy = requests.get(f"/api/v1/retention/{policy_id}").json()
if policy['last_deleted_count'] > 10000:
print("Warning: Large deletion detected!")
```
### 6. Use measurement-specific policies [#6-use-measurement-specific-policies]
Create granular policies for different data types:
```python
# High-frequency metrics - shorter retention
{"measurement": "cpu", "retention_days": 30}
# Business metrics - longer retention
{"measurement": "revenue", "retention_days": 730}
```
## Important limitations [#important-limitations]
### Local storage only [#local-storage-only]
Currently, retention policies only work with local filesystem storage. Cloud storage backends (S3, MinIO, GCS) are not yet implemented.
### File-level granularity [#file-level-granularity]
Retention operates at the file level, not row level. A file is only deleted if **all** rows are older than the cutoff date.
For optimal retention policy effectiveness, ensure your data is properly compacted. Files with mixed timestamps may not be eligible for deletion.
### No rollback [#no-rollback]
Deleted data cannot be recovered. Always:
1. Use dry-run mode first
2. Maintain backups of critical data
3. Test in non-production environments
### Sequential processing [#sequential-processing]
Retention policies process measurements sequentially. Large databases may take time to process.
### Works best with compacted files [#works-best-with-compacted-files]
Retention policies are most effective when files contain data from similar time periods. Enable [automatic compaction](/arc/advanced/compaction/) for better results.
## Troubleshooting [#troubleshooting]
### No files being deleted [#no-files-being-deleted]
**Problem**: Dry run shows 0 files to delete.
**Solutions**:
* Check that data actually exists older than `retention_days + buffer_days`
* Verify the policy targets the correct database and measurement
* Ensure files are fully older than the cutoff (file-level granularity)
### Policy not executing [#policy-not-executing]
**Problem**: Manual execution returns an error.
**Solutions**:
* Verify the policy `is_active` is set to `true`
* Check that `confirm: true` is set for actual execution
* Ensure you have write permissions on the data directory
### Unexpected file count [#unexpected-file-count]
**Problem**: More/fewer files than expected are being deleted.
**Solutions**:
* Remember: Only files where **all rows** are older than cutoff are deleted
* Check file timestamps using `ls -l` on the measurement directory
* Review recent compaction activity that may have merged files
## Related topics [#related-topics]
* [Delete Operations](/arc/data-lifecycle/delete-operations/) - Manual delete operations for specific data
* [Continuous Queries](/arc/data-lifecycle/continuous-queries/) - Downsample data before deletion
* [Compaction](/arc/advanced/compaction/) - Optimize file structure for better retention
# Decimal Precision (/arc/guides/decimal-precision)
Native Decimal128 type support requires Arc v26.04.1 or later.
Arc supports native Decimal128 columns for precision-sensitive use cases such as financial data, scientific measurements, and cryptocurrency trading. Declare decimal columns via per-measurement configuration, and Arc stores them as native Parquet DECIMAL type — preserving exact precision instead of coercing to float64.
## Why Decimal128? [#why-decimal128]
IEEE 754 float64 has \~15-17 significant digits of precision. For financial data, this causes silent rounding:
```text
float64(0.1 + 0.2) = 0.30000000000000004 // not 0.3
float64(9007199254740993) = 9007199254740992 // off by 1
```
Decimal128 provides up to 38 significant digits with exact precision — no rounding, no surprises.
## Configuration [#configuration]
Declare which columns should be stored as DECIMAL using the `[ingest]` configuration:
```toml
[ingest]
# Per-measurement decimal columns: "measurement:col=precision,scale;col2=p,s"
decimal_columns = [
"trades:price=18,8;amount=18,8",
"balances:balance=38,18"
]
# Default decimal columns for measurements not listed above (optional)
default_decimal_columns = "value=18,6"
```
**Environment variables:**
```ini
ARC_INGEST_DECIMAL_COLUMNS="trades:price=18,8;amount=18,8 balances:balance=38,18"
ARC_INGEST_DEFAULT_DECIMAL_COLUMNS="value=18,6"
```
### Format [#format]
Each entry follows the pattern `measurement:column=precision,scale;column2=precision,scale`:
| Component | Description | Range |
| ------------- | ------------------------------- | ---------------- |
| `measurement` | Target measurement name | Non-empty string |
| `column` | Column name to store as DECIMAL | Non-empty string |
| `precision` | Total significant digits | 1-38 |
| `scale` | Digits after decimal point | 0-precision |
**Common configurations:**
| Use Case | Config | Example Value |
| ------------------------ | --------------- | ---------------------- |
| Cryptocurrency prices | `price=18,8` | `99999.12345678` |
| USD amounts | `amount=18,2` | `1234567890123456.78` |
| Scientific measurements | `reading=38,18` | `3.141592653589793238` |
| Integer counters (exact) | `count=38,0` | `99999999999999999999` |
## Ingestion [#ingestion]
Decimal conversion happens automatically at ingestion time for configured columns. All ingestion paths are supported:
### MessagePack (recommended) [#messagepack-recommended]
**Float values** — converted via `decimal128.FromFloat64`:
```python
import msgpack
import urllib.request
payload = {
"m": "trades",
"t": 1711152000000000,
"fields": {
"price": 123.45678901, # float64 → DECIMAL(18,8)
"amount": 999.12345678, # float64 → DECIMAL(18,8)
"volume": 42, # not configured → stays BIGINT
},
"tags": {"symbol": "BTC-USD"}
}
data = msgpack.packb(payload)
req = urllib.request.Request(
"http://localhost:8000/api/v1/write/msgpack",
data=data,
headers={
"Content-Type": "application/msgpack",
"x-arc-database": "mydb",
}
)
urllib.request.urlopen(req)
```
**String values** — for highest precision, send values as strings over MessagePack. String-to-decimal conversion is exact with no float64 intermediate:
```python
payload = {
"m": "trades",
"t": 1711152000000000,
"fields": {
"price": "99999.12345678", # string → exact DECIMAL(18,8)
"amount": "0.00000001", # smallest representable at scale 8
},
"tags": {"symbol": "ETH-USD"}
}
```
For values requiring more than 15-17 significant digits, always send as strings over MessagePack. Float64 values lose precision beyond ~15 digits before they even reach Arc.
### Line Protocol [#line-protocol]
Float values in Line Protocol are converted to the configured decimal type:
```bash
curl -X POST "http://localhost:8000/write?db=mydb" \
-d 'trades,symbol=BTC-USD price=123.45678901,amount=999.12345678,volume=42i'
```
Line Protocol transmits all float values as text, so precision is preserved through the wire format. However, Arc's LP parser converts to float64 internally before decimal conversion — for maximum precision beyond 15 digits, use MessagePack with string values.
## Querying [#querying]
Arc reads the Parquet DECIMAL type natively — no query changes needed:
```sql
SELECT price, amount, typeof(price) as price_type
FROM trades
LIMIT 5
```
Response:
```json
{
"columns": ["price", "amount", "price_type"],
"data": [
["123.45678901", "999.12345678", "DECIMAL(18,8)"],
["99999.12345678", "0.00000001", "DECIMAL(18,8)"]
]
}
```
All SQL operations work with decimal columns — aggregation, filtering, ordering, window functions:
```sql
-- Aggregation preserves decimal precision
SELECT symbol,
AVG(price) as avg_price,
SUM(amount) as total_amount,
COUNT(*) as trade_count
FROM trades
GROUP BY symbol
-- Filtering works as expected
SELECT * FROM trades WHERE price > 50000.00
-- Arithmetic on decimal columns
SELECT price * amount as notional_value FROM trades
```
## How it works [#how-it-works]
1. **Ingestion**: Float64, int64, or string values arriving for declared decimal columns are converted to Arrow Decimal128 at buffer time
2. **Storage**: Stored as Parquet DECIMAL logical type (16 bytes per value, exact precision)
3. **Metadata**: Decimal column specs are stored as Parquet metadata (`arc:decimals`) for self-describing files
4. **Querying**: Arc reads Parquet DECIMAL natively
5. **Compaction**: DECIMAL types are preserved automatically during compaction
## Performance [#performance]
* **Zero overhead when not configured** — one empty map lookup per column during ingestion
* **Decimal conversion cost**: \~100ns per value. A flush batch of 50K rows with 2 decimal columns adds \~10ms (flush runs in background)
* **Storage**: 16 bytes per value (vs 8 bytes for float64) — 2x for decimal columns only
* **Query performance**: Identical to float64 — DECIMAL is handled natively
## Best practices [#best-practices]
1. **Use string values for maximum precision** — Send values as strings over MessagePack when precision beyond 15 significant digits is required.
2. **Choose precision and scale carefully** — Use the smallest precision/scale that fits your data. `price=18,8` handles up to 10 billion with 8 decimal places. Over-specifying (e.g., `38,18`) wastes no storage but may affect sort performance.
3. **Use default decimal columns sparingly** — Per-measurement config is preferred. Defaults apply to all measurements not explicitly listed, which may cause unexpected type conversions.
4. **Non-decimal columns are unaffected** — Columns not listed in the config continue to use automatic type detection (float64, int64, string, bool).
5. **Backwards compatible** — Files written before enabling decimal config have no decimal metadata and are read normally. You can enable decimal config on existing deployments without any migration.
## Troubleshooting [#troubleshooting]
### Decimal config not taking effect [#decimal-config-not-taking-effect]
Check the Arc startup logs for:
```bash
WRN Invalid decimal columns config, decimal support disabled error="..."
```
This means the configuration format is invalid. Verify the format: `"measurement:col=precision,scale"`.
### Values still showing as float64 [#values-still-showing-as-float64]
* Verify the column name in config matches exactly (case-sensitive)
* Check that the measurement name matches
* Use `typeof(column)` in SQL to verify: `SELECT typeof(price) FROM trades LIMIT 1`
### Scientific notation in query results [#scientific-notation-in-query-results]
Arc may serialize very small decimals using scientific notation (e.g., `1e-08` instead of `0.00000001`). The underlying precision is preserved — this is a display format choice. Use `CAST(column AS VARCHAR)` for explicit string formatting.
# Guides (/arc/guides)
Task-oriented guides for working with data once Arc is running.
# SQL Querying Guide (/arc/guides/querying)
Arc runs a full analytical SQL engine over data stored as Parquet files, so window functions, CTEs, and joins are all available.
## SQL syntax [#sql-syntax]
Queries use the format `database.measurement` as the table name:
```sql
SELECT * FROM mydb.cpu LIMIT 10
```
If your database is named `default`, you can omit it:
```sql
SELECT * FROM default.cpu LIMIT 10
```
## Query endpoints [#query-endpoints]
| Endpoint | Response Format | Best For |
| -------------------------------- | ---------------- | ------------------------------------ |
| `POST /api/v1/query` | JSON | Small results, debugging, dashboards |
| `POST /api/v1/query/arrow` | Apache Arrow IPC | Large result sets |
| `GET /api/v1/query/:measurement` | JSON | Quick measurement queries |
### JSON query [#json-query]
```bash
curl -X POST "http://localhost:8000/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL '\''1 hour'\'' LIMIT 100"}'
```
### Arrow query [#arrow-query]
For large result sets, Arrow IPC provides \~2x throughput vs JSON:
```bash
curl -X POST "http://localhost:8000/api/v1/query/arrow" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM default.cpu LIMIT 1000000"}' \
-o results.arrow
```
## Time filtering [#time-filtering]
Arc stores timestamps in the `time` column. Use standard SQL intervals for time-range queries:
```sql
-- Last hour
SELECT * FROM default.cpu
WHERE time > NOW() - INTERVAL '1 hour';
-- Last 7 days
SELECT * FROM default.cpu
WHERE time > NOW() - INTERVAL '7 days';
-- Specific date range
SELECT * FROM default.cpu
WHERE time BETWEEN '2026-01-01' AND '2026-01-31';
```
Time-range filters using the `time` column automatically trigger partition pruning, skipping Parquet files outside the range. Always include a time filter for best performance.
## Time-series aggregation [#time-series-aggregation]
### time\_bucket [#time_bucket]
Group data into fixed-size time intervals:
```sql
-- Hourly averages for the last 7 days
SELECT
time_bucket('1 hour', time) AS bucket,
AVG(cpu_usage) AS avg_cpu,
MAX(cpu_usage) AS max_cpu,
COUNT(*) AS samples
FROM default.cpu
WHERE time > NOW() - INTERVAL '7 days'
GROUP BY bucket
ORDER BY bucket;
```
### date\_trunc [#date_trunc]
Truncate timestamps to calendar boundaries:
```sql
-- Daily summary for the last 30 days
SELECT
date_trunc('day', time) AS day,
host,
AVG(cpu_usage) AS avg_cpu,
AVG(mem_usage) AS avg_mem
FROM default.cpu
WHERE time > NOW() - INTERVAL '30 days'
GROUP BY day, host
ORDER BY day DESC, host;
```
## Window functions [#window-functions]
Compute rolling metrics and detect anomalies:
```sql
-- 10-minute moving average with anomaly detection
SELECT
time,
host,
cpu_usage,
AVG(cpu_usage) OVER (
PARTITION BY host
ORDER BY time
ROWS BETWEEN 10 PRECEDING AND CURRENT ROW
) AS moving_avg,
cpu_usage - AVG(cpu_usage) OVER (
PARTITION BY host
ORDER BY time
ROWS BETWEEN 60 PRECEDING AND CURRENT ROW
) AS deviation
FROM default.cpu
WHERE time > NOW() - INTERVAL '1 hour';
```
## Common table expressions (CTEs) [#common-table-expressions-ctes]
Break complex queries into readable steps:
```sql
-- Find hosts with anomalous CPU spikes
WITH hourly_stats AS (
SELECT
host,
time_bucket('1 hour', time) AS bucket,
AVG(cpu_usage) AS avg_cpu,
STDDEV(cpu_usage) AS std_cpu
FROM default.cpu
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY host, bucket
),
anomalies AS (
SELECT *
FROM hourly_stats
WHERE avg_cpu > 80 OR std_cpu > 20
)
SELECT host, bucket, avg_cpu, std_cpu
FROM anomalies
ORDER BY avg_cpu DESC;
```
## Cross-database queries [#cross-database-queries]
Join data across databases and measurements:
```sql
-- Join CPU metrics with deployment events
SELECT
c.time,
c.host,
c.cpu_usage,
d.version
FROM production.cpu c
JOIN production.deployments d
ON c.host = d.host
AND c.time BETWEEN d.time AND d.time + INTERVAL '1 hour'
WHERE c.time > NOW() - INTERVAL '24 hours';
```
## Useful SQL functions [#useful-sql-functions]
These are the functions most useful for analytical and time-series queries:
| Function | Description | Example |
| ----------------------------- | ------------------------------- | ------------------------------------------------------- |
| `NOW()` | Current timestamp | `WHERE time > NOW() - INTERVAL '1h'` |
| `time_bucket(interval, time)` | Fixed-size time buckets | `time_bucket('5 minutes', time)` |
| `date_trunc(part, time)` | Calendar truncation | `date_trunc('day', time)` |
| `epoch(time)` | Timestamp to epoch seconds | `epoch(time)` |
| `PERCENTILE_CONT(p)` | Percentile (continuous) | `PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY latency)` |
| `APPROX_QUANTILE(col, p)` | Approximate percentile (faster) | `APPROX_QUANTILE(latency, 0.99)` |
| `STDDEV(col)` | Standard deviation | `STDDEV(cpu_usage)` |
| `LAG(col) OVER (...)` | Previous row value | `LAG(value) OVER (ORDER BY time)` |
| `LEAD(col) OVER (...)` | Next row value | `LEAD(value) OVER (ORDER BY time)` |
## Aggregate functions [#aggregate-functions]
Arc supports the **full analytical SQL aggregate set** — there is no allowlist. Beyond the standard `COUNT`/`SUM`/`AVG`/`MIN`/`MAX`, the following are commonly useful for analytics:
| Function | Description | Example |
| ------------------------------------------- | --------------------------------------- | -------------------------------- |
| `COUNT(DISTINCT col)` | Distinct count | `COUNT(DISTINCT host)` |
| `APPROX_COUNT_DISTINCT(col)` | Fast approximate distinct count | `APPROX_COUNT_DISTINCT(user_id)` |
| `MEDIAN(col)` | Median value | `MEDIAN(latency)` |
| `MODE(col)` | Most frequent value | `MODE(status_code)` |
| `QUANTILE_CONT(col, p)` | Continuous quantile | `QUANTILE_CONT(latency, 0.95)` |
| `STDDEV(col)` / `VARIANCE(col)` | Standard deviation / variance | `STDDEV(cpu_usage)` |
| `ARG_MAX(arg, val)` / `ARG_MIN(arg, val)` | `arg` at the row where `val` is max/min | `ARG_MAX(host, cpu_usage)` |
| `FIRST(col)` / `LAST(col)` | First / last value in group | `LAST(value)` |
| `STRING_AGG(col, sep)` | Concatenate values | `STRING_AGG(host, ', ')` |
| `LIST(col)` / `ARRAY_AGG(col)` | Collect values into a list | `LIST(value)` |
| `HISTOGRAM(col)` | Value-count map | `HISTOGRAM(status_code)` |
| `CORR(y, x)` | Correlation coefficient | `CORR(cpu_usage, mem_usage)` |
| `REGR_SLOPE(y, x)` / `REGR_INTERCEPT(y, x)` | Linear regression slope / intercept | `REGR_SLOPE(value, epoch(time))` |
| `ENTROPY(col)` | Shannon entropy | `ENTROPY(status_code)` |
This is a selection, not the full set. There is no allowlist — if a standard analytical aggregate exists, Arc accepts it. An unrecognised function returns a query error naming it, so trying one is safe.
## Performance tips [#performance-tips]
1. **Always filter by time** -- Partition pruning skips entire Parquet files outside the range, often by a wide margin.
2. **Use Arrow for large results** -- Arrow IPC provides \~2x throughput vs JSON for result sets over 100K rows.
3. **Limit result sets** -- Add `LIMIT` when exploring data. Scanning millions of rows without a limit is expensive.
4. **Use aggregations server-side** -- Compute `AVG`, `COUNT`, `SUM` in SQL rather than fetching raw rows and aggregating client-side.
5. **Prefer `APPROX_QUANTILE` over `PERCENTILE_CONT`** -- For large datasets, approximate percentiles are substantially faster.
6. **Use `time_bucket` over `date_trunc`** -- `time_bucket` supports arbitrary intervals (5 min, 15 min, 4 hours) while `date_trunc` is limited to calendar boundaries.
## Next steps [#next-steps]
* **[API Reference](/arc/api-reference/overview/)** -- Full endpoint documentation
* **[Python SDK Querying](/arc/sdks/python/querying/)** -- Query with pandas, polars, and PyArrow
* **[Retention Policies](/arc/data-lifecycle/retention-policies/)** -- Automatic data expiration
* **[Continuous Queries](/arc/data-lifecycle/continuous-queries/)** -- Real-time aggregations and downsampling
# AWS Marketplace (/arc/installation/aws-marketplace)
Deploy Arc on AWS with a single click using our pre-configured AMI from AWS Marketplace.
## Overview [#overview]
Arc is available on AWS Marketplace as a ready-to-run AMI. No license keys, no sales calls—just subscribe and launch.
**What you get:**
* Pre-installed Arc with systemd service
* Ubuntu-based AMI
* All features enabled
* Free to use (AGPL-3.0)
* Optional enterprise support ($500/month)
## Prerequisites [#prerequisites]
* AWS Account
* EC2 key pair for SSH access
* VPC with appropriate subnets
* (Production) ACM certificate for HTTPS
## Quick start (dev/testing only) [#quick-start-devtesting-only]
This pattern exposes your database directly to the internet. Use only for testing and evaluation.
1. **Subscribe to Arc on AWS Marketplace**
[Open Arc on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-mkhhh2vk4fyss)
2. **Launch an EC2 instance** with a public IP
3. **Configure security group** to allow port 8000 (restrict to your IP)
4. **Get your admin token:**
```bash
# SSH into your instance (Ubuntu-based AMI)
ssh -i your-key.pem ubuntu@your-instance-ip
# View Arc logs to find the admin token
sudo journalctl -u arc | grep "Admin token"
# You'll see something like:
# Admin token: ark_abc123...xyz
```
5. **Test the connection:**
```bash
export ARC_URL="http://your-instance-ip:8000"
export ARC_TOKEN="your-token-here"
curl $ARC_URL/health
```
## Production deployment (recommended) [#production-deployment-recommended]
For production workloads, deploy Arc behind an Application Load Balancer in a private subnet.
### Architecture [#architecture]
```text
┌──────────────────────────────────────────────────────────┐
│ VPC (10.0.0.0/16) │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Public Subnet (10.0.0.0/24) │ │
│ │ │ │
│ │ ┌──────────────────────────────────┐ │ │
│ │ │ Application Load Balancer (ALB) │ │ │
│ │ │ - HTTPS (443) with SSL cert │ │ │
│ │ │ - Terminates TLS │ │ │
│ │ │ - Health checks │ │ │
│ │ └────────────┬─────────────────────┘ │ │
│ │ │ │ │
│ └───────────────┼────────────────────────────────────┘ │
│ │ HTTP (8000) │
│ ┌───────────────┼────────────────────────────────────┐ │
│ │ Private Subnet (10.0.1.0/24) │ │ │
│ │ ▼ │ │
│ │ ┌──────────────────────────────────────┐ │ │
│ │ │ Arc Instance │ │ │
│ │ │ - No public IP │ │ │
│ │ │ - Port 8000 from ALB only │ │ │
│ │ │ - EBS storage for Parquet files │ │ │
│ │ └──────────────────────────────────────┘ │ │
│ │ │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ Optional: │
│ • Telegraf in same VPC │
│ • Grafana in same VPC │
│ • NAT Gateway for outbound │
└──────────────────────────────────────────────────────────┘
```
**Benefits:**
* SSL termination at ALB (free certificate from ACM)
* Arc in private subnet (no internet exposure)
* Security groups restrict traffic
* Health checks and monitoring
### Step 1: subscribe to Arc [#step-1-subscribe-to-arc]
1. Go to [Arc on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-mkhhh2vk4fyss)
2. Click **View purchase options**
3. Click **Subscribe**
4. Wait for subscription to activate, then click **Continue to Configuration**
### Step 2: Launch configuration [#step-2-launch-configuration]
1. Select your **Region**
2. Choose an **Instance type**:
* Testing: `t3.large` (2 vCPU, 8 GB RAM)
* Production: `m8a.xlarge` or larger
3. Click **Continue to Launch**
### Step 3: configure network settings [#step-3-configure-network-settings]
1. **VPC:** Select your existing VPC or create a new one
2. **Subnet:** Choose a private subnet (no internet gateway route)
3. **Security Group:** Create a new one that allows:
* Inbound: Port 8000 from your ALB security group only
* Outbound: As needed for your environment
4. Click **Launch**
### Step 4: create a target group [#step-4-create-a-target-group]
Before creating the load balancer, create a target group.
1. Go to **EC2 Console** → **Target Groups** → **Create target group**
2. Configure:
* **Target type:** Instances
* **Target group name:** `arc-target-group`
* **Protocol:** HTTP
* **Port:** 8000
* **VPC:** Same VPC as Arc instance
3. **Health checks:**
* **Path:** `/health`
* **Healthy threshold:** 2
* **Unhealthy threshold:** 2
* **Timeout:** 5 seconds
* **Interval:** 30 seconds
4. Click **Next**, select your Arc instance, click **Include as pending below**
5. Click **Create target group**
### Step 5: create the application load balancer [#step-5-create-the-application-load-balancer]
1. Go to **EC2 Console** → **Load Balancers** → **Create load balancer**
2. Select **Application Load Balancer**
3. Configure:
* **Name:** `arc-alb`
* **Scheme:** Internet-facing
* **IP address type:** IPv4
4. **Network mapping:**
* Select your VPC
* Select at least two public subnets (one per AZ)
5. **Security groups:**
* Allow inbound HTTPS (443) from your allowed IP ranges
6. **Listeners:**
* Protocol: HTTPS, Port: 443
* Default action: Forward to `arc-target-group`
7. **Secure listener settings:**
* Security policy: `ELBSecurityPolicy-TLS13-1-2-2021-06`
* Certificate: Select from ACM (free) or import your own
8. Click **Create load balancer**
### Step 6: point your domain to the load balancer [#step-6-point-your-domain-to-the-load-balancer]
1. Go to **Load Balancers** and copy the **DNS name**
**Using Route 53:**
1. Go to **Route 53** → **Hosted zones** → your domain
2. Click **Create record**
3. Configure:
* **Record name:** `arc` (for `arc.yourdomain.com`)
* **Record type:** A
* **Alias:** Yes
* **Route traffic to:** Application Load Balancer
* **Region:** Your ALB's region
* **Load balancer:** Select your ALB
4. Click **Create records**
**Using External DNS:**
Create a CNAME record pointing to the ALB DNS name:
* **Name:** `arc`
* **Type:** CNAME
* **Value:** `arc-alb-xxxxx.us-east-1.elb.amazonaws.com`
CNAME records don't work for apex domains. Use a subdomain like `arc.yourdomain.com`.
### Step 7: verify security groups [#step-7-verify-security-groups]
Ensure you have two properly configured security groups:
**ALB Security Group:**
* Inbound: HTTPS (443) from `0.0.0.0/0` (or your allowed IP ranges)
* Outbound: HTTP (8000) to Arc Security Group
**Arc Security Group:**
* Inbound: HTTP (8000) from ALB Security Group only
* Outbound: All traffic (or restrict as needed)
### Step 8: verify target health [#step-8-verify-target-health]
1. Go to **Target Groups** → `arc-target-group`
2. Click **Targets** tab
3. Wait for status to change from "initial" to "healthy"
If unhealthy, check:
* Security group allows port 8000 from ALB
* Arc is running: `sudo systemctl status arc`
* Health check path is correct: `/health`
### Step 9: get your admin token [#step-9-get-your-admin-token]
SSH into your Arc instance through a bastion host or Session Manager:
```bash
# Using a bastion host (Ubuntu-based AMI)
ssh -i your-key.pem -J ubuntu@bastion-ip ubuntu@arc-private-ip
# Or use AWS Systems Manager Session Manager
aws ssm start-session --target i-your-instance-id
# Get the admin token
sudo journalctl -u arc | grep "Admin token"
# You'll see:
# Admin token: ark_abc123...xyz
```
### Step 10: verify deployment [#step-10-verify-deployment]
Test the health endpoint through your ALB:
```bash
curl https://arc.yourdomain.com/health
# Expected: {"status":"healthy"}
```
Test ingestion using MessagePack columnar format:
```bash
export ARC_URL="https://arc.yourdomain.com"
export ARC_TOKEN="your-token-here"
# Write test data
echo '{"m":"cpu","columns":{"time":[1734372000000],"host":["server01"],"usage":[95.0]}}' | \
python3 -c "import sys,msgpack,json; sys.stdout.buffer.write(msgpack.packb(json.load(sys.stdin)))" | \
curl -X POST "$ARC_URL/api/v1/write/msgpack" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/msgpack" \
-H "x-arc-database: default" \
--data-binary @-
# Query it back
curl -X POST "$ARC_URL/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql":"SELECT * FROM default.cpu","format":"json"}'
```
## Instance types [#instance-types]
| Use Case | Instance Type | vCPU | Memory | Notes |
| ----------------- | ------------- | ---- | ------ | ------------------------- |
| Testing | t3.large | 2 | 8 GB | Burstable, cost-effective |
| Small Production | m8a.xlarge | 4 | 16 GB | General purpose |
| Medium Production | m8a.2xlarge | 8 | 32 GB | Recommended |
| High Throughput | c7i.4xlarge | 16 | 32 GB | Compute optimized |
## Storage [#storage]
Arc stores data in Parquet files on the EBS volume attached to the instance.
**Recommendations:**
* Use `gp3` volumes for best price/performance
* Size based on your data retention needs
* Enable EBS encryption for data at rest
```bash
# Check disk usage
df -h /app/data
```
## Pricing [#pricing]
| Component | Cost | Notes |
| ------------------ | ---------------- | ------------------------------------------------------ |
| Arc Software | Free | AGPL-3.0 license |
| EC2 Instance | Variable | See [EC2 Pricing](https://aws.amazon.com/ec2/pricing/) |
| EBS Storage | \~$0.08/GB/month | gp3 pricing |
| ALB | \~$20/month | Plus data transfer |
| SSL Certificate | Free | AWS Certificate Manager |
| Enterprise Support | $500/month | Optional |
## Service management [#service-management]
Arc runs as a systemd service on the AMI.
```bash
# Check status
sudo systemctl status arc
# View logs
sudo journalctl -u arc -f
# Restart service
sudo systemctl restart arc
# Stop service
sudo systemctl stop arc
```
## Troubleshooting [#troubleshooting]
### Target shows unhealthy [#target-shows-unhealthy]
1. Check Arc is running:
```bash
sudo systemctl status arc
```
2. Check Arc is listening on port 8000:
```bash
sudo ss -tlnp | grep 8000
```
3. Test health endpoint locally:
```bash
curl http://localhost:8000/health
```
4. Check security group allows traffic from ALB
### 504 Gateway Timeout [#504-gateway-timeout]
The ALB can't reach the Arc instance. Check:
* Security group allows port 8000 from ALB
* Arc instance is in the correct subnet
* Target group has the correct port (8000)
### Can't find admin token [#cant-find-admin-token]
```bash
# Check all Arc logs
sudo journalctl -u arc | head -200
# Or search specifically
sudo journalctl -u arc | grep -i "admin\|token"
```
## Next steps [#next-steps]
* [Configure Telegraf integration](/arc/integrations/telegraf/)
* [Set up Grafana dashboards](/arc/integrations/grafana/)
* [Configure retention policies](/arc/data-lifecycle/retention-policies/)
# Docker Installation (/arc/installation/docker)
Install and run Arc using Docker for quick setup and isolated environments.
## Prerequisites [#prerequisites]
* Docker 20.10 or higher
* 4GB RAM minimum, 8GB+ recommended
## Quick start [#quick-start]
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
Verify it's running:
```bash
curl http://localhost:8000/health
```
## Get your admin token [#get-your-admin-token]
When Arc starts for the first time, it generates an admin token.
Copy this token immediately - you won't see it again!
```bash
docker logs arc 2>&1 | grep -i "admin"
```
You should see:
```bash
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Initial admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save it:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
## Storage backends [#storage-backends]
**Local Filesystem** - Default, data stored in Docker volume.
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-e ARC_STORAGE_BACKEND=local \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
**Data locations:**
| Path | Description |
| ----------------------- | ------------- |
| `/app/data/arc/` | Parquet files |
| `/app/data/arc_auth.db` | Auth tokens |
**AWS S3** - Production cloud storage.
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-e ARC_STORAGE_BACKEND=s3 \
-e ARC_STORAGE_S3_BUCKET=arc-data \
-e ARC_STORAGE_S3_REGION=us-east-1 \
-e AWS_ACCESS_KEY_ID=your_key \
-e AWS_SECRET_ACCESS_KEY=your_secret \
ghcr.io/basekick-labs/arc:latest
```
On EC2, use IAM roles instead of access keys for better security.
**MinIO** - Self-hosted S3-compatible storage.
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-e ARC_STORAGE_BACKEND=minio \
-e ARC_STORAGE_S3_ENDPOINT=minio:9000 \
-e ARC_STORAGE_S3_BUCKET=arc \
-e ARC_STORAGE_S3_ACCESS_KEY=minioadmin \
-e ARC_STORAGE_S3_SECRET_KEY=minioadmin123 \
-e ARC_STORAGE_S3_USE_SSL=false \
ghcr.io/basekick-labs/arc:latest
```
**Azure Blob Storage** - For Azure deployments.
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-e ARC_STORAGE_BACKEND=azure \
-e ARC_STORAGE_AZURE_CONTAINER=arc-data \
-e ARC_STORAGE_AZURE_ACCOUNT_NAME=your_account \
-e ARC_STORAGE_AZURE_ACCOUNT_KEY=your_key \
ghcr.io/basekick-labs/arc:latest
```
## Configuration [#configuration]
### Environment variables [#environment-variables]
Common configuration options:
| Variable | Default | Description |
| -------------------------- | --------- | --------------------------------------------------------------------- |
| `ARC_SERVER_PORT` | `8000` | HTTP port |
| `ARC_STORAGE_BACKEND` | `local` | Storage: `local`, `s3`, `minio`, `azure` |
| `ARC_LOG_LEVEL` | `info` | Logging: `debug`, `info`, `warn`, `error` |
| `ARC_AUTH_ENABLED` | `true` | Enable authentication |
| `ARC_AUTH_BOOTSTRAP_TOKEN` | *(unset)* | Pre-set admin token value on first run (v26.04.1+) |
| `ARC_AUTH_FORCE_BOOTSTRAP` | `false` | Add a recovery admin token without removing existing ones (v26.04.1+) |
| `ARC_COMPACTION_ENABLED` | `true` | Enable auto-compaction |
| `ARC_WAL_ENABLED` | `false` | Enable WAL for durability |
### Custom configuration file [#custom-configuration-file]
Mount a custom `arc.toml`:
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
-v /path/to/arc.toml:/app/arc.toml \
ghcr.io/basekick-labs/arc:latest
```
## Container management [#container-management]
### View logs [#view-logs]
```bash
docker logs -f arc # Follow logs
docker logs --tail=100 arc # Last 100 lines
```
### Start/stop/restart [#startstoprestart]
```bash
docker start arc
docker stop arc
docker restart arc
```
### Update Arc [#update-arc]
```bash
docker stop arc && docker rm arc
docker pull ghcr.io/basekick-labs/arc:latest
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
## Production deployment [#production-deployment]
### Pin version + resource limits [#pin-version--resource-limits]
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
--memory="8g" \
--cpus="4" \
--restart unless-stopped \
ghcr.io/basekick-labs/arc:latest
```
### Health check [#health-check]
```bash
docker ps --filter "name=arc" --filter "health=healthy"
```
## Docker Compose [#docker-compose]
```yaml
version: '3.8'
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
container_name: arc
ports:
- "8000:8000"
environment:
- ARC_STORAGE_BACKEND=local
- ARC_AUTH_ENABLED=true
- ARC_COMPACTION_ENABLED=true
volumes:
- arc-data:/app/data
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
volumes:
arc-data:
```
```yaml
version: '3.8'
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
container_name: arc
ports:
- "8000:8000"
environment:
- ARC_STORAGE_BACKEND=minio
- ARC_STORAGE_S3_ENDPOINT=minio:9000
- ARC_STORAGE_S3_BUCKET=arc
- ARC_STORAGE_S3_ACCESS_KEY=minioadmin
- ARC_STORAGE_S3_SECRET_KEY=minioadmin123
- ARC_STORAGE_S3_USE_SSL=false
depends_on:
- minio
restart: unless-stopped
minio:
image: minio/minio:latest
container_name: minio
ports:
- "9000:9000"
- "9001:9001"
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin123
command: server /data --console-address ":9001"
volumes:
- minio-data:/data
volumes:
minio-data:
```
```yaml
version: '3.8'
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
container_name: arc
ports:
- "8000:8000"
environment:
- ARC_STORAGE_BACKEND=local
- ARC_AUTH_ENABLED=true
- ARC_COMPACTION_ENABLED=true
- ARC_WAL_ENABLED=true
- ARC_WAL_SYNC_MODE=fdatasync
- ARC_LOG_LEVEL=info
- ARC_LOG_FORMAT=json
volumes:
- arc-data:/app/data
- arc-wal:/app/data/wal
deploy:
resources:
limits:
memory: 8G
cpus: '4'
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
volumes:
arc-data:
arc-wal:
```
## Troubleshooting [#troubleshooting]
### Container won't start [#container-wont-start]
```bash
# Check logs
docker logs arc
# Check port availability
sudo lsof -i :8000
# Check container status
docker ps -a
```
### Permission errors [#permission-errors]
```bash
# Remove and recreate volume
docker stop arc && docker rm arc
docker volume rm arc-data
# Restart with docker run command
```
### Out of memory [#out-of-memory]
```bash
# Check memory usage
docker stats arc
# Restart with memory limit
docker run -d --name arc --memory="4g" ...
```
### Can't find admin token [#cant-find-admin-token]
```bash
docker logs arc 2>&1 | grep -i "admin"
docker logs arc | head -100
```
## Next steps [#next-steps]
* [Write your first data](/arc/getting-started/#write-data)
* [Configure storage backends](/arc/configuration/overview/)
* [Deploy on Kubernetes](/arc/installation/kubernetes/)
# Installation (/arc/installation)
Arc ships as a single Go binary, so every install path below gets you the same server. Pick the one that matches where you are running it.
# Kubernetes Installation (/arc/installation/kubernetes)
Deploy Arc on Kubernetes using Helm for production-grade analytical data management.
## Prerequisites [#prerequisites]
* Kubernetes 1.24+
* Helm 3.0+
* `kubectl` configured to access your cluster
* Persistent storage (for local storage backend)
## Quick start [#quick-start]
```bash
# Install Arc
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
helm install arc https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc-${LATEST}.tgz
# Port forward to access locally
kubectl port-forward svc/arc 8000:8000
# Verify installation
curl http://localhost:8000/health
```
## Get your admin token [#get-your-admin-token]
```bash
# Get the pod name
kubectl get pods -l app=arc
# View logs to find admin token
kubectl logs -l app=arc | grep -i "admin"
```
You should see:
```bash
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Initial admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Copy this token immediately - you won't see it again!
In a multi-pod Arc Enterprise cluster, **only one pod prints the banner** — the Raft leader that wins the bootstrap election. Other pods log `INFO Deferring initial token bootstrap until cluster Raft proposer is wired` during startup and then `INFO Cluster auth state replication enabled — token writes now propagate via Raft` once the leader is elected. The non-leader pods silently no-op the bootstrap (they get an "already exists" response from the leader's FSM) and converge on the leader's token via Raft. The `kubectl logs -l app=arc | grep -i "admin"` command above still works — it just returns the single banner from whichever pod won the election. See [Cluster auth replication](/arc/configuration/authentication/#cluster-auth-replication-enterprise) for the full semantics, including token-propagation behaviour and the divergence-detection error log.
## Installation methods [#installation-methods]
```bash
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
helm install arc https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc-${LATEST}.tgz
```
```bash
# Download chart
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
helm pull https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc-${LATEST}.tgz
tar -xzf arc-${LATEST}.tgz
# Edit values
vim arc/values.yaml
# Install with custom values
helm install arc ./arc -f custom-values.yaml
```
```bash
# Create namespace
kubectl create namespace arc
# Install in namespace
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
helm install arc \
https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc-${LATEST}.tgz \
--namespace arc
```
## Storage backends [#storage-backends]
The OSS chart selects the backend with `arc.storageBackend` (`local`, `s3`, or
`minio`). The chart auto-sets `ARC_STORAGE_BACKEND` and `ARC_STORAGE_LOCAL_PATH`;
everything else (bucket, region, keys, endpoint) is passed through as
`ARC_STORAGE_*` environment variables via the free-form `arc.env[]` list.
**Local disk on a Persistent Volume** - the default. The chart provisions a PVC
mounted at `/app/data/arc`.
```yaml
# values.yaml
arc:
storageBackend: local
persistence:
enabled: true
size: 100Gi
accessMode: ReadWriteOnce
storageClass: "" # default storage class
```
```bash
helm install arc ./arc -f values.yaml
```
**AWS S3** - recommended for EKS. Pass the S3 settings through `arc.env[]`.
```yaml
# values.yaml
arc:
storageBackend: s3
env:
- name: ARC_STORAGE_S3_BUCKET
value: arc-production
- name: ARC_STORAGE_S3_REGION
value: us-east-1
- name: ARC_STORAGE_S3_USE_SSL
value: "true"
```
```bash
helm install arc ./arc -f values.yaml
```
The OSS chart never auto-injects S3 keys, so IRSA works by simply **omitting**
`ARC_STORAGE_S3_ACCESS_KEY` / `ARC_STORAGE_S3_SECRET_KEY` from `arc.env[]` and
attaching an IAM role to the ServiceAccount. Arc's AWS credential chain then
resolves the pod role automatically.
```yaml
arc:
storageBackend: s3
env:
- name: ARC_STORAGE_S3_BUCKET
value: arc-production
- name: ARC_STORAGE_S3_REGION
value: us-east-1
# no access/secret key — resolved via the pod IAM role
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/arc-s3
```
Full IRSA support requires Arc **26.09.1** or later. Earlier binaries have two
separate problems:
* On **26.06.1**, IRSA authenticates writes but not query reads at all.
* On **26.06.2 and 26.06.3**, query reads work for roughly **one hour** after each
pod starts and then fail with `ExpiredToken` until the pod is restarted. Writes
and `/health` keep working, so liveness and readiness probes do not catch it.
([#600](https://github.com/Basekick-Labs/arc/issues/600))
Set `image.tag: "26.09.1"` (or later) for IRSA that keeps working past the first
hour.
As of 26.09.1 Arc's credential refresher covers every source the AWS SDK
resolves: **IRSA** and **EKS Pod Identity** both work, as do EC2 instance
roles for non-EKS deployments. Static keys, when configured, always win.
With static keys (not recommended — prefer IRSA):
```yaml
arc:
storageBackend: s3
env:
- name: ARC_STORAGE_S3_BUCKET
value: arc-production
- name: ARC_STORAGE_S3_REGION
value: us-east-1
- name: ARC_STORAGE_S3_ACCESS_KEY
value: your_key
- name: ARC_STORAGE_S3_SECRET_KEY
value: your_secret
```
**MinIO** - self-hosted S3-compatible storage.
```yaml
# values.yaml
arc:
storageBackend: minio
env:
- name: ARC_STORAGE_S3_ENDPOINT
value: minio.minio-system.svc.cluster.local:9000
- name: ARC_STORAGE_S3_BUCKET
value: arc
- name: ARC_STORAGE_S3_ACCESS_KEY
value: minioadmin
- name: ARC_STORAGE_S3_SECRET_KEY
value: minioadmin123
- name: ARC_STORAGE_S3_USE_SSL
value: "false"
- name: ARC_STORAGE_S3_PATH_STYLE
value: "true"
```
Deploy with in-cluster MinIO:
```bash
# Install MinIO first
helm repo add minio https://charts.min.io/
helm install minio minio/minio --namespace minio-system --create-namespace
# Then install Arc
helm install arc ./arc -f values.yaml
```
## Configuration profiles [#configuration-profiles]
The OSS chart is a single Deployment. Arc's own settings (auth, WAL, log level,
ingest buffers, etc.) are passed as environment variables through `arc.env[]`,
not a structured `config:` block.
Minimal resources for development/testing:
```yaml
# values-dev.yaml
replicaCount: 1
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "2Gi"
cpu: "1"
arc:
storageBackend: local
env:
- name: ARC_AUTH_ENABLED
value: "false"
- name: ARC_LOG_LEVEL
value: "debug"
persistence:
enabled: true
size: 10Gi
```
```bash
helm install arc ./arc -f values-dev.yaml
```
Production-ready configuration backed by S3:
```yaml
# values-prod.yaml
replicaCount: 1
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "16Gi"
cpu: "8"
arc:
storageBackend: s3
env:
- name: ARC_STORAGE_S3_BUCKET
value: arc-production
- name: ARC_STORAGE_S3_REGION
value: us-east-1
- name: ARC_AUTH_ENABLED
value: "true"
- name: ARC_LOG_LEVEL
value: "info"
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/arc-s3
ingress:
enabled: true
className: nginx
hosts:
- host: arc.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: arc-tls
hosts:
- arc.example.com
```
```bash
helm install arc ./arc -f values-prod.yaml --namespace arc
```
Local disk on fast storage with WAL enabled:
```yaml
# values-durable.yaml
replicaCount: 1
resources:
requests:
memory: "8Gi"
cpu: "4"
limits:
memory: "32Gi"
cpu: "16"
arc:
storageBackend: local
env:
- name: ARC_AUTH_ENABLED
value: "true"
- name: ARC_WAL_ENABLED
value: "true"
- name: ARC_LOG_LEVEL
value: "info"
persistence:
enabled: true
size: 500Gi
storageClass: fast-ssd # use a fast storage class
nodeSelector:
node-type: high-memory
tolerations:
- key: "dedicated"
operator: "Equal"
value: "arc"
effect: "NoSchedule"
```
```bash
helm install arc ./arc -f values-durable.yaml --namespace arc
```
Arc reads its configuration from `ARC_*` environment variables. Confirm exact
names against the [configuration reference](/arc/configuration/overview/) before
relying on a specific key.
## Sizing the data volume when the WAL is on [#sizing-the-data-volume-when-the-wal-is-on]
With an object-storage backend (`storageBackend: s3`), the pod's PersistentVolume
holds only the WAL and local cache — but that does **not** make it small. The
WAL absorbs everything between ingest and flush, so size it against your **peak
ingest rate × worst-case flush lag**, with generous headroom. The default
`persistence.size: 10Gi` is a development default: at a bulk-load rate of
100 MB/s it fills in under two minutes.
A full WAL volume is a bad failure: the writer can't create its startup WAL file
on a full disk, so the pod cannot boot again until the volume is grown or
cleaned by hand (tracked in
[arc#676](https://github.com/Basekick-Labs/arc/issues/676)). For one-off bulk
migrations, either size the volume for the burst or set `ARC_WAL_ENABLED=false`
for the migration window and re-enable it afterwards.
## Helm values reference [#helm-values-reference]
### Core settings [#core-settings]
```yaml
# Number of pod replicas (single Deployment).
replicaCount: 1
# Container image (tag defaults to the chart appVersion).
image:
repository: ghcr.io/basekick-labs/arc
tag: "" # set "26.06.2" for full IRSA query-read support
pullPolicy: IfNotPresent
imagePullSecrets: []
# Service configuration
service:
type: ClusterIP
port: 8000
```
### Resources [#resources]
```yaml
resources: {} # empty by default; set requests/limits as needed
# requests:
# memory: "2Gi"
# cpu: "1"
# limits:
# memory: "8Gi"
# cpu: "4"
```
### Storage [#storage]
```yaml
arc:
# local | s3 | minio. The chart sets ARC_STORAGE_BACKEND from this and
# ARC_STORAGE_LOCAL_PATH automatically.
storageBackend: local
# Free-form env vars passed straight to the Arc container. Use ARC_STORAGE_*
# for bucket/region/keys/endpoint, plus any other ARC_* settings.
env: []
# - name: ARC_STORAGE_S3_BUCKET
# value: arc-production
# - name: ARC_STORAGE_S3_REGION
# value: us-east-1
# - name: ARC_STORAGE_S3_ENDPOINT
# value: https://s3.us-east-1.amazonaws.com
# - name: ARC_STORAGE_S3_ACCESS_KEY # omit for IRSA
# value: ""
# - name: ARC_STORAGE_S3_SECRET_KEY # omit for IRSA
# value: ""
# - name: ARC_STORAGE_S3_USE_SSL
# value: "true"
# - name: ARC_STORAGE_S3_PATH_STYLE
# value: "false"
# PVC used when storageBackend is local (mounted at /app/data/arc).
persistence:
enabled: true
accessMode: ReadWriteOnce
size: 10Gi
storageClass: ""
# existingClaim: ""
```
### Ingress [#ingress]
```yaml
ingress:
enabled: false
className: ""
annotations: {}
hosts:
- host: arc.local
paths:
- path: /
pathType: Prefix
tls: []
```
### Service account [#service-account]
```yaml
serviceAccount:
create: true
automount: true
name: ""
annotations: {} # eks.amazonaws.com/role-arn for IRSA on EKS
```
The OSS chart runs Arc as a single Deployment with no writer/reader/compactor
roles, Raft clustering, or multi-writer failover. For high availability,
multi-writer ingest, and peer replication see the
[Arc Enterprise Kubernetes guide](/arc-enterprise/installation/kubernetes/).
## Operations [#operations]
### View logs [#view-logs]
```bash
# Follow logs
kubectl logs -l app=arc -f
# Last 100 lines
kubectl logs -l app=arc --tail=100
# Logs from last hour
kubectl logs -l app=arc --since=1h
```
### Check status [#check-status]
```bash
# Pod status
kubectl get pods -l app=arc
# Describe pod
kubectl describe pod -l app=arc
# Check events
kubectl get events --field-selector involvedObject.name=arc-0
```
### Scale (restart) [#scale-restart]
```bash
# Restart pod
kubectl rollout restart deployment arc
# Or delete pod (will be recreated)
kubectl delete pod -l app=arc
```
### Port forward [#port-forward]
```bash
kubectl port-forward svc/arc 8000:8000
```
### Access shell [#access-shell]
```bash
kubectl exec -it $(kubectl get pod -l app=arc -o jsonpath='{.items[0].metadata.name}') -- /bin/sh
```
## Upgrade [#upgrade]
```bash
# Upgrade to new version
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
helm upgrade arc https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc-${LATEST}.tgz
# With custom values
helm upgrade arc ./arc -f values-prod.yaml
```
## Uninstall [#uninstall]
```bash
# Uninstall Arc
helm uninstall arc
# Delete PVCs (optional - removes all data!)
kubectl delete pvc -l app=arc
# Delete namespace (if dedicated)
kubectl delete namespace arc
```
## Monitoring [#monitoring]
### Prometheus metrics [#prometheus-metrics]
Arc exposes Prometheus metrics at `/metrics`:
```yaml
# ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: arc
spec:
selector:
matchLabels:
app: arc
endpoints:
- port: http
path: /metrics
interval: 30s
```
### Readiness/liveness probes [#readinessliveness-probes]
```yaml
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 30
readinessProbe:
httpGet:
path: /ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
```
As of **26.09.1**, `/health` includes a `storage` field with per-tier credential
state (`ok` / `degraded` / `expired` / `fallback` / `unknown`), computed from
Arc's credential refresher — no S3/Azure probing from the probe path. Alert on
`storage.*.state != "ok"` or on `expires_at` approaching. For **reader pools**,
setting `server.storage_credentials_fail_ready = true` makes `/ready` return
503 while any tier's credentials are `expired`, so Kubernetes recycles the pod
(a restart re-resolves credentials). Leave it off for writers — ingest keeps
working through credential expiry, and Arc warns at startup if it is enabled on
a cluster writer.
## Troubleshooting [#troubleshooting]
### Pod won't start [#pod-wont-start]
```bash
# Check pod status
kubectl describe pod -l app=arc
# Check events
kubectl get events --sort-by='.lastTimestamp'
# Common issues:
# - ImagePullBackOff: Check image name/tag
# - Pending: Check PVC status, node resources
# - CrashLoopBackOff: Check logs
```
### Storage issues [#storage-issues]
```bash
# Check PVC status
kubectl get pvc -l app=arc
# Check PV
kubectl get pv
# Describe PVC for errors
kubectl describe pvc -l app=arc
```
### Connection issues [#connection-issues]
```bash
# Check service
kubectl get svc arc
# Test from within cluster
kubectl run curl --image=curlimages/curl -it --rm -- curl http://arc:8000/health
```
### Memory issues [#memory-issues]
```bash
# Check resource usage
kubectl top pod -l app=arc
# Increase limits in values.yaml
resources:
limits:
memory: "16Gi"
```
## Next steps [#next-steps]
* [Write your first data](/arc/getting-started/#write-data)
* [Configure storage backends](/arc/configuration/overview/)
* [Set up monitoring](/arc/operations/telemetry/)
* [Enable WAL for durability](/arc/advanced/wal/)
# Native Installation (/arc/installation/native)
Install Arc directly on Linux using native packages (.deb, .rpm) or build from source.
Current release: ****
## Prerequisites [#prerequisites]
* Linux (x86\_64 or ARM64)
* 4GB RAM minimum, 8GB+ recommended
* systemd (for service management)
## Quick install [#quick-install]
The following commands automatically fetch and install the latest Arc release.
**x86\_64 (AMD/Intel):**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc_${LATEST_VERSION}_amd64.deb
sudo dpkg -i arc_${LATEST_VERSION}_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
curl http://localhost:8000/health
```
**ARM64:**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc_${LATEST_VERSION}_arm64.deb
sudo dpkg -i arc_${LATEST_VERSION}_arm64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
**x86\_64 (AMD/Intel):**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1.x86_64.rpm
sudo rpm -i arc-${LATEST_VERSION}-1.x86_64.rpm
sudo systemctl enable arc && sudo systemctl start arc
curl http://localhost:8000/health
```
**ARM64:**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1.aarch64.rpm
sudo rpm -i arc-${LATEST_VERSION}-1.aarch64.rpm
sudo systemctl enable arc && sudo systemctl start arc
```
Works on Arch Linux and Arch-based distros such as Omarchy.
**x86\_64 (AMD/Intel):**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1-x86_64.pkg.tar.zst
sudo pacman -U arc-${LATEST_VERSION}-1-x86_64.pkg.tar.zst
sudo systemctl enable arc && sudo systemctl start arc
curl http://localhost:8000/health
```
**ARM64:**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1-aarch64.pkg.tar.zst
sudo pacman -U arc-${LATEST_VERSION}-1-aarch64.pkg.tar.zst
sudo systemctl enable arc && sudo systemctl start arc
```
Prerequisites: Go 1.25+, Git, Make
```bash
# Clone and build
git clone https://github.com/basekick-labs/arc.git
cd arc
make build
# Run
./arc
```
Development commands:
```bash
make deps # Install dependencies
make build # Build binary
make run # Run without building
make test # Run tests
make test-coverage # Run tests with coverage
make bench # Run benchmarks
make lint # Run linter
make clean # Clean build artifacts
```
If you need validated cryptography (FIPS 140-2/3, FedRAMP, NIST SP 800-171 /
CMMC), install the `arc-fips` build instead — same version, same features, built
against the CMVP-certified Go Cryptographic Module. See
[FIPS 140-3 mode](/arc/configuration/fips/). Note there is no `arc-fips` Arch
package; on Arch use the standalone binary.
## Get your admin token [#get-your-admin-token]
When Arc starts for the first time, it generates an admin token.
Copy this token immediately - you won't see it again!
```bash
sudo journalctl -u arc | grep -i "admin"
```
You should see:
```bash
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Initial admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save it:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
## Storage backends [#storage-backends]
**Local Filesystem** - Default, data stored on disk.
Edit `/etc/arc/arc.toml`:
```toml
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
```
Or via environment:
```ini
ARC_STORAGE_BACKEND=local
ARC_STORAGE_LOCAL_PATH=/var/lib/arc/data
```
**AWS S3** - Production cloud storage.
Edit `/etc/arc/arc.toml`:
```toml
[storage]
backend = "s3"
s3_bucket = "arc-production"
s3_region = "us-east-1"
# Use IAM roles or environment variables for credentials
```
Environment variables:
```ini
ARC_STORAGE_BACKEND=s3
ARC_STORAGE_S3_BUCKET=arc-data
ARC_STORAGE_S3_REGION=us-east-1
AWS_ACCESS_KEY_ID=your_key
AWS_SECRET_ACCESS_KEY=your_secret
```
On EC2, use IAM instance profiles for automatic credential management.
**MinIO** - Self-hosted S3-compatible storage.
Edit `/etc/arc/arc.toml`:
```toml
[storage]
backend = "minio"
s3_bucket = "arc"
s3_endpoint = "minio.local:9000"
s3_access_key = "minioadmin"
s3_secret_key = "minioadmin123"
s3_use_ssl = false
s3_path_style = true
```
**Azure Blob Storage** - For Azure deployments.
Edit `/etc/arc/arc.toml`:
```toml
[storage]
backend = "azure"
azure_container = "arc-data"
azure_account_name = "your_account"
azure_account_key = "your_key"
```
## Service management [#service-management]
### Start/stop/restart [#startstoprestart]
```bash
sudo systemctl start arc # Start
sudo systemctl stop arc # Stop
sudo systemctl restart arc # Restart
sudo systemctl status arc # Status
```
### View logs [#view-logs]
```bash
sudo journalctl -u arc -f # Follow logs
sudo journalctl -u arc -n 100 # Last 100 lines
sudo journalctl -u arc --since "1 hour ago"
```
### Enable/disable auto-start [#enabledisable-auto-start]
```bash
sudo systemctl enable arc # Enable on boot
sudo systemctl disable arc # Disable on boot
```
## Configuration [#configuration]
Configuration file: `/etc/arc/arc.toml`
```bash
sudo nano /etc/arc/arc.toml
sudo systemctl restart arc
```
### Common options [#common-options]
```toml
[server]
port = 8000
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
daily_enabled = true
[wal]
enabled = false # Enable for zero data loss
sync_mode = "fdatasync"
[log]
level = "info"
format = "json"
```
See [Configuration Overview](/arc/configuration/overview/) for all options.
## Data directory [#data-directory]
| Installation Type | Default Data Directory |
| ----------------- | ---------------------- |
| Package install | `/var/lib/arc/data` |
| Source build | `./data/arc` |
## Updating Arc [#updating-arc]
```bash
# Automatic update to latest version
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc_${LATEST_VERSION}_amd64.deb
sudo dpkg -i arc_${LATEST_VERSION}_amd64.deb
sudo systemctl restart arc
```
```bash
# Automatic update to latest version
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1.x86_64.rpm
sudo rpm -U arc-${LATEST_VERSION}-1.x86_64.rpm
sudo systemctl restart arc
```
```bash
# Automatic update to latest version
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1-x86_64.pkg.tar.zst
sudo pacman -U arc-${LATEST_VERSION}-1-x86_64.pkg.tar.zst
sudo systemctl restart arc
```
Your edited `/etc/arc/arc.toml` is preserved across upgrades; if the packaged default changed, pacman writes it as `arc.toml.pacnew`.
```bash
cd arc
git pull
make build
# Restart Arc manually
```
## Uninstalling [#uninstalling]
```bash
sudo systemctl stop arc
sudo dpkg -r arc
# Optional: Remove data
sudo rm -rf /var/lib/arc /etc/arc
```
```bash
sudo systemctl stop arc
sudo rpm -e arc
# Optional: Remove data
sudo rm -rf /var/lib/arc /etc/arc
```
```bash
sudo pacman -R arc # stops and disables the service automatically
# Optional: Remove data
sudo rm -rf /var/lib/arc /etc/arc
```
## Troubleshooting [#troubleshooting]
### Arc won't start [#arc-wont-start]
```bash
# Check logs
sudo journalctl -u arc -n 50
# Check port availability
sudo lsof -i :8000
```
### Permission errors [#permission-errors]
```bash
sudo mkdir -p /var/lib/arc/data
sudo chown -R arc:arc /var/lib/arc
```
### Memory issues [#memory-issues]
Override in `/etc/arc/arc.toml`:
```toml
[database]
memory_limit = "4GB"
max_connections = 16
thread_count = 8
```
## Next steps [#next-steps]
* [Write your first data](/arc/getting-started/#write-data)
* [Configure storage backends](/arc/configuration/overview/)
* [Deploy on Kubernetes](/arc/installation/kubernetes/)
* [Set up compaction](/arc/advanced/compaction/)
# Apache Iceberg Export (/arc/integrations/apache-iceberg)
Publish Arc's data as **Apache Iceberg** tables so any Iceberg-aware engine — Spark, Trino, DuckDB, Snowflake, PyIceberg — can query it directly, without going through Arc.
Iceberg export is available in **Arc 26.09.1+**. Enable it with `iceberg.enabled = true`.
## What it is [#what-it-is]
[Apache Iceberg](https://iceberg.apache.org/) is an open **table format**: a metadata layer that turns a collection of data files into a coherent, transactional table with schema, snapshots, and partition information. It is **not a file format** — Iceberg tables are backed by Parquet (which Arc already writes).
Arc's Iceberg export is a background **reconciler** that registers Arc's **existing** Parquet files into an Iceberg table *by reference*, and keeps the table's file list in sync as compaction and retention change the underlying files.
## Why use it [#why-use-it]
* **No lock-in, taken further.** Arc already stores open Parquet files you own. Iceberg export makes those same files a standard lakehouse table that the entire Iceberg ecosystem can read — no proprietary API in the path.
* **Zero-copy.** Files are registered in place using Iceberg's `add_files` semantics. **No data is copied or rewritten**, so there is effectively no storage overhead beyond small Iceberg metadata. (This is a real differentiator: many streaming/TSDB systems re-export data into *new* Parquet; Arc doesn't need to, because it already writes Parquet.)
* **Query with your existing tools.** Spark, Trino, DuckDB, Snowflake, Dremio, and PyIceberg all read Iceberg. Point them at Arc's data instead of building a custom connector.
* **Ingest is untouched.** The export runs on a background timer and never touches Arc's high-throughput write path.
## How it works [#how-it-works]
1. Arc ingests as usual, writing Parquet under `{database}/{measurement}/{Y}/{M}/{D}/{H}/`.
2. On a timer (default every 5 minutes), the reconciler walks each measurement's Parquet files and diffs them against the Iceberg table's current file set.
3. It commits the delta in one Iceberg snapshot — adding newly-written files and removing files that compaction/retention deleted — **without rewriting any data**.
4. Old snapshots are expired on a retention policy so metadata stays bounded.
Because the reconciler is driven by what's actually on storage (not a transient event stream), it is **self-healing**: a missed or failed pass simply converges on the next tick. Measurements whose file set hasn't changed since the last pass are skipped entirely, so steady state is cheap.
This holds all the way to zero: if retention ages out *every* file in a measurement, the table is reconciled to empty rather than left pointing at files that no longer exist — so external engines see an empty table instead of failing on missing paths.
Tables are created per database/measurement in the namespace `_` (default prefix `arc`), e.g. `arc_mydb.sensors`.
## Quick start [#quick-start]
```toml
# arc.toml
[storage]
backend = "local" # Iceberg export requires a local backend in v1 (see Limitations)
[iceberg]
enabled = true # default: false
```
Or via environment variable:
```ini
ARC_ICEBERG_ENABLED=true
```
Restart Arc. Within one reconcile interval, ingested measurements appear as Iceberg tables under your storage root (`{local_path}/arc_.db//`).
## Reading the tables [#reading-the-tables]
### DuckDB [#duckdb]
The simplest way to read a table — point `iceberg_scan` at the table directory:
```sql
INSTALL iceberg; LOAD iceberg;
SELECT count(*)
FROM iceberg_scan('/var/lib/arc/data/arc_mydb.db/cpu');
```
Arc emits a `version-hint.text` in each table's `metadata/` directory, so directory-based readers like DuckDB resolve the current snapshot without needing the exact metadata filename.
### PyIceberg [#pyiceberg]
PyIceberg can read through Arc's SQLite catalog directly:
```python
from pyiceberg.catalog.sql import SqlCatalog
cat = SqlCatalog("arc", **{
"uri": "sqlite:////var/lib/arc/data/arc.db", # Arc's catalog DB
"warehouse": "file:///var/lib/arc/data", # storage root
})
table = cat.load_table(("arc_mydb", "cpu"))
df = table.scan().to_arrow()
print(df.num_rows, df.column_names)
```
### Apache Spark [#apache-spark]
Read a table from its directory with the Iceberg runtime and a Hadoop-style catalog, or load it directly:
```python
df = (spark.read.format("iceberg")
.load("file:///var/lib/arc/data/arc_mydb.db/cpu"))
df.printSchema()
df.createOrReplaceTempView("cpu")
spark.sql("SELECT count(*) FROM cpu").show()
```
### Trino / Snowflake / others [#trino--snowflake--others]
Any engine with an Iceberg connector can read the tables. For engines that require a shared catalog service (Trino's JDBC catalog supports PostgreSQL/MySQL, not SQLite), point the connector at the Iceberg metadata; a shared REST or JDBC catalog in front of the warehouse is the path for broad multi-engine access. See **Limitations** below.
## Configuration reference [#configuration-reference]
| Key (`arc.toml`) | Env var | Default | Meaning |
| ---------------------------- | -------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------- |
| `iceberg.enabled` | `ARC_ICEBERG_ENABLED` | `false` | Enable the export reconciler. |
| `iceberg.reconcile_interval` | `ARC_ICEBERG_RECONCILE_INTERVAL` | `300` | Seconds between reconcile passes. |
| `iceberg.retain_snapshots` | `ARC_ICEBERG_RETAIN_SNAPSHOTS` | `10` | Iceberg snapshots (and metadata versions) kept per table; older are expired to bound metadata growth. |
| `iceberg.namespace_prefix` | `ARC_ICEBERG_NAMESPACE_PREFIX` | `arc` | Namespace prefix; tables land in `_`. |
| `iceberg.warehouse` | `ARC_ICEBERG_WAREHOUSE` | *storage root* | Root URI where table metadata is written. Defaults alongside the data. |
| `iceberg.catalog_db_path` | `ARC_ICEBERG_CATALOG_DB_PATH` | *shared auth DB* | SQLite catalog location. |
## Schema mapping [#schema-mapping]
Arc's columns map to Iceberg types as follows:
| Arc / Arrow | Iceberg |
| -------------------------- | -------------- |
| `time` (Timestamp µs, UTC) | `timestamptz` |
| int64 | `long` |
| int32 | `int` |
| float64 | `double` |
| float32 | `float` |
| string | `string` |
| bool | `boolean` |
| decimal128 | `decimal(P,S)` |
Arc's own ingest writes int64, float64, string, bool, and timestamps; the 32-bit and decimal types are there for Parquet that arrives through the bulk import API.
Tables are partitioned by `day(time)`.
### Schema changes [#schema-changes]
**Adding columns is automatic.** If a measurement gains a column over time, the Iceberg table's schema is evolved to include it, and older files (written before the column existed) stay readable — the new column simply reads as `NULL` for those rows.
**Changing a column's type is not supported.** If a column's type changes (say `value` was written as an integer and later arrives as a float), Iceberg cannot represent both in one column. Arc refuses to reconcile that measurement and logs an error naming the column and both types:
```bash
column "value" type mismatch: Iceberg table has long, new Parquet files have double
(the measurement's column type changed; Iceberg cannot represent both in one column)
```
The Iceberg table is left untouched and stays readable at its last good state, and your other measurements keep exporting normally — one bad measurement never blocks the rest. Arc retries on every reconcile pass, so once the underlying type conflict is resolved the measurement recovers on its own without a restart.
The conflict lives in the data, not in Iceberg, so fix it at the source: keep a measurement's column type stable, or write the new type under a different column or measurement name.
## Limitations (v1) [#limitations-v1]
* **Local storage only.** Iceberg export requires `storage.backend = "local"`. Arc **refuses to start** if Iceberg export is enabled with a non-local primary backend or with cold-tier tiering, because a file migrated to object storage would silently leave the Iceberg table.
* **No cold-tier tiering.** Same reason as above — disable one of `iceberg.enabled` / `tiered_storage.cold.enabled`.
* **Eventual consistency.** The Iceberg view reflects the last reconcile pass, so it lags live ingest by up to `reconcile_interval`. This is expected for a lakehouse export.
* **On-disk portability.** Iceberg metadata references files by absolute path. Tables read fine on the same host; moving a local table to a different path/host requires re-pointing (object-store warehouses avoid this).
* **Catalog discovery.** Catalog-aware access works today via the SQLite catalog (PyIceberg) or by pointing engines at the table directory (DuckDB, Spark). Broad multi-engine *catalog* discovery (Trino, Glue) would use a REST/JDBC catalog in front of the warehouse — a deployment choice beyond v1.
* **Clustered deployments.** Exactly one node must run the reconciler. Under the compactor failover lease this is guaranteed; in a static-role cluster, enable Iceberg export where a single compaction-capable node runs.
## Backup & restore [#backup--restore]
Arc's backup includes the Iceberg warehouse metadata (`metadata.json`, manifest `.avro`, `version-hint.text`) alongside the Parquet data, so a restored deployment keeps its Iceberg tables intact.
## FAQ [#faq]
**Does this slow down ingestion?** No. The export is a background reconciler; the write path is unchanged.
**Does it duplicate my data?** No. It registers your existing Parquet files by reference. Only small Iceberg metadata is written.
**Can I still use Arc's SQL API?** Yes. Iceberg export is additive — Arc's native query API is unaffected. The same Parquet files serve both.
**What happens to compacted/retained files?** The reconciler reflects them: when compaction replaces small files with a larger one, or retention deletes old files, the next pass updates the Iceberg table's file set accordingly.
# Grafana Integration (/arc/integrations/grafana)
Connect Arc to Grafana for real-time monitoring, alerting, and beautiful visualizations using the Arc datasource plugin.
## Overview [#overview]
The Arc datasource plugin for Grafana provides:
* **Apache Arrow Protocol**: High-performance columnar data transfer
* **Native SQL Support**: Full analytical SQL with syntax highlighting
* **Template Variables**: Dynamic dashboards with filters
* **Alerting**: Built-in alert rule support
* **Multi-database**: Query across different Arc databases
* **Real-time Dashboards**: Sub-second query performance
## Installation [#installation]
### From Grafana plugin catalog [#from-grafana-plugin-catalog]
1. In Grafana, go to **Configuration** → **Plugins**
2. Search for **Arc**
3. Click **Install**
4. Restart Grafana if prompted
### From release [#from-release]
```bash
# Resolve the latest release tag, then download the matching plugin archive
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/grafana-arc-datasource/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/grafana-arc-datasource/releases/download/v${LATEST}/basekick-arc-datasource-${LATEST}.zip
# Extract to Grafana plugins directory
unzip basekick-arc-datasource-${LATEST}.zip -d /var/lib/grafana/plugins/
# Restart Grafana
systemctl restart grafana-server
```
### From source [#from-source]
```bash
# Clone repository
git clone https://github.com/basekick-labs/grafana-arc-datasource
cd grafana-arc-datasource
# Install dependencies
npm install
# Build plugin
npm run build
# Build backend
mage -v
# Install to Grafana
cp -r dist /var/lib/grafana/plugins/basekick-arc-datasource
systemctl restart grafana-server
```
## Configuration [#configuration]
### 1. Add data source [#1-add-data-source]
1. In Grafana, go to **Configuration** → **Data sources**
2. Click **Add data source**
3. Search for and select **Arc**
4. Configure connection settings
### 2. Connection settings [#2-connection-settings]
| Setting | Description | Required | Default |
| ------------- | ---------------------------- | -------- | ----------------------- |
| **URL** | Arc API endpoint | Yes | `http://localhost:8000` |
| **API Key** | Authentication token | Yes | - |
| **Database** | Default database name | No | `default` |
| **Timeout** | Query timeout in seconds | No | `30` |
| **Use Arrow** | Enable Apache Arrow protocol | No | `true` |
### 3. Example configuration [#3-example-configuration]
```text
URL: http://localhost:8000
API Key: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Database: prod
Timeout: 30
```
Click **Save & Test** to verify the connection.
### 4. Get your API token [#4-get-your-api-token]
```bash
# Docker - check logs for admin token
docker logs 2>&1 | grep "Admin token"
# Or create a new token specifically for Grafana
curl -X POST http://localhost:8000/api/v1/auth/tokens \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "grafana-datasource",
"description": "Grafana datasource access"
}'
```
## Creating queries [#creating-queries]
### Query editor [#query-editor]
The Arc datasource provides a SQL query editor with:
* Syntax highlighting
* Auto-completion
* Time range macros
* Multi-database support
### Basic query example [#basic-query-example]
**CPU Usage:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(usage_idle) * -1 + 100 AS cpu_usage,
host
FROM prod.cpu
WHERE cpu = 'cpu-total'
AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
### Time macros [#time-macros]
Grafana provides powerful time macros for dynamic queries:
| Macro | Description | Example |
| --------------------------- | -------------------------- | ------------------------------------------- |
| `$__timeFilter(columnName)` | Complete time range filter | `WHERE $__timeFilter(time)` |
| `$__timeFrom()` | Start of time range | `time >= $__timeFrom()` |
| `$__timeTo()` | End of time range | `time < $__timeTo()` |
| `$__interval` | Auto-calculated interval | `time_bucket(INTERVAL '$__interval', time)` |
**How macros expand:**
```sql
-- Your query
WHERE $__timeFilter(time)
-- Expands to
WHERE time >= '2025-01-17 10:00:00' AND time < '2025-01-17 11:00:00'
```
### Example queries [#example-queries]
**Memory Usage:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(used_percent) AS memory_used,
host
FROM prod.mem
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
**Network Traffic (bytes to bits):**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(bytes_recv) * 8 AS bits_in,
AVG(bytes_sent) * 8 AS bits_out,
host,
interface
FROM prod.net
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host, interface
ORDER BY time ASC
```
**Disk I/O:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(read_bytes) AS disk_read,
AVG(write_bytes) AS disk_write,
host
FROM prod.diskio
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
## Template variables [#template-variables]
Create dynamic dashboards with variables that filter your data.
### Creating variables [#creating-variables]
1. Go to **Dashboard settings** → **Variables**
2. Click **Add variable**
3. Configure variable settings
### Variable examples [#variable-examples]
**Host Variable:**
```sql
SELECT DISTINCT host FROM prod.cpu ORDER BY host
```
**Interface Variable:**
```sql
SELECT DISTINCT interface FROM prod.net ORDER BY interface
```
**Database Variable:**
```sql
SELECT DISTINCT schema_name FROM information_schema.schemata
WHERE schema_name NOT IN ('information_schema', 'pg_catalog')
ORDER BY schema_name
```
### Using variables in queries [#using-variables-in-queries]
Reference variables with `$variable` syntax:
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(usage_idle) * -1 + 100 AS cpu_usage
FROM $database.cpu
WHERE host = '$server'
AND cpu = 'cpu-total'
AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time)
ORDER BY time ASC
```
### Multi-select variables [#multi-select-variables]
Enable **Multi-value** in variable settings, then use `IN`:
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(usage_idle) * -1 + 100 AS cpu_usage,
host
FROM prod.cpu
WHERE host IN ($hosts) -- Multi-select variable
AND cpu = 'cpu-total'
AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
## Alerting [#alerting]
The Arc datasource fully supports Grafana alerting.
### Creating alert rules [#creating-alert-rules]
1. Open a panel with an Arc query
2. Go to **Alert** tab
3. Click **Create alert rule from this panel**
4. Configure alert conditions
### Example alert query [#example-alert-query]
**High CPU Usage (> 80%):**
```sql
SELECT
time,
100 - usage_idle AS cpu_usage,
host
FROM prod.cpu
WHERE cpu = 'cpu-total'
AND time >= NOW() - INTERVAL '5 minutes'
ORDER BY time ASC
```
**Alert Condition:**
* `WHEN avg() OF query(A, 5m, now) IS ABOVE 80`
### Example alert: Memory usage [#example-alert-memory-usage]
**Query:**
```sql
SELECT
time,
used_percent AS memory_used,
host
FROM prod.mem
WHERE time >= NOW() - INTERVAL '5 minutes'
ORDER BY time ASC
```
**Alert Condition:**
* `WHEN avg() OF query(A, 5m, now) IS ABOVE 90`
### Alert notifications [#alert-notifications]
Configure notification channels:
1. Go to **Alerting** → **Contact points**
2. Add notification channel (Email, Slack, PagerDuty, etc.)
3. Link alert rules to notification channels
## Dashboard examples [#dashboard-examples]
### System monitoring dashboard [#system-monitoring-dashboard]
Create a comprehensive system monitoring dashboard:
**Panels:**
1. **CPU Usage by Host** (Time series)
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(100 - usage_idle) AS cpu_usage,
host
FROM prod.cpu
WHERE cpu = 'cpu-total' AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
2. **Memory Usage** (Time series)
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(used_percent) AS memory_used,
host
FROM prod.mem
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
3. **Disk Usage** (Gauge)
```sql
SELECT
host,
AVG(used_percent) AS disk_used
FROM prod.disk
WHERE $__timeFilter(time)
GROUP BY host
```
4. **Network Traffic** (Graph)
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
SUM(bytes_recv) * 8 / 1000000 AS mbps_in,
SUM(bytes_sent) * 8 / 1000000 AS mbps_out,
host
FROM prod.net
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
5. **Top Hosts by CPU** (Bar gauge)
```sql
SELECT
host,
AVG(100 - usage_idle) AS avg_cpu
FROM prod.cpu
WHERE cpu = 'cpu-total'
AND time >= NOW() - INTERVAL '1 hour'
GROUP BY host
ORDER BY avg_cpu DESC
LIMIT 10
```
### Dashboard layout [#dashboard-layout]
```text
┌─────────────────────────────────────────────────┐
│ System Overview - Last 24 Hours │
│ [Host: All ▼] [Refresh: 30s ▼] │
├───────────────────────┬─────────────────────────┤
│ │ │
│ CPU Usage │ Memory Usage │
│ (Time Series) │ (Time Series) │
│ │ │
├───────────────────────┼─────────────────────────┤
│ │ │
│ Network Traffic │ Disk I/O │
│ (Graph) │ (Graph) │
│ │ │
├───────────────────────┴─────────────────────────┤
│ Top 10 Hosts by CPU Usage (Bar Gauge) │
└─────────────────────────────────────────────────┘
```
## Advanced queries [#advanced-queries]
### Window functions [#window-functions]
**Moving Average:**
```sql
SELECT
time,
usage_idle,
host,
AVG(usage_idle) OVER (
PARTITION BY host
ORDER BY time
ROWS BETWEEN 5 PRECEDING AND CURRENT ROW
) as moving_avg
FROM prod.cpu
WHERE cpu = 'cpu-total' AND $__timeFilter(time)
ORDER BY time ASC
```
### Percentiles [#percentiles]
**CPU Usage Percentiles:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
host,
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY usage_idle) as p50,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
FROM prod.cpu
WHERE cpu = 'cpu-total' AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
### Cross-database queries [#cross-database-queries]
**Production vs Staging Comparison:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(p.usage_idle) as prod_cpu_idle,
AVG(s.usage_idle) as staging_cpu_idle
FROM prod.cpu p
JOIN staging.cpu s ON p.time = s.time AND p.host = s.host
WHERE p.cpu = 'cpu-total'
AND s.cpu = 'cpu-total'
AND $__timeFilter(p.time)
GROUP BY time_bucket(INTERVAL '$__interval', time)
ORDER BY time ASC
```
## Performance optimization [#performance-optimization]
### 1. Use Apache Arrow [#1-use-apache-arrow]
Arrow protocol is enabled by default and provides significantly faster data transfer:
* Substantially faster than JSON for large result sets
* Zero-copy deserialization
* Columnar format perfect for time-series
### 2. Optimize time ranges [#2-optimize-time-ranges]
* Use Grafana's time picker to limit data scanned
* Add time filters with `$__timeFilter()`
* Avoid querying months of data for real-time dashboards
### 3. Leverage time\_bucket() [#3-leverage-time_bucket]
Grafana automatically adjusts `$__interval` based on dashboard width:
```sql
-- Good: Automatic interval adjustment
time_bucket(INTERVAL '$__interval', time)
-- Bad: Fixed interval (too many points)
time_bucket(INTERVAL '1 second', time)
```
### 4. Use LIMIT for exploration [#4-use-limit-for-exploration]
```sql
SELECT * FROM prod.cpu
WHERE $__timeFilter(time)
LIMIT 1000 -- Limit result size
```
### 5. Enable query caching [#5-enable-query-caching]
In Grafana's data source settings:
* Enable **Cache timeout**: 60 seconds
* Repeated queries return instantly from cache
## Troubleshooting [#troubleshooting]
### Plugin not appearing [#plugin-not-appearing]
```bash
# Check plugin directory permissions
ls -la /var/lib/grafana/plugins/basekick-arc-datasource
# Verify plugin.json exists
cat /var/lib/grafana/plugins/basekick-arc-datasource/plugin.json
# Check Grafana logs
tail -f /var/log/grafana/grafana.log
# Restart Grafana
systemctl restart grafana-server
```
### Connection failed [#connection-failed]
```bash
# Verify Arc is running
curl http://localhost:8000/health
# Test API token
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/auth/verify
# Check network connectivity
ping localhost
```
### Query errors [#query-errors]
**"Table not found":**
```sql
-- List available tables
SHOW TABLES FROM prod;
-- Verify database exists
SHOW DATABASES;
```
**"Column not found":**
```sql
-- Describe table schema
DESCRIBE prod.cpu;
```
### Slow queries [#slow-queries]
```bash
# Check Arc query performance
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"sql": "EXPLAIN SELECT * FROM prod.cpu WHERE time > NOW() - INTERVAL '\''1 hour'\''",
"format": "json"
}'
# Trigger compaction
curl -X POST http://localhost:8000/api/v1/compaction/trigger \
-H "Authorization: Bearer $ARC_TOKEN"
```
### Backend plugin issues [#backend-plugin-issues]
```bash
# Ensure backend binary is compiled
cd /path/to/grafana-arc-datasource
mage -v
# Check binary permissions
chmod +x dist/gpx_arc-datasource_*
# Verify Go version
go version # Should be 1.21+
```
## Performance tips [#performance-tips]
1. **Use Arrow Protocol**: Enabled by default, provides considerably faster data transfer
2. **Optimize Time Ranges**: Smaller ranges = faster queries
3. **Leverage time\_bucket()**: Use `$__interval` for automatic aggregation
4. **Add Indexes**: Arc automatically indexes time columns
5. **Enable Caching**: Configure query caching in datasource settings
6. **Limit Result Size**: Use `LIMIT` for exploratory queries
7. **Use Variables**: Filter data with template variables instead of loading everything
## Resources [#resources]
* **[Grafana Arc Datasource GitHub](https://github.com/basekick-labs/grafana-arc-datasource)**
* **[Grafana Documentation](https://grafana.com/docs/grafana/latest/)**
* **[Arc Query API](/arc/api-reference/overview/#querying)**
* **[DuckDB SQL Reference](https://duckdb.org/docs/sql/introduction)**
## Next steps [#next-steps]
* **[Query API Reference](/arc/api-reference/overview/)**
* **[Telegraf Integration](/arc/integrations/telegraf/)** - Collect system metrics
* **[Apache Superset Integration](/arc/integrations/superset/)** - BI dashboards
# Integrations (/arc/integrations)
Arc speaks Line Protocol and standard SQL, so most tooling connects without a translation layer. These pages cover the integrations with a native plugin or dialect.
# InfluxDB Client Compatibility (/arc/integrations/influxdb-clients)
Arc's Line Protocol endpoints use the same paths as InfluxDB, enabling drop-in compatibility with all official InfluxDB client libraries. Point your existing InfluxDB client at Arc - it just works.
## Supported clients [#supported-clients]
All official InfluxDB client libraries work with Arc without code changes:
| Language | Library | Version |
| ------------------ | ------------------------------------------ | ------- |
| Go | `github.com/influxdata/influxdb-client-go` | v2.x |
| Python | `influxdb-client` | v1.x |
| JavaScript/Node.js | `@influxdata/influxdb-client` | v1.x |
| Java | `influxdb-client-java` | v6.x |
| C# | `InfluxDB.Client` | v4.x |
| PHP | `influxdb-client-php` | v3.x |
| Ruby | `influxdb-client-ruby` | v2.x |
**Also supported:**
* Telegraf (InfluxDB output plugin)
* Node-RED (`node-red-contrib-influxdb`)
* Grafana InfluxDB datasource
* Any tool using InfluxDB Line Protocol
## Endpoint mapping [#endpoint-mapping]
| InfluxDB Endpoint | Arc Endpoint | Use Case |
| ----------------- | --------------- | -------------------- |
| `/write` | `/write` | InfluxDB 1.x clients |
| `/api/v2/write` | `/api/v2/write` | InfluxDB 2.x clients |
## Authentication methods [#authentication-methods]
Arc supports all InfluxDB authentication styles:
| Method | Header/Parameter | Example |
| --------------- | ------------------------------- | ------------------ |
| Bearer Token | `Authorization: Bearer ` | Standard OAuth2 |
| Token Header | `Authorization: Token ` | InfluxDB 2.x style |
| Query Parameter | `?p=` | InfluxDB 1.x style |
## Quick start examples [#quick-start-examples]
### Python (influxdb-client) [#python-influxdb-client]
```python
from influxdb_client import InfluxDBClient, Point
from influxdb_client.client.write_api import SYNCHRONOUS
# Point to Arc instead of InfluxDB
client = InfluxDBClient(
url="http://localhost:8000",
token="your-arc-token",
org="myorg" # Required but ignored by Arc
)
write_api = client.write_api(write_options=SYNCHRONOUS)
# Write data - works exactly like InfluxDB
point = Point("cpu") \
.tag("host", "server01") \
.field("usage", 45.2)
write_api.write(bucket="mydb", record=point)
client.close()
```
### Go (influxdb-client-go) [#go-influxdb-client-go]
```go
package main
import (
"context"
"time"
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
)
func main() {
// Point to Arc instead of InfluxDB
client := influxdb2.NewClient("http://localhost:8000", "your-arc-token")
defer client.Close()
writeAPI := client.WriteAPIBlocking("myorg", "mydb")
// Write data - works exactly like InfluxDB
p := influxdb2.NewPoint(
"cpu",
map[string]string{"host": "server01"},
map[string]interface{}{"usage": 45.2},
time.Now(),
)
writeAPI.WritePoint(context.Background(), p)
}
```
### JavaScript/Node.js (@influxdata/influxdb-client) [#javascriptnodejs-influxdatainfluxdb-client]
```javascript
const { InfluxDB, Point } = require('@influxdata/influxdb-client');
// Point to Arc instead of InfluxDB
const client = new InfluxDB({
url: 'http://localhost:8000',
token: 'your-arc-token'
});
const writeApi = client.getWriteApi('myorg', 'mydb');
// Write data - works exactly like InfluxDB
const point = new Point('cpu')
.tag('host', 'server01')
.floatField('usage', 45.2);
writeApi.writePoint(point);
writeApi.close();
```
### Node-RED (node-red-contrib-influxdb) [#node-red-node-red-contrib-influxdb]
Configure the InfluxDB node with:
* **Version**: 2.0
* **URL**: `http://your-arc-host:8000`
* **Token**: Your Arc API token
* **Organization**: Any value (ignored by Arc)
* **Bucket**: Your Arc database name
The node will automatically use `/api/v2/write` which Arc supports natively.
### Telegraf [#telegraf]
```toml
[[outputs.influxdb_v2]]
urls = ["http://localhost:8000"]
token = "your-arc-token"
organization = "myorg"
bucket = "telegraf"
```
Or use the native Arc output plugin for better performance:
```toml
[[outputs.arc]]
url = "http://localhost:8000/api/v1/write/msgpack"
api_key = "your-arc-token"
content_encoding = "gzip"
database = "telegraf"
```
## Migration from InfluxDB [#migration-from-influxdb]
### Step 1: update connection URL [#step-1-update-connection-url]
Change your InfluxDB URL to point to Arc:
```python
# Before (InfluxDB)
client = InfluxDBClient(url="http://influxdb:8086", ...)
# After (Arc)
client = InfluxDBClient(url="http://arc:8000", ...)
```
### Step 2: use Arc token [#step-2-use-arc-token]
Replace your InfluxDB token with an Arc API token:
```bash
# Get token from Arc logs on first startup
docker logs arc 2>&1 | grep -i "admin"
```
### Step 3: map buckets to databases [#step-3-map-buckets-to-databases]
InfluxDB "buckets" map to Arc "databases":
| InfluxDB | Arc |
| ------------ | ----------- |
| Organization | Ignored |
| Bucket | Database |
| Measurement | Measurement |
### Step 4: verify connection [#step-4-verify-connection]
```bash
# Test write
curl -X POST "http://localhost:8000/api/v2/write?bucket=mydb&org=myorg" \
-H "Authorization: Token your-arc-token" \
-d 'test,host=server01 value=1'
# Query data
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer your-arc-token" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM mydb.test LIMIT 10", "format": "json"}'
```
## Connection pooling [#connection-pooling]
For high-throughput applications, enable HTTP connection pooling in your client. This reuses TCP connections instead of opening new ones for each request.
### Python [#python]
```python
import os
from influxdb_client import InfluxDBClient
import urllib3
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Enable connection pooling
http = urllib3.PoolManager(
num_pools=10,
maxsize=50,
retries=urllib3.Retry(3)
)
client = InfluxDBClient(
url="http://localhost:8000",
token=os.environ["ARC_TOKEN"],
org="myorg"
)
```
### Node.js [#nodejs]
```javascript
const { InfluxDB } = require('@influxdata/influxdb-client');
const http = require('http');
// Create agent with connection pooling
const agent = new http.Agent({
keepAlive: true,
maxSockets: 50,
maxFreeSockets: 10
});
const client = new InfluxDB({
url: 'http://localhost:8000',
token: process.env.ARC_TOKEN,
transportOptions: {
agent: agent
}
});
```
## Differences from InfluxDB [#differences-from-influxdb]
While Arc is compatible with InfluxDB clients, there are some differences:
| Feature | InfluxDB | Arc |
| ------------------ | -------------- | ----------------------- |
| Query Language | Flux, InfluxQL | Standard analytical SQL |
| Organizations | Supported | Ignored |
| Retention Policies | Per-bucket | Via retention API |
| Tasks | Built-in | Via continuous queries |
| Flux Functions | Full support | Not supported |
## Querying data [#querying-data]
Arc uses SQL instead of Flux or InfluxQL. Use the Arc query API:
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"sql": "SELECT time, host, usage FROM mydb.cpu WHERE time > NOW() - INTERVAL '\''1 hour'\'' ORDER BY time DESC LIMIT 100",
"format": "json"
}'
```
Or use the [Arc Python SDK](/arc/sdks/python/) for DataFrame support:
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
df = client.query.query_pandas(
"SELECT * FROM mydb.cpu WHERE time > NOW() - INTERVAL '1 hour'"
)
print(df.head())
```
## Troubleshooting [#troubleshooting]
### "404 Not Found" on /write [#404-not-found-on-write]
Ensure you're using Arc version 26.02.1 or later which includes the InfluxDB-compatible endpoints.
### Authentication errors [#authentication-errors]
Arc accepts tokens via:
* `Authorization: Bearer `
* `Authorization: Token `
* `?p=` query parameter
### "Organization not found" [#organization-not-found]
Arc ignores the organization parameter. Any value works.
### Data not appearing [#data-not-appearing]
1. Check the database exists or will be auto-created
2. Force a flush: `POST /api/v1/write/line-protocol/flush`
3. Verify with: `SELECT * FROM mydb.measurement LIMIT 1`
## Next steps [#next-steps]
* **[Python SDK](/arc/sdks/python/)** - Native Arc client with DataFrame support
* **[Telegraf Integration](/arc/integrations/telegraf/)** - Native Arc output plugin
* **[API Reference](/arc/api-reference/overview/)** - Full endpoint documentation
# MQTT Integration (/arc/integrations/mqtt)
Ingest data directly from MQTT brokers into Arc. Connect to IoT devices, industrial sensors, and message brokers without middleware.
MQTT integration with API-driven subscription management is available starting Arc v26.02.1 (February 2026).
## Overview [#overview]
Arc provides native MQTT subscription with dynamic, API-driven configuration. Manage multiple MQTT brokers and subscriptions at runtime without server restarts.
**Key features:**
* **API-driven subscription management** - Create, update, delete, start/stop subscriptions via REST API
* **Multiple simultaneous brokers** - Connect to different MQTT brokers for different data sources
* **Topic wildcards** - Subscribe using `+` (single level) and `#` (multi-level) wildcards
* **Auto-detection** - Automatically detects JSON and MessagePack message formats
* **High performance** - MessagePack columnar format
* **Topic mapping** - Extract tags from topic path segments
* **TLS/SSL support** - Client certificates and CA verification
* **Encrypted credentials** - Passwords encrypted at rest using AES-256-GCM
* **Auto-reconnect** - Exponential backoff on connection loss
* **QoS support** - QoS 0, 1, and 2
## Prerequisites [#prerequisites]
* Arc server running (v26.02.1 or higher)
* Arc API token (if authentication is enabled)
* MQTT broker accessible from Arc server
## Quick start [#quick-start]
### 1. Enable MQTT in Arc [#1-enable-mqtt-in-arc]
MQTT is **disabled by default**. Enable it via either:
**Option A — environment variable (recommended for Docker / Kubernetes):**
```ini
ARC_MQTT_ENABLED=true
```
In `docker-compose.yml`:
```yaml
services:
arc:
environment:
ARC_MQTT_ENABLED: "true"
```
Or with `docker run`:
```bash
docker run -e ARC_MQTT_ENABLED=true ...
```
**Option B — `arc.toml`:**
```toml
[mqtt]
enabled = true
```
Arc looks for `arc.toml` in `.`, `/etc/arc/`, and `$HOME/.arc/`. In containers, editing the file inside a running container does **not** persist across restarts — use the env-var path or mount the file as a volume.
Restart Arc after changing either source. Verify with:
```bash
curl http://localhost:8000/api/v1/mqtt/health
```
| Response | Meaning |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `200` with `{"status":"healthy", ...}` | MQTT is enabled and running. |
| `200` with `{"status":"disabled","healthy":false}` | MQTT was not enabled at startup. Check `ARC_MQTT_ENABLED` / `[mqtt].enabled` and restart. |
| `503` with `{"error":"MQTT subsystem disabled"}` | Same as above, on older builds. |
| Plaintext `Cannot GET /api/v1/mqtt/health` | Your Arc build pre-dates the MQTT API. Upgrade to a release that includes PR #416 (v26.05.1 or later). |
### 2. Create a subscription [#2-create-a-subscription]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"auto_start": true
}'
```
Response:
```json
{
"id": "sub_abc123",
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"status": "running",
"created_at": "2026-02-01T10:00:00Z"
}
```
### 3. Send test data [#3-send-test-data]
Publish a message to your MQTT broker:
```bash
mosquitto_pub -h localhost -t "sensors/temperature" \
-m '{"time": 1706745600000000, "value": 23.5, "device_id": "sensor-001"}'
```
### 4. Query the data [#4-query-the-data]
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"sql": "SELECT * FROM iot.temperature ORDER BY time DESC LIMIT 10",
"format": "json"
}'
```
## REST API reference [#rest-api-reference]
### Subscription management [#subscription-management]
| Method | Endpoint | Description |
| -------- | --------------------------------- | ------------------------- |
| `POST` | `/api/v1/mqtt/subscriptions` | Create a new subscription |
| `GET` | `/api/v1/mqtt/subscriptions` | List all subscriptions |
| `GET` | `/api/v1/mqtt/subscriptions/{id}` | Get subscription details |
| `PUT` | `/api/v1/mqtt/subscriptions/{id}` | Update subscription |
| `DELETE` | `/api/v1/mqtt/subscriptions/{id}` | Delete subscription |
### Lifecycle control [#lifecycle-control]
| Method | Endpoint | Description |
| ------ | ----------------------------------------- | -------------------- |
| `POST` | `/api/v1/mqtt/subscriptions/{id}/start` | Start subscription |
| `POST` | `/api/v1/mqtt/subscriptions/{id}/stop` | Stop subscription |
| `POST` | `/api/v1/mqtt/subscriptions/{id}/restart` | Restart subscription |
### Monitoring [#monitoring]
| Method | Endpoint | Description |
| ------ | --------------------------------------- | ----------------------------------- |
| `GET` | `/api/v1/mqtt/subscriptions/{id}/stats` | Get subscription stats |
| `GET` | `/api/v1/mqtt/stats` | Aggregate stats (all subscriptions) |
| `GET` | `/api/v1/mqtt/health` | Health check |
## Subscription options [#subscription-options]
### Create subscription request [#create-subscription-request]
```json
{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#", "factory/+/metrics"],
"database": "iot",
"qos": 1,
"client_id": "arc-factory",
"username": "mqtt_user",
"password": "mqtt_pass",
"tls_enabled": false,
"tls_cert_path": "/path/to/client.crt",
"tls_key_path": "/path/to/client.key",
"tls_ca_path": "/path/to/ca.crt",
"topic_mapping": {},
"keep_alive_seconds": 60,
"connect_timeout_seconds": 30,
"reconnect_max_seconds": 60,
"auto_start": true
}
```
### Field reference [#field-reference]
| Field | Type | Required | Default | Description |
| ------------------------- | --------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | string | Yes | - | Unique subscription name |
| `broker` | string | Yes | - | MQTT broker URL (tcp\://, ssl://, ws\://, wss\://) |
| `topics` | array | Yes | - | List of topics to subscribe |
| `database` | string | Yes | - | Target Arc database |
| `qos` | int | No | 1 | QoS level: 0, 1, or 2 |
| `client_id` | string | No | auto | MQTT client ID |
| `username` | string | No | - | MQTT username |
| `password` | string | No | - | MQTT password (encrypted at rest) |
| `tls_enabled` | bool | No | false | Enable TLS/SSL |
| `tls_cert_path` | string | No | - | Client certificate path |
| `tls_key_path` | string | No | - | Client key path |
| `tls_ca_path` | string | No | - | CA certificate path |
| `topic_mapping` | object (`{string: string}`) | No | \{} | Per-topic target-database override: maps an exact MQTT topic to a database name, overriding `database` for messages on that topic. See [Topic Mapping](#topic-mapping-per-topic-database-override). |
| `keep_alive_seconds` | int | No | 60 | MQTT keep-alive interval |
| `connect_timeout_seconds` | int | No | 30 | Connection timeout |
| `reconnect_max_seconds` | int | No | 60 | Maximum reconnect backoff delay. The reconnect delay starts at 1 second and doubles up to this cap (the 1-second minimum is fixed by the MQTT client library and is not configurable). |
| `auto_start` | bool | No | true | Start on creation and server restart |
## Message formats [#message-formats]
Arc automatically detects the message format based on content.
### JSON single record [#json-single-record]
```json
{
"time": 1706745600000000,
"temperature": 23.5,
"humidity": 65.2,
"device_id": "sensor-001"
}
```
### JSON batch [#json-batch]
```json
[
{"time": 1706745600000000, "temperature": 23.5},
{"time": 1706745601000000, "temperature": 23.6},
{"time": 1706745602000000, "temperature": 23.4}
]
```
### MessagePack row-based [#messagepack-row-based]
Same structure as JSON, but MessagePack encoded. Detected via magic bytes.
### MessagePack columnar (fastest) [#messagepack-columnar-fastest]
```json
{
"m": "temperature",
"columns": {
"time": [1706745600000000, 1706745601000000],
"value": [23.5, 23.6],
"device_id": ["sensor-001", "sensor-001"]
}
}
```
**Performance:** Uses the MessagePack columnar format for high-throughput ingestion.
### Timestamp handling [#timestamp-handling]
* If `time` field is present: used as-is (auto-detects milliseconds/microseconds/nanoseconds)
* If `time` field is missing: current UTC time is used
## Measurement, tags, and fields [#measurement-tags-and-fields]
Arc derives the measurement, tags, and fields **from the message payload**, not from the topic structure. The topic itself is not parsed for the measurement name or for tag values.
For each decoded message:
* **Measurement** — taken from the payload's `m` field, or `measurement` field. If neither is present, it defaults to `mqtt`.
* **Tags** — taken from a `tags` object in the payload (string values).
* **Fields** — taken from a `fields` object if present; otherwise every remaining top-level key (anything other than `m`/`measurement`, `t`/`time`/`timestamp`, `tags`, `fields`) is treated as a field.
* **Timestamp** — from `t`, `time`, or `timestamp` (auto-detects ms/µs/ns); current UTC time if absent.
So to land in measurement `machine_metrics` with tags `line` and `machine_id`, publish a payload like:
```json
{
"m": "machine_metrics",
"time": 1706745600000000,
"tags": { "line": "A", "machine_id": "42" },
"fields": { "temperature": 71.5, "rpm": 1480 }
}
```
A flat payload with no `m`/`tags`/`fields` (e.g. `{"time": ..., "temperature": 23.5}`) is also accepted: it lands in the default `mqtt` measurement with the remaining keys as fields and no tags.
Deriving the measurement or tags from topic path segments (e.g. `tags_from_topic` / positional extraction) is **not** currently supported. Set the measurement and tags in the published payload as shown above.
## Topic mapping (per-topic database override) [#topic-mapping-per-topic-database-override]
`topic_mapping` maps an **exact MQTT topic string to a target database name**, overriding the subscription's `database` for messages received on that topic. It is a flat `{ "": "" }` object — it does not configure measurements or tags.
```json
{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["factory/line1/metrics", "factory/line2/metrics"],
"database": "iot",
"topic_mapping": {
"factory/line2/metrics": "iot_line2"
}
}
```
In this example, messages on `factory/line1/metrics` are written to the default `iot` database, while messages on `factory/line2/metrics` are routed to `iot_line2`.
The mapping key is matched against the message's actual topic by exact string equality — wildcard topic patterns (`+`, `#`) are not expanded for matching. A subscription may use wildcards in `topics`, but `topic_mapping` keys must be the concrete topics you want to route to a different database.
## Authentication [#authentication]
### Basic authentication [#basic-authentication]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-d '{
"name": "authenticated-broker",
"broker": "tcp://broker.example.com:1883",
"topics": ["data/#"],
"database": "production",
"username": "mqtt_user",
"password": "mqtt_password"
}'
```
### Password encryption [#password-encryption]
Passwords are encrypted at rest using AES-256-GCM. Set the encryption key:
```bash
# Generate a 32-byte key
openssl rand -base64 32
# Set environment variable before starting Arc
export ARC_ENCRYPTION_KEY="your-base64-encoded-32-byte-key"
```
**Note:** The encryption key is only required when subscriptions have passwords. Subscriptions without credentials work without the key.
## TLS/SSL configuration [#tlsssl-configuration]
### Server certificate verification [#server-certificate-verification]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-d '{
"name": "secure-broker",
"broker": "ssl://broker.example.com:8883",
"topics": ["secure/#"],
"database": "production",
"tls_enabled": true,
"tls_ca_path": "/etc/arc/certs/ca.crt"
}'
```
### Client certificate authentication [#client-certificate-authentication]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-d '{
"name": "mtls-broker",
"broker": "ssl://broker.example.com:8883",
"topics": ["secure/#"],
"database": "production",
"tls_enabled": true,
"tls_cert_path": "/etc/arc/certs/client.crt",
"tls_key_path": "/etc/arc/certs/client.key",
"tls_ca_path": "/etc/arc/certs/ca.crt"
}'
```
## Configuration examples [#configuration-examples]
### Multiple brokers [#multiple-brokers]
Connect to different brokers for different environments:
```bash
# Production broker
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-d '{
"name": "production",
"broker": "ssl://prod-mqtt.example.com:8883",
"topics": ["prod/#"],
"database": "production",
"tls_enabled": true
}'
# Development broker
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-d '{
"name": "development",
"broker": "tcp://dev-mqtt.example.com:1883",
"topics": ["dev/#"],
"database": "development"
}'
```
### IoT sensor network [#iot-sensor-network]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-d '{
"name": "iot-sensors",
"broker": "tcp://mosquitto:1883",
"topics": [
"sensors/+/temperature",
"sensors/+/humidity",
"sensors/+/pressure"
],
"database": "iot",
"qos": 1
}'
```
Devices set the measurement and tags in the payload — e.g. a temperature sensor publishes:
```json
{ "m": "temperature", "tags": { "sensor_id": "temp-001" }, "fields": { "value": 23.5 } }
```
### Industrial factory [#industrial-factory]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-d '{
"name": "factory-floor",
"broker": "tcp://factory-mqtt:1883",
"topics": ["factory/+/+/metrics"],
"database": "manufacturing",
"qos": 2
}'
```
Machines publish the measurement and tags in the payload:
```json
{
"m": "machine_metrics",
"tags": { "line": "A", "machine_id": "42" },
"fields": { "temperature": 71.5, "rpm": 1480 }
}
```
## Monitoring [#monitoring-1]
### Subscription stats [#subscription-stats]
```bash
# Stats for a specific subscription
curl http://localhost:8000/api/v1/mqtt/subscriptions/{id}/stats
# Aggregate stats for all subscriptions
curl http://localhost:8000/api/v1/mqtt/stats
```
Response:
```json
{
"status": "success",
"running_count": 2,
"subscriptions_stats": {
"sub_abc123": {
"messages_received": 15420,
"bytes_received": 2458320,
"decode_errors": 0,
"last_message_at": "2026-02-01T10:30:15Z",
"topics": {
"sensors/temperature": 8500,
"sensors/humidity": 6920
}
}
}
}
```
`last_message_at` and `connected_since` are **omitted** when the subscription has not yet received a message / is not connected (rather than reporting a zero timestamp). When present they are always UTC.
### Health check [#health-check]
```bash
curl http://localhost:8000/api/v1/mqtt/health
```
Response:
```json
{
"status": "healthy",
"healthy": true,
"running_count": 2,
"connected_count": 2,
"disconnected_count": 0,
"service": "mqtt_subscriptions"
}
```
### Prometheus metrics [#prometheus-metrics]
Arc exposes MQTT metrics for Prometheus:
| Metric | Type | Description |
| ---------------------------------- | ------- | ------------------------------- |
| `arc_mqtt_messages_received_total` | Counter | Total messages received |
| `arc_mqtt_bytes_received_total` | Counter | Total bytes received |
| `arc_mqtt_decode_errors_total` | Counter | Message decode errors |
| `arc_mqtt_connection_status` | Gauge | Connection status (1=connected) |
## Querying MQTT data [#querying-mqtt-data]
### List measurements [#list-measurements]
```sql
SHOW TABLES FROM iot;
```
### Basic query [#basic-query]
```sql
SELECT * FROM iot.temperature
ORDER BY time DESC
LIMIT 10;
```
### Time-based aggregation [#time-based-aggregation]
```sql
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(value) as avg_temp,
MIN(value) as min_temp,
MAX(value) as max_temp
FROM iot.temperature
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;
```
### Filter by tag [#filter-by-tag]
```sql
SELECT * FROM iot.sensor_data
WHERE sensor_id = 'temp-001'
AND time > NOW() - INTERVAL '24 hours'
ORDER BY time DESC;
```
### Cross-measurement join [#cross-measurement-join]
```sql
SELECT
t.time,
t.value as temperature,
h.value as humidity
FROM iot.temperature t
JOIN iot.humidity h ON t.time = h.time AND t.sensor_id = h.sensor_id
WHERE t.time > NOW() - INTERVAL '1 hour'
ORDER BY t.time DESC;
```
## Troubleshooting [#troubleshooting]
### Connection failed [#connection-failed]
```bash
# Check subscription status
curl http://localhost:8000/api/v1/mqtt/subscriptions/{id}
```
If status is `error`, check:
* Broker URL is correct (tcp\://, ssl://, ws\://)
* Broker is reachable from Arc server
* Credentials are correct
* TLS certificates are valid
### No data appearing [#no-data-appearing]
1. Verify subscription is running:
```bash
curl http://localhost:8000/api/v1/mqtt/subscriptions/{id}
# status should be "running"
```
2. Check stats for received messages:
```bash
curl http://localhost:8000/api/v1/mqtt/subscriptions/{id}/stats
```
3. Verify topic pattern matches published topics
4. Check Arc logs for decode errors
### Messages not decoding [#messages-not-decoding]
Ensure messages are valid JSON or MessagePack:
```bash
# Test with simple JSON
mosquitto_pub -h localhost -t "test/data" \
-m '{"time": 1706745600000000, "value": 42}'
```
Check for decode errors in stats:
```bash
curl http://localhost:8000/api/v1/mqtt/subscriptions/{id}/stats | jq '.decode_errors'
```
### Subscription won't start [#subscription-wont-start]
Check for errors:
```bash
curl http://localhost:8000/api/v1/mqtt/subscriptions/{id} | jq '.error_message'
```
Common issues:
* Another client using same client\_id
* Invalid broker URL
* Network connectivity issues
## Best practices [#best-practices]
### 1. Use descriptive names [#1-use-descriptive-names]
```json
{
"name": "prod-factory-floor-sensors",
"broker": "ssl://prod-mqtt.example.com:8883"
}
```
### 2. Separate databases by environment [#2-separate-databases-by-environment]
```json
// Production
{"database": "production", "topics": ["prod/#"]}
// Staging
{"database": "staging", "topics": ["staging/#"]}
```
### 3. Use QoS appropriately [#3-use-qos-appropriately]
* **QoS 0**: Fire-and-forget, no guarantees (highest throughput)
* **QoS 1**: At least once delivery (recommended for most cases)
* **QoS 2**: Exactly once delivery (highest overhead)
### 4. Cap the reconnect backoff [#4-cap-the-reconnect-backoff]
Reconnect uses an exponential backoff that starts at 1 second (fixed by the MQTT
client library) and doubles up to `reconnect_max_seconds`. Set the cap to bound
how long the client waits between attempts when a broker is down for a while:
```json
{
"reconnect_max_seconds": 60
}
```
### 5. Use topic wildcards efficiently [#5-use-topic-wildcards-efficiently]
```bash
# Good - specific wildcards
sensors/+/temperature
factory/line1/+/metrics
# Avoid - too broad
#
sensors/#/#
```
### 6. Monitor subscription health [#6-monitor-subscription-health]
Set up alerts on:
* `arc_mqtt_connection_status == 0` (disconnected)
* `rate(arc_mqtt_decode_errors_total[5m]) > 0` (decode errors)
## Docker Compose example [#docker-compose-example]
```yaml
version: '3.8'
services:
arc:
image: basekick/arc:latest
ports:
- "8000:8000"
volumes:
- arc-data:/data
environment:
- ARC_ENCRYPTION_KEY=${ARC_ENCRYPTION_KEY}
depends_on:
- mosquitto
mosquitto:
image: eclipse-mosquitto:2
ports:
- "1883:1883"
volumes:
- ./mosquitto.conf:/mosquitto/config/mosquitto.conf
volumes:
arc-data:
```
**mosquitto.conf:**
```bash
listener 1883
allow_anonymous true
```
## Next steps [#next-steps]
* **[Query MQTT data](/arc/guides/querying/)** - Learn SQL analytics
* **[Create Grafana dashboards](/arc/integrations/grafana/)** - Visualize MQTT data
* **[Set up retention policies](/arc/data-lifecycle/retention-policies/)** - Manage data lifecycle
* **[Configure alerts](/arc/integrations/grafana/)** - Set up Grafana alerting on MQTT data
# OpenTelemetry Integration (/arc/integrations/opentelemetry)
Send traces, metrics, and logs from OpenTelemetry Collector to Arc for unified observability.
## Overview [#overview]
The Arc OpenTelemetry Exporter enables you to send all your telemetry data from the OpenTelemetry Collector to Arc:
* **✅ Traces**: Distributed traces with full span hierarchy
* **✅ Metrics**: All metric types (gauge, counter, histogram, summary)
* **✅ Logs**: Structured logs with attributes
* **🚀 High Performance**: Uses Arc's columnar MessagePack format
* **📦 Compression**: Automatic gzip compression
* **🔄 Retry Logic**: Configurable retry with exponential backoff
* **🔐 Authentication**: Bearer token support
**Performance:**
* Traces: 500K-1M spans/sec
* Metrics: 3M-6M data points/sec
* Logs: 1M-2M logs/sec
## Why OpenTelemetry + Arc? [#why-opentelemetry--arc]
Traditional observability requires 3+ separate systems:
* Jaeger for traces
* Prometheus for metrics
* Loki/Elasticsearch for logs
* **Manual correlation** between systems
**With Arc + OpenTelemetry:**
* ✅ All signals in one database
* ✅ Join traces, metrics, and logs in SQL
* ✅ No manual correlation needed
* ✅ Single query for complete context
* ✅ One storage backend to manage
This is **unified observability**.
## Installation [#installation]
### Option 1: OpenTelemetry Collector Builder (OCB) [#option-1-opentelemetry-collector-builder-ocb]
Add to your `builder-config.yaml`:
```yaml
exporters:
- gomod: github.com/basekick-labs/arc-opentelemetry-exporter v0.1.8
```
Build the collector:
```bash
ocb --config builder-config.yaml
```
### Option 2: pre-built binary [#option-2-pre-built-binary]
Download from the [releases page](https://github.com/basekick-labs/arc-opentelemetry-exporter/releases):
```bash
# Resolve the latest release tag, then download the matching linux-amd64 binary
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc-opentelemetry-exporter/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc-opentelemetry-exporter/releases/download/v${LATEST}/arc-exporter-${LATEST}-linux-amd64
# Make executable
chmod +x arc-exporter-${LATEST}-linux-amd64
# Run
./arc-exporter-${LATEST}-linux-amd64 --config=config.yaml
```
## Quick start [#quick-start]
### 1. Start Arc [#1-start-arc]
```bash
docker run -d -p 8000:8000 \
-e STORAGE_BACKEND=local \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
### 2. Get your API token [#2-get-your-api-token]
```bash
# Check logs for admin token
docker logs 2>&1 | grep "Admin token"
# Or create a new token
curl -X POST http://localhost:8000/api/v1/auth/tokens \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "otel-collector",
"description": "OpenTelemetry Collector access"
}'
```
### 3. Create collector configuration [#3-create-collector-configuration]
Create `otel-config.yaml`:
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1000
exporters:
arc:
endpoint: http://localhost:8000
auth_token: your-arc-token-here
# Recommended: Separate databases per signal type
traces_database: traces
metrics_database: metrics
logs_database: logs
# Optional: Custom measurement names
traces_measurement: distributed_traces
logs_measurement: logs
# Optional: HTTP settings
timeout: 30s
compression: gzip
# Optional: Retry configuration
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [arc]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [arc]
logs:
receivers: [otlp]
processors: [batch]
exporters: [arc]
```
### 4. Run OpenTelemetry Collector [#4-run-opentelemetry-collector]
```bash
./otelcol-arc-linux-amd64 --config=otel-config.yaml
```
### 5. Send telemetry data [#5-send-telemetry-data]
Your applications instrumented with OpenTelemetry SDKs will now send data to Arc!
**Example: Python Application**
```python
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# Configure tracer
trace.set_tracer_provider(TracerProvider())
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
# Create spans
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-operation"):
# Your code here
print("Trace sent to Arc via OTel Collector!")
```
## Configuration [#configuration]
### Database organization strategies [#database-organization-strategies]
#### Strategy 1: single database (simple) [#strategy-1-single-database-simple]
All signals in one database:
```yaml
exporters:
arc:
endpoint: http://localhost:8000
database: default
```
**Structure:**
```text
default/
├── distributed_traces
├── logs
├── system_cpu_usage
└── http_requests_total
```
**Pros:** Simple, easy cross-signal correlation
**Cons:** All data in one namespace
#### Strategy 2: database per signal (recommended) [#strategy-2-database-per-signal-recommended]
Separate databases for each signal type:
```yaml
exporters:
arc:
endpoint: http://localhost:8000
traces_database: traces
metrics_database: metrics
logs_database: logs
```
**Structure:**
```text
traces/
└── distributed_traces
metrics/
├── system_cpu_usage
├── system_memory_usage
└── http_requests_total
logs/
└── logs
```
**Pros:**
* Clean separation of concerns
* Independent retention policies per signal
* Independent scaling and storage backends
* Easier permission management
* Matches traditional observability architecture
**Cons:** Slightly more complex configuration
**✅ Recommended for production deployments.**
### Configuration options [#configuration-options]
| Option | Description | Default |
| ----------------------------------- | ------------------------------- | -------------------- |
| `endpoint` | Arc API endpoint | Required |
| `auth_token` | Authentication token | Optional |
| `database` | Single database for all signals | `default` |
| `traces_database` | Database for traces | - |
| `metrics_database` | Database for metrics | - |
| `logs_database` | Database for logs | - |
| `traces_measurement` | Table name for traces | `distributed_traces` |
| `logs_measurement` | Table name for logs | `logs` |
| `timeout` | HTTP request timeout | `30s` |
| `compression` | Compression type | `gzip` |
| `retry_on_failure.enabled` | Enable retry logic | `true` |
| `retry_on_failure.initial_interval` | Initial retry interval | `5s` |
| `retry_on_failure.max_interval` | Maximum retry interval | `30s` |
| `retry_on_failure.max_elapsed_time` | Maximum total retry time | `300s` |
## Data format [#data-format]
The exporter uses Arc's high-performance **columnar MessagePack format** with **dynamic columns**. All OpenTelemetry attributes automatically become individual columns for optimal query performance.
### Traces [#traces]
All span attributes and resource attributes become columns:
```json
{
"m": "distributed_traces",
"columns": {
"time": [1699900000000],
"trace_id": ["5b8efff798038103d269b633813fc60c"],
"span_id": ["def456..."],
"parent_span_id": ["ghi789..."],
"service_name": ["api-gateway"],
"operation_name": ["HTTP GET /users"],
"span_kind": ["server"],
"duration_ns": [1234567],
"status_code": [0],
"http.method": ["GET"],
"http.status_code": [200],
"http.url": ["/api/users"],
"host.name": ["server-1"]
}
}
```
**Dynamic schema**: Columns created automatically from span and resource attributes.
### Metrics [#metrics]
Each metric name becomes its own table. All attributes become columns:
```json
{
"m": "http_requests_total",
"columns": {
"time": [1699900000000],
"value": [42.0],
"service": ["api"],
"method": ["GET"],
"status": ["200"],
"host.name": ["server-1"]
}
}
```
**Metric name sanitization:**
* `system.cpu.usage` → `system_cpu_usage`
* `http.server.duration` → `http_server_duration`
* `process-memory-bytes` → `process_memory_bytes`
### Logs [#logs]
All log attributes and resource attributes become columns:
```json
{
"m": "logs",
"columns": {
"time": [1699900000000],
"severity": ["ERROR"],
"severity_number": [17],
"body": ["Database connection failed"],
"trace_id": ["abc123..."],
"span_id": ["def456..."],
"service_name": ["api-gateway"],
"http.method": ["POST"],
"user_id": ["12345"],
"host.name": ["server-1"]
}
}
```
## Querying data [#querying-data]
### Traces [#traces-1]
```sql
-- Recent traces
SELECT * FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '1 hour'
LIMIT 100;
-- Traces by service
SELECT
service_name,
operation_name,
duration_ns / 1000000 AS duration_ms,
"http.method",
"http.status_code"
FROM traces.distributed_traces
WHERE service_name = 'api-gateway'
AND time > NOW() - INTERVAL '1 hour'
ORDER BY time DESC;
-- Debug specific trace
SELECT * FROM traces.distributed_traces
WHERE trace_id = '5b8efff798038103d269b633813fc60c'
ORDER BY time;
-- Slow requests (p99 latency)
SELECT
service_name,
operation_name,
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY duration_ns) / 1000000 AS p99_ms
FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY service_name, operation_name
ORDER BY p99_ms DESC;
```
### Metrics [#metrics-1]
Each metric is its own table with all attributes as columns:
```sql
-- CPU usage
SELECT
time,
value,
"host.name",
cpu,
state
FROM metrics.system_cpu_usage
WHERE time > NOW() - INTERVAL '1 hour'
AND "host.name" = 'server-1'
ORDER BY time DESC;
-- HTTP requests by method and status
SELECT
time_bucket(INTERVAL '1 minute', time) AS minute,
method,
status,
SUM(value) AS total_requests
FROM metrics.http_requests_total
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY minute, method, status
ORDER BY minute DESC;
-- Memory usage aggregated
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
"host.name",
AVG(value) AS avg_memory_bytes
FROM metrics.system_memory_usage
WHERE time > NOW() - INTERVAL '6 hours'
GROUP BY bucket, "host.name"
ORDER BY bucket DESC;
```
### Logs [#logs-1]
All attributes are individual columns for fast filtering:
```sql
-- Recent error logs
SELECT
time,
severity,
body,
service_name,
"host.name",
trace_id
FROM logs.logs
WHERE severity IN ('ERROR', 'FATAL')
AND time > NOW() - INTERVAL '1 hour'
ORDER BY time DESC;
-- Logs for specific trace (correlation)
SELECT
time,
severity,
body,
service_name
FROM logs.logs
WHERE trace_id = '5b8efff798038103d269b633813fc60c'
ORDER BY time;
-- Count errors by service
SELECT
service_name,
"host.name",
COUNT(*) AS error_count
FROM logs.logs
WHERE severity IN ('ERROR', 'FATAL')
AND time > NOW() - INTERVAL '1 hour'
GROUP BY service_name, "host.name"
ORDER BY error_count DESC;
```
## Unified observability: Join across signals [#unified-observability-join-across-signals]
Arc's most powerful feature: **correlate traces, metrics, and logs in a single SQL query**.
### Example 1: failed requests with full context [#example-1-failed-requests-with-full-context]
Get traces, error logs, and CPU metrics for failed requests:
```sql
SELECT
t.time,
t.trace_id,
t.service_name,
t.operation_name,
t.duration_ns / 1000000 AS duration_ms,
t."http.status_code",
l.severity,
l.body AS error_message,
cpu.value AS cpu_usage
FROM traces.distributed_traces t
LEFT JOIN logs.logs l
ON t.trace_id = l.trace_id
LEFT JOIN metrics.system_cpu_usage cpu
ON t.service_name = cpu.service_name
AND time_bucket(INTERVAL '1 minute', t.time) = time_bucket(INTERVAL '1 minute', cpu.time)
WHERE t.status_code >= 2 -- OTel status: 2 = Error
AND t.time > NOW() - INTERVAL '1 hour'
ORDER BY t.time DESC
LIMIT 100;
```
**Result:** Traces + error logs + CPU usage at time of failure — all in one query!
### Example 2: service health dashboard [#example-2-service-health-dashboard]
Complete service health metrics:
```sql
WITH trace_stats AS (
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
service_name,
COUNT(*) AS request_count,
AVG(duration_ns / 1000000) AS avg_latency_ms,
SUM(CASE WHEN status_code >= 2 THEN 1 ELSE 0 END) AS error_count
FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, service_name
),
error_logs AS (
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
service_name,
COUNT(*) AS log_error_count
FROM logs.logs
WHERE severity IN ('ERROR', 'FATAL')
AND time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, service_name
),
cpu_stats AS (
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
service_name,
AVG(value) AS avg_cpu
FROM metrics.system_cpu_usage
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, service_name
)
SELECT
ts.bucket AS time,
ts.service_name,
ts.request_count,
ROUND(ts.avg_latency_ms, 2) AS avg_latency_ms,
ts.error_count,
ROUND((ts.error_count::float / NULLIF(ts.request_count, 0) * 100), 2) AS error_rate_pct,
el.log_error_count,
ROUND(cs.avg_cpu, 2) AS avg_cpu_usage
FROM trace_stats ts
LEFT JOIN error_logs el ON ts.bucket = el.bucket AND ts.service_name = el.service_name
LEFT JOIN cpu_stats cs ON ts.bucket = cs.bucket AND ts.service_name = cs.service_name
ORDER BY ts.bucket DESC, ts.service_name;
```
**Result:**
* Request volume and latency (traces)
* Error rate (traces)
* Error log count (logs)
* CPU usage (metrics)
All from one database, in one query!
### Example 3: debug incident timeline [#example-3-debug-incident-timeline]
Unified timeline of traces and logs for a single request:
```sql
SELECT
t.time,
'trace' AS signal_type,
t.operation_name AS event,
t.duration_ns / 1000000 AS duration_ms,
t.status_code,
NULL AS severity,
NULL AS body
FROM traces.distributed_traces t
WHERE t.trace_id = '5b8efff798038103d269b633813fc60c'
UNION ALL
SELECT
l.time,
'log' AS signal_type,
l.service_name AS event,
NULL AS duration_ms,
NULL AS status_code,
l.severity,
l.body
FROM logs.logs l
WHERE l.trace_id = '5b8efff798038103d269b633813fc60c'
ORDER BY time;
```
**Result:** Complete chronological view of all events for a single request!
## Use cases [#use-cases]
### Application performance monitoring [#application-performance-monitoring]
Monitor service latency, error rates, and throughput:
```sql
SELECT
time_bucket(INTERVAL '1 minute', time) AS minute,
service_name,
COUNT(*) AS requests,
AVG(duration_ns / 1000000) AS avg_latency_ms,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ns) / 1000000 AS p95_ms,
SUM(CASE WHEN status_code >= 2 THEN 1 ELSE 0 END) AS errors
FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY minute, service_name
ORDER BY minute DESC;
```
### Distributed tracing [#distributed-tracing]
Analyze request flows across microservices:
```sql
-- Trace all spans in a distributed transaction
SELECT
span_id,
parent_span_id,
service_name,
operation_name,
duration_ns / 1000000 AS duration_ms,
"http.method",
"http.url"
FROM traces.distributed_traces
WHERE trace_id = 'your-trace-id'
ORDER BY time;
```
### Log analysis [#log-analysis]
Search and analyze structured logs:
```sql
-- Find all errors from a specific user session
SELECT
time,
severity,
body,
service_name,
user_id,
"http.method",
"http.url"
FROM logs.logs
WHERE user_id = '12345'
AND severity IN ('ERROR', 'WARN')
AND time > NOW() - INTERVAL '24 hours'
ORDER BY time DESC;
```
## Performance optimization [#performance-optimization]
### 1. Use batch processor [#1-use-batch-processor]
Always use the `batch` processor for high throughput:
```yaml
processors:
batch:
timeout: 1s
send_batch_size: 1000 # Adjust based on your load
```
### 2. Configure retry logic [#2-configure-retry-logic]
Handle transient failures:
```yaml
exporters:
arc:
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
```
### 3. Enable compression [#3-enable-compression]
Reduce network bandwidth:
```yaml
exporters:
arc:
compression: gzip # Default and recommended
```
### 4. Tune collector resources [#4-tune-collector-resources]
For high-volume deployments:
```yaml
service:
telemetry:
metrics:
level: detailed
extensions: [health_check]
```
### 5. Use separate databases [#5-use-separate-databases]
For production, use database-per-signal strategy:
```yaml
exporters:
arc:
traces_database: traces
metrics_database: metrics
logs_database: logs
```
## Troubleshooting [#troubleshooting]
### Collector not sending data [#collector-not-sending-data]
```bash
# Check collector logs
./otelcol-arc-linux-amd64 --config=config.yaml
# Verify Arc is accessible
curl http://localhost:8000/health
# Test authentication
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/auth/verify
```
### High memory usage [#high-memory-usage]
Reduce batch size:
```yaml
processors:
batch:
timeout: 1s
send_batch_size: 500 # Reduce from 1000
```
### Data not appearing in Arc [#data-not-appearing-in-arc]
```sql
-- Check if data is being written
SHOW TABLES FROM traces;
SHOW TABLES FROM metrics;
SHOW TABLES FROM logs;
-- Verify recent data
SELECT COUNT(*) FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '5 minutes';
```
### Connection timeouts [#connection-timeouts]
Increase timeout:
```yaml
exporters:
arc:
timeout: 60s # Increase from 30s
```
## Resources [#resources]
* **[Arc OpenTelemetry Exporter GitHub](https://github.com/basekick-labs/arc-opentelemetry-exporter)**
* **[OpenTelemetry Collector Docs](https://opentelemetry.io/docs/collector/)**
* **[Arc Query API](/arc/api-reference/overview/)**
* **[OpenTelemetry SDK Documentation](https://opentelemetry.io/docs/instrumentation/)**
## Next steps [#next-steps]
* **[Getting Started with Arc](/arc/getting-started/)** - Install Arc
* **[Grafana Integration](/arc/integrations/grafana/)** - Visualize OpenTelemetry data
* **[Query API Reference](/arc/api-reference/overview/)** - Learn Arc SQL
* **[Data Lifecycle](/arc/data-lifecycle/retention-policies/)** - Manage retention policies
***
**Ready for unified observability!**
Made with ❤️ by [Basekick Labs](https://github.com/basekick-labs)
# Redpanda Connect Integration (/arc/integrations/redpanda-connect)
Stream data from any of Redpanda Connect's 200+ sources directly into Arc using the native Arc output plugin.
## Overview [#overview]
[Redpanda Connect](https://github.com/redpanda-data/connect) (formerly Benthos) is a stream processor that connects sources to sinks with a single YAML config file. It handles transformations, filtering, batching, retries, and backpressure out of the box. Arc has a native output plugin that speaks Arc's MessagePack ingestion protocol directly, so data flows from your source into Arc's columnar storage with no translation layer.
**Benefits:**
* Native MessagePack columnar format with zstd compression
* 200+ input connectors (Kafka, HTTP, MQTT, S3, GCS, Postgres CDC, etc.)
* Bloblang transformations for reshaping, filtering, and enriching data in-flight
* Interpolated measurement names for per-message routing to different Arc tables
* Single binary, no JVM, no cluster required
## Why this matters [#why-this-matters]
Arc already has native ingestion paths for metrics ([Telegraf](/arc/integrations/telegraf/)) and IoT data ([MQTT](/arc/integrations/mqtt/)). Redpanda Connect covers a different gap: event-driven data that needs reshaping, filtering, or enrichment before it lands in Arc.
| Tool | Best For |
| ---------------- | ------------------------------------------------------------------------ |
| Telegraf | Pulling metrics from systems at fixed intervals |
| Native MQTT | Subscribing to IoT brokers directly |
| Redpanda Connect | Event streams, CDC, webhooks, complex transformations, fan-out pipelines |
Some concrete examples where Redpanda Connect fits:
* **Kafka to Arc** — consume events, filter out bot traffic, normalize timestamps, write to Arc
* **Webhooks to Arc** — receive HTTP webhooks from third-party APIs, reshape the payload, store for analytics
* **CDC to Arc** — capture Postgres/MySQL change events and stream them into Arc for historical tracking
* **Multi-destination** — send the same data to Arc and Kafka (or S3, or Elasticsearch) with different transformations per sink
## Prerequisites [#prerequisites]
* **Redpanda Connect 4.88 or higher** (required for the `arc` output)
* Arc server running and accessible
* Arc API token (if auth is enabled)
## Quick start [#quick-start]
### 1. Install Redpanda Connect [#1-install-redpanda-connect]
```bash
# Homebrew (macOS/Linux)
brew install redpanda-data/tap/redpanda-connect
# Docker
docker run --rm -v $(pwd)/config.yaml:/config.yaml \
docker.redpanda.com/redpandadata/connect:latest run /config.yaml
# Direct binary download
# https://github.com/redpanda-data/connect/releases
```
Verify you have 4.88+:
```bash
redpanda-connect --version
```
### 2. Create a pipeline config [#2-create-a-pipeline-config]
Create `arc-pipeline.yaml`:
```yaml
input:
generate:
count: 10
interval: 1s
mapping: |
root.vehicle_id = "truck-" + random_int(min: 1, max: 5).string()
root.lat = 40.7128 + (random_int(min: -1000, max: 1000).number() / 10000)
root.lon = -74.0060 + (random_int(min: -1000, max: 1000).number() / 10000)
root.speed_kmh = random_int(min: 0, max: 120)
output:
arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: logistics
measurement: fleet_tracking
format: columnar
compression: zstd
batching:
count: 100
period: 1s
```
### 3. Run the pipeline [#3-run-the-pipeline]
```bash
export ARC_TOKEN="your-arc-token"
redpanda-connect run arc-pipeline.yaml
```
Expected output:
```bash
INFO Running main config from specified file path=arc-pipeline.yaml
INFO Input type generate is now active
INFO Output type arc is now active
INFO Pipeline has terminated. Shutting down the service
```
### 4. Verify data in Arc [#4-verify-data-in-arc]
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT vehicle_id, speed_kmh FROM logistics.fleet_tracking ORDER BY time DESC LIMIT 10"}'
```
## Configuration reference [#configuration-reference]
| Option | Description | Default |
| ----------------- | --------------------------------------------------------- | ------------------------- |
| `base_url` | Base URL of the Arc instance | Required |
| `token` | Bearer token for authentication | Optional |
| `database` | Target database in Arc | `default` |
| `measurement` | Measurement (table) name, supports interpolation | Required |
| `format` | Payload format: `columnar` or `row` | `columnar` |
| `compression` | Compression: `zstd`, `gzip`, or `none` | `zstd` |
| `timestamp_field` | Field name in the message containing the timestamp | empty (uses current time) |
| `timestamp_unit` | Unit of numeric timestamps: `us`, `ms`, `s`, `ns`, `auto` | `auto` |
| `tags_mapping` | Bloblang mapping to extract tags (row format only) | Optional |
| `tls` | TLS configuration | Optional |
| `batching` | Batch policy (`count`, `period`, `byte_size`) | None |
| `max_in_flight` | Maximum parallel batches | `64` |
| `timeout` | HTTP request timeout | `5s` |
## Payload formats [#payload-formats]
### Columnar (default, recommended) [#columnar-default-recommended]
Transposes batched messages into column arrays. This is Arc's fastest ingestion path because it maps directly to Arc's Arrow buffers and avoids per-row overhead.
```yaml
output:
arc:
base_url: http://localhost:8000
database: logistics
measurement: fleet_tracking
format: columnar
compression: zstd
```
Requirement: all messages within a single batch must have the same set of fields. Arc validates this server-side and rejects batches with mismatched columns. Schema evolution across separate batches is fully supported.
### Row [#row]
Sends each message as an individual record with fields and optional tags. Useful when messages within a batch have varying schemas, or when you need per-message tags.
```yaml
output:
arc:
base_url: http://localhost:8000
database: logistics
measurement: fleet_tracking
format: row
tags_mapping: |
root = {"vehicle_id": this.vehicle_id, "fleet": this.fleet, "region": this.region}
```
## Real-world examples [#real-world-examples]
### Kafka events to Arc [#kafka-events-to-arc]
Consume JSON events from a Kafka topic, drop bot traffic, reshape fields, and normalize the timestamp:
```yaml
input:
kafka:
addresses: ["kafka:9092"]
topics: ["app-events"]
consumer_group: "arc-analytics"
pipeline:
processors:
- mapping: |
# Drop bot traffic
root = if this.user_id.has_prefix("bot-") { deleted() }
# Reshape the fields we care about
root.user_id = this.user_id
root.page = this.page
root.duration_ms = this.duration_ms
root.event_type = this.event
output:
arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: analytics
measurement: page_views
format: columnar
timestamp_field: timestamp
timestamp_unit: ms
compression: zstd
batching:
count: 5000
period: 5s
```
### HTTP webhooks to Arc [#http-webhooks-to-arc]
Expose an HTTP endpoint that receives webhooks and writes them to Arc:
```yaml
input:
http_server:
address: "0.0.0.0:8080"
path: /webhook
pipeline:
processors:
- mapping: |
root.source = meta("Http_Header_X_Webhook_Source")
root.received_at = now()
root.payload = this
output:
arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: webhooks
measurement: "${!metadata(\"Http_Header_X_Webhook_Source\")}"
format: row
compression: zstd
batching:
count: 100
period: 2s
```
### MQTT to Arc with transformations [#mqtt-to-arc-with-transformations]
When you want Redpanda Connect's transformation power on top of MQTT (instead of the native MQTT ingestion):
```yaml
input:
mqtt:
urls: ["tcp://broker.example.com:1883"]
topics: ["sensors/#"]
client_id: "arc-connect"
pipeline:
processors:
- mapping: |
root.device_id = meta("mqtt_topic").split("/").index(1)
root.reading = this.value
root.temperature_c = (this.value - 32) * 5 / 9
output:
arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: sensors
measurement: readings
format: columnar
compression: zstd
batching:
count: 1000
period: 1s
```
### Multi-destination fan-out [#multi-destination-fan-out]
Send the same events to Arc and Kafka simultaneously:
```yaml
output:
broker:
pattern: fan_out
outputs:
- arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: events
measurement: user_actions
format: columnar
- kafka:
addresses: ["kafka:9092"]
topic: processed-events
```
## Dynamic measurement routing [#dynamic-measurement-routing]
The `measurement` field supports Redpanda Connect's Bloblang interpolation. Messages with different types can be routed to different Arc tables in a single pipeline:
```yaml
output:
arc:
base_url: http://localhost:8000
database: telemetry
# Messages with {"asset_type": "truck", ...} go to the "truck" table
# Messages with {"asset_type": "drone", ...} go to the "drone" table
measurement: ${!json("asset_type")}
```
Or route from message metadata (e.g., from Kafka headers, HTTP headers, or MQTT topics):
```yaml
output:
arc:
base_url: http://localhost:8000
database: telemetry
measurement: ${!metadata("measurement")}
```
## Bloblang transformations [#bloblang-transformations]
[Bloblang](https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/) is Redpanda Connect's built-in mapping language. A few patterns that come up when writing to Arc:
### Drop messages conditionally [#drop-messages-conditionally]
```yaml
processors:
- mapping: |
root = if this.value == null { deleted() }
```
### Flatten nested structures [#flatten-nested-structures]
```yaml
processors:
- mapping: |
root.device_id = this.device.id
root.device_model = this.device.model
root.reading = this.payload.reading
```
### Parse timestamps from strings [#parse-timestamps-from-strings]
```yaml
processors:
- mapping: |
root.event_time = this.timestamp.ts_parse("2006-01-02T15:04:05Z")
root.event_name = this.event
```
### Enrich with static or derived fields [#enrich-with-static-or-derived-fields]
```yaml
processors:
- mapping: |
root = this
root.region = env("DEPLOY_REGION")
root.ingested_at = now()
```
## Querying the data [#querying-the-data]
Once data is in Arc, query it with standard SQL:
```sql
-- Latest records per vehicle
SELECT vehicle_id, lat, lon, speed_kmh, time
FROM logistics.fleet_tracking
WHERE time > NOW() - INTERVAL '1 hour'
ORDER BY time DESC
LIMIT 100;
-- Average speed by vehicle over the last 24h
SELECT
vehicle_id,
AVG(speed_kmh) as avg_speed,
MAX(speed_kmh) as max_speed,
COUNT(*) as reading_count
FROM logistics.fleet_tracking
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY vehicle_id
ORDER BY avg_speed DESC;
-- Hourly throughput of ingested events
SELECT
time_bucket(INTERVAL '1 hour', time) as hour,
COUNT(*) as events
FROM analytics.page_views
WHERE time > NOW() - INTERVAL '7 days'
GROUP BY hour
ORDER BY hour DESC;
```
## Performance tuning [#performance-tuning]
### Batch size [#batch-size]
Arc's columnar format is significantly more efficient with larger batches. Tune `batching.count` and `batching.period` based on your volume.
| Volume | Recommended `batching.count` |
| --------------------- | ---------------------------- |
| Low (\<1K msg/sec) | 100 – 500 |
| Medium (1K – 10K/sec) | 1000 – 5000 |
| High (>10K/sec) | 5000 – 10000 |
### Max in flight [#max-in-flight]
`max_in_flight` controls how many batches can be sent concurrently. Default is `64`. For very high throughput, increase it along with the Arc server's resources:
```yaml
output:
arc:
max_in_flight: 128
batching:
count: 5000
period: 1s
```
### Compression choice [#compression-choice]
* **`zstd`** (default) — Best decompression performance on the Arc server. Recommended for most workloads.
* **`gzip`** — Slightly smaller payloads but higher CPU. Use if the Arc server is I/O bound and CPU is plentiful.
* **`none`** — Only useful for debugging or when running on localhost with very small payloads.
### Format choice [#format-choice]
Prefer `columnar` whenever batches share a consistent schema. It is significantly faster end-to-end. Use `row` only when you need per-message tags or flexible per-message fields.
## Troubleshooting [#troubleshooting]
### 401 Unauthorized [#401-unauthorized]
The Arc token is missing, invalid, or not being expanded by the shell.
```yaml
output:
arc:
token: "${ARC_TOKEN}" # Make sure ARC_TOKEN is exported in your env
```
Test the token directly:
```bash
curl -H "Authorization: Bearer $ARC_TOKEN" http://localhost:8000/api/v1/query \
-d '{"sql": "SHOW DATABASES"}'
```
### 400 Bad Request with "column length mismatch" [#400-bad-request-with-column-length-mismatch]
Columnar format requires all messages in a batch to share the same set of fields. If some messages have extra or missing fields, Arc rejects the batch.
Options:
* Switch to `format: row` if messages have varying schemas
* Add a Bloblang step that normalizes fields before the output
* Reduce batch size so each batch is more homogeneous
### Messages written but nothing queryable [#messages-written-but-nothing-queryable]
Arc buffers data in memory before flushing to Parquet (default 5 seconds). If you're checking immediately after writing, wait a few seconds and try again. For very small batches in local dev, set:
```yaml
batching:
count: 10
period: 1s
```
### Measurement name rejected [#measurement-name-rejected]
Arc validates measurement names (alphanumeric, underscores, hyphens, max 64 chars, must start with a letter). If you're using interpolation, make sure the value is clean:
```yaml
measurement: ${!json("type").string()}
```
### Timestamps in the wrong unit [#timestamps-in-the-wrong-unit]
If your source produces timestamps in milliseconds but Arc is interpreting them as something else, set `timestamp_unit` explicitly:
```yaml
timestamp_field: ts
timestamp_unit: ms # us | ms | s | ns | auto
```
The `auto` default detects the unit from magnitude, which is usually correct but fails for edge cases (e.g. very small timestamps from the 1970s).
## Resources [#resources]
* [Arc output plugin source](https://github.com/redpanda-data/connect/tree/main/internal/impl/arc)
* [Arc output reference docs](https://docs.redpanda.com/redpanda-connect/components/outputs/arc/)
* [Redpanda Connect documentation](https://docs.redpanda.com/redpanda-connect/about/)
* [Bloblang language reference](https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/)
* [Basekick blog post on the integration](https://basekick.net/blog/arc-redpanda-connect-output-plugin?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)
## Next steps [#next-steps]
* Pair with [Grafana](/arc/integrations/grafana/) to visualize the data Redpanda Connect ingests
* Use [Arc's native MQTT](/arc/integrations/mqtt/) when you don't need transformations
* Use [Telegraf](/arc/integrations/telegraf/) for system/infrastructure metrics
# Apache Superset Integration (/arc/integrations/superset)
Connect Arc to Apache Superset for interactive dashboards and visualizations.
## Overview [#overview]
Arc provides a native SQLAlchemy dialect for Apache Superset, enabling:
* Full analytical SQL query support
* Multi-database schema support
* Cross-database joins
* Time-series visualizations
* Interactive dashboards
## Installation [#installation]
### Option 1: install in existing Superset [#option-1-install-in-existing-superset]
```bash
# Activate Superset environment
source venv/bin/activate
# Install Arc dialect
pip install arc-superset-dialect
```
### Option 2: Docker with Arc pre-configured [#option-2-docker-with-arc-pre-configured]
```bash
# Clone Arc Superset dialect repo
git clone https://github.com/basekick-labs/arc-superset-dialect.git
cd arc-superset-dialect
# Build and run
docker build -t superset-arc .
docker run -d \
-p 8088:8088 \
--name superset-arc \
superset-arc
```
Access Superset at `http://localhost:8088` (admin/admin)
## Connecting to Arc [#connecting-to-arc]
### 1. Add database connection [#1-add-database-connection]
In Superset UI:
1. Click **Settings** → **Database Connections**
2. Click **+ Database**
3. Select **Other** from database list
4. Enter connection string
### 2. Connection string format [#2-connection-string-format]
```text
arc://{api_token}@{host}:{port}/{database}
```
**Example:**
```text
arc://YourAPITokenHere@localhost:8000/default
```
### 3. Test connection [#3-test-connection]
Click **Test Connection** to verify Arc is reachable.
## Multi-database support [#multi-database-support]
Arc databases appear as **schemas** in Superset:
```text
Connection: arc://token@localhost:8000/default
Schemas available:
├── default
│ ├── cpu
│ ├── mem
│ └── disk
├── production
│ ├── cpu
│ └── mem
└── staging
├── cpu
└── mem
```
### Querying different databases [#querying-different-databases]
```sql
-- Query default database
SELECT * FROM default.cpu LIMIT 10;
-- Query production database
SELECT * FROM production.cpu LIMIT 10;
-- Cross-database query
SELECT
p.timestamp,
p.host,
p.usage_idle as prod_cpu,
s.usage_idle as staging_cpu
FROM production.cpu p
JOIN staging.cpu s ON p.timestamp = s.timestamp AND p.host = s.host
WHERE p.timestamp > NOW() - INTERVAL 1 HOUR;
```
## Creating charts [#creating-charts]
### Time-series line chart [#time-series-line-chart]
**SQL Query:**
```sql
SELECT
time_bucket(INTERVAL '5 minutes', timestamp) as time,
host,
AVG(usage_idle) as avg_idle
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 6 HOUR
GROUP BY time, host
ORDER BY time DESC;
```
**Chart Configuration:**
* **Chart Type**: Line Chart
* **Time Column**: time
* **Metrics**: avg\_idle
* **Group By**: host
### CPU vs memory correlation [#cpu-vs-memory-correlation]
**SQL Query:**
```sql
SELECT
c.timestamp,
c.host,
c.usage_idle as cpu_idle,
m.used_percent as mem_used
FROM default.cpu c
JOIN default.mem m ON c.timestamp = m.timestamp AND c.host = m.host
WHERE c.timestamp > NOW() - INTERVAL 1 HOUR
ORDER BY c.timestamp DESC;
```
**Chart Configuration:**
* **Chart Type**: Mixed Chart (Line + Bar)
* **X-axis**: timestamp
* **Y-axis 1**: cpu\_idle
* **Y-axis 2**: mem\_used
### Top hosts by CPU usage [#top-hosts-by-cpu-usage]
**SQL Query:**
```sql
SELECT
host,
AVG(usage_user + usage_system) as avg_usage,
MAX(usage_user + usage_system) as max_usage
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY host
ORDER BY avg_usage DESC
LIMIT 10;
```
**Chart Configuration:**
* **Chart Type**: Bar Chart
* **X-axis**: host
* **Y-axis**: avg\_usage
* **Sort**: Descending
### Heatmap - host activity [#heatmap---host-activity]
**SQL Query:**
```sql
SELECT
DATE_TRUNC('hour', timestamp) as hour,
host,
AVG(100 - usage_idle) as cpu_activity
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 7 DAY
GROUP BY hour, host;
```
**Chart Configuration:**
* **Chart Type**: Heatmap
* **X-axis**: hour
* **Y-axis**: host
* **Color**: cpu\_activity
## Creating dashboards [#creating-dashboards]
### 1. Create dashboard [#1-create-dashboard]
1. Click **Dashboards** → **+ Dashboard**
2. Name it: "System Monitoring"
3. Click **Edit Dashboard**
### 2. Add charts [#2-add-charts]
Drag and drop charts from the chart list or create new ones.
### 3. Add filters [#3-add-filters]
```sql
-- Host filter
SELECT DISTINCT host FROM default.cpu ORDER BY host;
-- Time range filter
-- Use Superset's built-in time range filter
```
### 4. Dashboard layout [#4-dashboard-layout]
Example monitoring dashboard layout:
```text
┌─────────────────────────────────────────┐
│ System Overview - Last 24 Hours │
├─────────────────┬───────────────────────┤
│ │ │
│ CPU Usage │ Memory Usage │
│ (Line Chart) │ (Line Chart) │
│ │ │
├─────────────────┼───────────────────────┤
│ │ │
│ Top 10 Hosts │ Disk I/O │
│ (Bar Chart) │ (Area Chart) │
│ │ │
├─────────────────┴───────────────────────┤
│ │
│ Host Activity Heatmap (7 days) │
│ (Heatmap) │
│ │
└─────────────────────────────────────────┘
```
## Advanced features [#advanced-features]
### Custom SQL [#custom-sql]
Superset supports the full analytical SQL dialect:
```sql
-- Window functions
SELECT
timestamp,
host,
usage_idle,
AVG(usage_idle) OVER (
PARTITION BY host
ORDER BY timestamp
ROWS BETWEEN 5 PRECEDING AND CURRENT ROW
) as moving_avg
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 1 HOUR;
-- CTEs (Common Table Expressions)
WITH hourly_avg AS (
SELECT
DATE_TRUNC('hour', timestamp) as hour,
host,
AVG(usage_idle) as avg_idle
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY hour, host
)
SELECT * FROM hourly_avg
WHERE avg_idle < 50
ORDER BY hour DESC;
-- Percentiles
SELECT
host,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY usage_idle) as p50,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY host;
```
### Alerts [#alerts]
Configure alerts in Superset:
1. Go to **Settings** → **Alerts & Reports**
2. Click **+ Alert**
3. Configure:
* **Chart**: Select your chart
* **Condition**: Greater than, Less than, etc.
* **Threshold**: Value
* **Recipients**: Email addresses
* **Schedule**: Cron expression
**Example Alert - High CPU Usage:**
```sql
SELECT
host,
AVG(100 - usage_idle) as cpu_usage
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 5 MINUTE
GROUP BY host
HAVING AVG(100 - usage_idle) > 80;
```
Alert when query returns rows (CPU > 80%)
### Scheduled reports [#scheduled-reports]
Email dashboards on a schedule:
1. Go to **Dashboards** → Your Dashboard
2. Click **...** → **Set up email report**
3. Configure:
* **Recipients**: Email list
* **Schedule**: Daily at 8 AM
* **Format**: PNG or PDF
## Performance tips [#performance-tips]
### 1. Use time filters [#1-use-time-filters]
Always filter by time to reduce data scanned:
```sql
-- Good: Time filter
WHERE timestamp > NOW() - INTERVAL 24 HOUR
-- Bad: No filter (scans all data)
SELECT * FROM default.cpu
```
### 2. Limit result size [#2-limit-result-size]
```sql
-- Add LIMIT to exploratory queries
SELECT * FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 1 HOUR
LIMIT 1000;
```
### 3. Enable query caching [#3-enable-query-caching]
When Arc reads from S3-compatible storage, caching fetched blocks speeds up the
repeated queries a dashboard produces. In Arc's `arc.toml`:
```toml
[query]
enable_s3_cache = true
s3_cache_size = "128MB"
s3_cache_ttl_seconds = 3600
```
See [query caching](/arc/advanced/caching/) for the caches Arc keeps on the
query path.
### 4. Use materialized queries [#4-use-materialized-queries]
For slow dashboards, create materialized views:
```sql
-- Pre-aggregate data
CREATE TABLE default.cpu_hourly AS
SELECT
DATE_TRUNC('hour', timestamp) as hour,
host,
AVG(usage_idle) as avg_idle,
MAX(usage_idle) as max_idle,
MIN(usage_idle) as min_idle
FROM default.cpu
GROUP BY hour, host;
-- Query materialized data
SELECT * FROM default.cpu_hourly
WHERE hour > NOW() - INTERVAL 7 DAY;
```
### 5. Optimize chart SQL [#5-optimize-chart-sql]
```sql
-- Good: Aggregate first
SELECT
DATE_TRUNC('hour', timestamp) as hour,
AVG(usage_idle) as avg_idle
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY hour;
-- Bad: Return all rows
SELECT timestamp, usage_idle
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR;
-- Then aggregate in Superset (slow)
```
## Troubleshooting [#troubleshooting]
### Connection refused [#connection-refused]
```bash
# Check Arc is running
curl http://localhost:8000/health
# Verify token
curl -H "Authorization: Bearer $ARC_TOKEN" http://localhost:8000/auth/verify
```
### No schemas showing [#no-schemas-showing]
```sql
-- Verify databases exist
SHOW DATABASES;
-- Check tables in database
SHOW TABLES;
```
### Slow queries [#slow-queries]
```bash
# Check compaction status
curl http://localhost:8000/api/compaction/status
# Manually trigger compaction
curl -X POST http://localhost:8000/api/compaction/trigger \
-H "Authorization: Bearer $ARC_TOKEN"
```
### Token expired [#token-expired]
Create a new token:
```bash
# Docker
docker exec -it arc-api python3 -c "
from api.auth import AuthManager
auth = AuthManager(db_path='/data/arc.db')
token = auth.create_token('superset', description='Superset connection')
print(token)
"
# Native
python3 -c "
from api.auth import AuthManager
auth = AuthManager(db_path='./data/arc.db')
token = auth.create_token('superset', description='Superset connection')
print(token)
"
```
Update connection string in Superset with new token.
## Example dashboards [#example-dashboards]
### System monitoring dashboard [#system-monitoring-dashboard]
**Queries Included:**
* CPU Usage by Host (last 24h)
* Memory Usage Trends
* Disk I/O Operations
* Network Traffic
* Top 10 Busiest Hosts
* System Health Heatmap
### IoT sensor dashboard [#iot-sensor-dashboard]
**Queries Included:**
* Temperature Trends
* Sensor Online/Offline Status
* Alert History
* Anomaly Detection
* Geographic Distribution
## Resources [#resources]
* **[Arc Superset Dialect GitHub](https://github.com/basekick-labs/arc-superset-dialect)**
* **[PyPI Package](https://pypi.org/project/arc-superset-dialect/)**
* **[Superset Documentation](https://superset.apache.org/docs/intro)**
* **[DuckDB SQL Reference](https://duckdb.org/docs/sql/introduction)**
## Next steps [#next-steps]
* **[Query API Reference](/arc/api-reference/overview/#querying)**
* **[SQL Query Guide](/arc/guides/querying/)**
# Telegraf Integration (/arc/integrations/telegraf)
Use Telegraf to collect system metrics and send them directly to Arc using the native Arc output plugin.
## Overview [#overview]
Arc provides a native Telegraf output plugin that sends metrics in MessagePack columnar format for maximum performance. The plugin supports gzip compression and integrates seamlessly with Arc's multi-database architecture.
**Benefits:**
* Native MessagePack columnar format
* Built-in gzip compression
* Direct database targeting
* All 300+ Telegraf input plugins supported
* Full analytical SQL support
## Prerequisites [#prerequisites]
* **Telegraf 1.37 or higher** (required for Arc output plugin)
* Arc server running and accessible
* Arc API token
## Quick start [#quick-start]
### 1. Install Telegraf [#1-install-telegraf]
```bash
# Ubuntu/Debian
wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
echo "deb https://repos.influxdata.com/ubuntu focal stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt update && sudo apt install telegraf
# macOS
brew install telegraf
# Or download from https://portal.influxdata.com/downloads/
```
Verify you have Telegraf 1.37+:
```bash
telegraf --version
```
### 2. Configure Telegraf for Arc [#2-configure-telegraf-for-arc]
Edit `/etc/telegraf/telegraf.conf`:
```toml
# Arc Output Plugin
[[outputs.arc]]
# Arc MessagePack endpoint
url = "http://localhost:8000/api/v1/write/msgpack"
# Arc API token
api_key = "$ARC_TOKEN"
# Enable gzip compression (recommended)
content_encoding = "gzip"
# Target database in Arc
database = "telegraf"
```
### 3. Enable input plugins [#3-enable-input-plugins]
```toml
# System metrics
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.mem]]
[[inputs.net]]
interfaces = ["eth*", "en*"]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
```
### 4. Start Telegraf [#4-start-telegraf]
```bash
# Start service
sudo systemctl start telegraf
# Enable on boot
sudo systemctl enable telegraf
# Check status
sudo systemctl status telegraf
# View logs
sudo journalctl -u telegraf -f
```
### 5. Verify data in Arc [#5-verify-data-in-arc]
```bash
# Check measurements
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SHOW TABLES FROM telegraf", "format": "json"}'
# Query CPU data
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM telegraf.cpu ORDER BY time DESC LIMIT 10", "format": "json"}'
```
## Configuration examples [#configuration-examples]
### Minimal configuration [#minimal-configuration]
```toml
[agent]
interval = "10s"
flush_interval = "10s"
[[outputs.arc]]
url = "http://localhost:8000/api/v1/write/msgpack"
api_key = "$ARC_TOKEN"
content_encoding = "gzip"
database = "telegraf"
[[inputs.cpu]]
[[inputs.mem]]
[[inputs.disk]]
```
### High-performance configuration [#high-performance-configuration]
```toml
[agent]
interval = "10s"
flush_interval = "10s"
metric_batch_size = 5000 # Larger batches for higher throughput
metric_buffer_limit = 50000 # Buffer more metrics
[[outputs.arc]]
url = "http://localhost:8000/api/v1/write/msgpack"
api_key = "$ARC_TOKEN"
content_encoding = "gzip"
database = "metrics"
# Enable all system metrics
[[inputs.cpu]]
percpu = true
totalcpu = true
[[inputs.disk]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.kernel]]
[[inputs.diskio]]
```
### Multi-environment configuration [#multi-environment-configuration]
```toml
# Production metrics → production database
[[outputs.arc]]
url = "https://arc-prod.example.com/api/v1/write/msgpack"
api_key = "$ARC_TOKEN"
content_encoding = "gzip"
database = "production"
# Staging metrics → staging database
[[outputs.arc]]
url = "https://arc-staging.example.com/api/v1/write/msgpack"
api_key = "$ARC_TOKEN"
content_encoding = "gzip"
database = "staging"
```
## Available input plugins [#available-input-plugins]
### System metrics [#system-metrics]
```toml
# CPU usage by core
[[inputs.cpu]]
percpu = true
totalcpu = true
# Memory usage
[[inputs.mem]]
# Disk usage and I/O
[[inputs.disk]]
[[inputs.diskio]]
# Network statistics
[[inputs.net]]
# Process information
[[inputs.processes]]
# System load
[[inputs.system]]
# Kernel statistics
[[inputs.kernel]]
# Swap usage
[[inputs.swap]]
```
### Docker monitoring [#docker-monitoring]
```toml
[[inputs.docker]]
endpoint = "unix:///var/run/docker.sock"
gather_services = false
container_names = []
timeout = "5s"
perdevice = true
total = true
```
### PostgreSQL monitoring [#postgresql-monitoring]
```toml
[[inputs.postgresql]]
address = "postgres://user:pass@localhost/dbname?sslmode=disable"
databases = ["mydb"]
```
### Redis monitoring [#redis-monitoring]
```toml
[[inputs.redis]]
servers = ["tcp://localhost:6379"]
```
### NGINX monitoring [#nginx-monitoring]
```toml
[[inputs.nginx]]
urls = ["http://localhost/nginx_status"]
```
### HTTP response time [#http-response-time]
```toml
[[inputs.http_response]]
urls = [
"https://example.com",
"https://api.example.com/health"
]
method = "GET"
response_timeout = "5s"
follow_redirects = true
```
### Custom exec plugin [#custom-exec-plugin]
```toml
[[inputs.exec]]
commands = ["/usr/local/bin/custom_metrics.sh"]
timeout = "5s"
data_format = "influx"
```
## Querying Telegraf data in Arc [#querying-telegraf-data-in-arc]
### View available measurements [#view-available-measurements]
```sql
SHOW TABLES FROM telegraf;
```
**Common measurements from Telegraf:**
* `cpu` - CPU usage per core
* `mem` - Memory statistics
* `disk` - Disk usage
* `diskio` - Disk I/O stats
* `net` - Network statistics
* `processes` - Process counts
* `system` - System load
* `docker` - Container metrics
### CPU usage analysis [#cpu-usage-analysis]
```sql
-- Average CPU usage by host (last hour)
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
host,
AVG(usage_user + usage_system) as avg_usage
FROM telegraf.cpu
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, host
ORDER BY bucket DESC;
-- Highest CPU usage instances
SELECT
host,
cpu,
MAX(usage_user + usage_system) as max_usage
FROM telegraf.cpu
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY host, cpu
ORDER BY max_usage DESC
LIMIT 10;
```
### Memory analysis [#memory-analysis]
```sql
-- Memory usage trend
SELECT
time_bucket(INTERVAL '1 hour', time) as hour,
host,
AVG(used_percent) as avg_mem_usage
FROM telegraf.mem
WHERE time > NOW() - INTERVAL '7 days'
GROUP BY hour, host
ORDER BY hour DESC;
-- Hosts with high memory usage
SELECT
host,
AVG(used_percent) as avg_usage,
MAX(used_percent) as max_usage
FROM telegraf.mem
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY host
HAVING AVG(used_percent) > 80
ORDER BY avg_usage DESC;
```
### Disk analysis [#disk-analysis]
```sql
-- Disk usage by mount point
SELECT
host,
path,
AVG(used_percent) as avg_usage
FROM telegraf.disk
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY host, path
ORDER BY avg_usage DESC;
-- Disk I/O operations
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
name,
SUM(reads) as total_reads,
SUM(writes) as total_writes
FROM telegraf.diskio
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, name
ORDER BY bucket DESC;
```
### Network analysis [#network-analysis]
```sql
-- Network throughput
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
interface,
SUM(bytes_sent) / (5 * 60) as bytes_sent_per_sec,
SUM(bytes_recv) / (5 * 60) as bytes_recv_per_sec
FROM telegraf.net
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, interface
ORDER BY bucket DESC;
```
### Docker container monitoring [#docker-container-monitoring]
```sql
-- Container CPU usage
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
container_name,
AVG(usage_percent) as avg_cpu
FROM telegraf.docker_container_cpu
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, container_name
ORDER BY bucket DESC;
-- Container memory usage
SELECT
container_name,
AVG(usage) as avg_memory_bytes,
MAX(usage) as max_memory_bytes
FROM telegraf.docker_container_mem
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY container_name
ORDER BY avg_memory_bytes DESC;
```
## Performance tuning [#performance-tuning]
### Optimize batch size [#optimize-batch-size]
```toml
[agent]
metric_batch_size = 5000 # Send 5000 metrics per request
metric_buffer_limit = 50000 # Buffer 50k metrics before dropping
```
**Guidelines:**
* **Low volume** (\<1000 metrics/sec): batch\_size = 1000
* **Medium volume** (1000-10000/sec): batch\_size = 5000
* **High volume** (>10000/sec): batch\_size = 10000
### Collection intervals [#collection-intervals]
```toml
[agent]
interval = "10s" # Collect every 10 seconds
flush_interval = "10s" # Send every 10 seconds
```
For real-time monitoring, use smaller intervals (5s). For cost optimization, use larger intervals (60s).
### Enable compression [#enable-compression]
Always use gzip compression for better network efficiency:
```toml
[[outputs.arc]]
content_encoding = "gzip" # Compress payloads
```
## Troubleshooting [#troubleshooting]
### Telegraf can't connect to Arc [#telegraf-cant-connect-to-arc]
```bash
# Test Arc connectivity
curl http://localhost:8000/health
# Test with token
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT 1", "format": "json"}'
# Check Telegraf logs
sudo journalctl -u telegraf -f | grep -i error
```
### No data appearing [#no-data-appearing]
```bash
# Verify Telegraf is running
sudo systemctl status telegraf
# Check Telegraf config syntax
telegraf --config /etc/telegraf/telegraf.conf --test
# Check Arc received data
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT COUNT(*) FROM telegraf.cpu", "format": "json"}'
```
### Authentication errors [#authentication-errors]
Ensure your API key is correct in the configuration:
```toml
[[outputs.arc]]
api_key = "$ARC_TOKEN" # Must be a valid Arc API token
```
### Metrics being dropped [#metrics-being-dropped]
```bash
# Increase buffer
[agent]
metric_buffer_limit = 100000 # Increase from default
# Check Arc health
curl http://localhost:8000/health
```
### Version check [#version-check]
The Arc output plugin requires Telegraf 1.37+:
```bash
telegraf --version
# Telegraf 1.37.0 (or higher required)
```
## Dashboard integration [#dashboard-integration]
### Grafana with Arc [#grafana-with-arc]
Use the [Arc Grafana datasource plugin](/arc/integrations/grafana/) for native integration:
```text
1. Install the Arc datasource from Grafana marketplace
2. Configure connection to your Arc instance
3. Use analytical SQL in your dashboard panels
```
See [Grafana Integration](/arc/integrations/grafana/) for detailed setup instructions.
## Best practices [#best-practices]
### 1. Use tags efficiently [#1-use-tags-efficiently]
```toml
[global_tags]
environment = "production"
datacenter = "us-east-1"
region = "us-east"
```
Tags enable powerful GROUP BY queries but increase cardinality.
### 2. Filter unnecessary metrics [#2-filter-unnecessary-metrics]
```toml
[[inputs.cpu]]
percpu = false # Aggregate across CPUs
totalcpu = true
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs"] # Skip temporary filesystems
```
### 3. Use measurement filters [#3-use-measurement-filters]
```toml
[[outputs.arc]]
namepass = ["cpu*", "mem*", "disk*"] # Only send specific metrics
# OR
namedrop = ["docker_*"] # Exclude Docker metrics
```
### 4. Set reasonable collection intervals [#4-set-reasonable-collection-intervals]
```toml
[[inputs.cpu]]
interval = "10s" # Fast-changing metrics
[[inputs.disk]]
interval = "60s" # Slow-changing metrics
```
## Resources [#resources]
* **[Telegraf Documentation](https://docs.influxdata.com/telegraf/)**
* **[Telegraf Plugins](https://docs.influxdata.com/telegraf/latest/plugins/)**
* **[Arc Query Guide](/arc/guides/querying/)**
* **[Arc Grafana Integration](/arc/integrations/grafana/)**
## Next steps [#next-steps]
* **[Query Telegraf metrics](/arc/guides/querying/)**
* **[Create Grafana dashboards](/arc/integrations/grafana/)**
* **[Set up alerts](/arc/integrations/grafana/)**
* **[Configuration Reference](/arc/configuration/overview/)**
# TLE (Satellite Orbital Data) (/arc/integrations/tle)
Ingest satellite orbital data in the standard Two-Line Element (TLE) format used by Space-Track.org, CelesTrak, and ground station pipelines.
TLE ingestion and import are available starting Arc v26.02.1 (February 2026).
## Overview [#overview]
Arc provides native TLE parsing with two ingestion modes:
* **Streaming** (`POST /api/v1/write/tle`) -- for continuous TLE feeds, cron jobs, and real-time updates
* **Bulk import** (`POST /api/v1/import/tle`) -- for historical backfill from Space-Track.org exports or CelesTrak catalog dumps
TLE data is parsed into a configurable measurement (default: `satellite_tle`) with orbital elements as fields and satellite identifiers as tags. Derived orbital metrics (semi-major axis, period, apogee, perigee, orbit classification) are computed automatically.
## TLE format [#tle-format]
Arc supports both 3-line (with satellite name) and 2-line (no name) TLE formats, including mixed-format files.
**3-line format:**
```bash
ISS (ZARYA)
1 25544U 98067A 24001.50000000 .00016717 00000-0 10270-3 0 9001
2 25544 51.6400 100.2000 0007420 35.5000 324.6000 15.49560000 09
```
**2-line format:**
```text
1 25544U 98067A 24001.50000000 .00016717 00000-0 10270-3 0 9001
2 25544 51.6400 100.2000 0007420 35.5000 324.6000 15.49560000 09
```
## Headers [#headers]
| Header | Required | Default | Description |
| ------------------- | -------- | --------------- | ----------------------- |
| `Authorization` | Yes | - | `Bearer $ARC_TOKEN` |
| `X-Arc-Database` | No | `default` | Target database |
| `X-Arc-Measurement` | No | `satellite_tle` | Target measurement name |
## Streaming ingestion [#streaming-ingestion]
For continuous TLE feeds, cron jobs, and real-time updates from ground stations.
### Endpoint [#endpoint]
```bash
POST /api/v1/write/tle
```
### Example [#example]
```bash
curl -X POST "http://localhost:8000/api/v1/write/tle" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: satellites" \
--data-binary @stations.tle
```
Returns `204 No Content` on success.
### Custom measurement [#custom-measurement]
```bash
curl -X POST "http://localhost:8000/api/v1/write/tle" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: satellites" \
-H "X-Arc-Measurement: iss_orbital_elements" \
--data-binary @iss.tle
```
### Statistics [#statistics]
```bash
curl "http://localhost:8000/api/v1/write/tle/stats" \
-H "Authorization: Bearer $ARC_TOKEN"
```
## Bulk import [#bulk-import]
For historical backfill from Space-Track.org exports or CelesTrak catalog dumps.
### Endpoint [#endpoint-1]
```bash
POST /api/v1/import/tle
```
### Example [#example-1]
```bash
curl -X POST "http://localhost:8000/api/v1/import/tle" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: satellites" \
-F "file=@catalog.tle"
```
### Response [#response]
```json
{
"status": "ok",
"result": {
"database": "satellites",
"measurement": "satellite_tle",
"satellite_count": 28000,
"rows_imported": 28000,
"duration_ms": 1250
}
}
```
## Schema [#schema]
The default measurement `satellite_tle` has the following schema:
### Tags [#tags]
| Column | Description |
| -------------------------- | --------------------------------------------- |
| `norad_id` | NORAD catalog number |
| `object_name` | Satellite name (from line 0 of 3-line format) |
| `classification` | U (unclassified), C (classified), S (secret) |
| `international_designator` | Launch year + piece identifier |
| `orbit_type` | Derived: LEO, MEO, GEO, HEO |
### Fields [#fields]
| Column | Type | Description |
| --------------------- | ----- | ------------------------------------------- |
| `inclination_deg` | float | Orbital inclination (degrees) |
| `raan_deg` | float | Right ascension of ascending node (degrees) |
| `eccentricity` | float | Orbital eccentricity |
| `arg_perigee_deg` | float | Argument of perigee (degrees) |
| `mean_anomaly_deg` | float | Mean anomaly (degrees) |
| `mean_motion_rev_day` | float | Revolutions per day |
| `bstar` | float | BSTAR drag coefficient |
| `semi_major_axis_km` | float | Derived: semi-major axis (km) |
| `period_min` | float | Derived: orbital period (minutes) |
| `apogee_km` | float | Derived: apogee altitude (km) |
| `perigee_km` | float | Derived: perigee altitude (km) |
## Example queries [#example-queries]
```sql
-- All LEO satellites sorted by orbital period
SELECT object_name, orbit_type, period_min, perigee_km, apogee_km
FROM satellite_tle
WHERE orbit_type = 'LEO'
ORDER BY period_min;
-- Track orbital decay for a specific satellite
SELECT object_name, time, mean_motion_rev_day, perigee_km
FROM satellite_tle
WHERE norad_id = '25544'
ORDER BY time DESC
LIMIT 100;
-- Count satellites by orbit type
SELECT orbit_type, COUNT(DISTINCT norad_id) as satellite_count
FROM satellite_tle
GROUP BY orbit_type
ORDER BY satellite_count DESC;
```
## Features [#features]
* **Pure Go parser** -- no external dependencies
* **Mixed format support** -- handles both 2-line and 3-line TLE in the same file
* **Gzip support** -- compressed payloads auto-detected via magic bytes
* **Checksum validation** -- graceful skip on bad entries (warnings collected, not fatal)
* **Derived metrics** -- semi-major axis, period, apogee, perigee, and orbit classification computed automatically
* **RBAC-aware** -- write permissions checked per measurement
* **Cluster routing** -- writes forwarded to writer nodes automatically
* **500 MB size limit** on bulk imports
## Performance [#performance]
TLE ingestion uses a typed columnar fast path that bypasses the generic `[]interface{}` intermediary. The parser operates directly on `[]byte` input with contiguous record allocation and single-pass typed column construction.
# VS Code Extension (/arc/integrations/vscode)
Complete development toolkit for Arc Database directly in Visual Studio Code.
## Overview [#overview]
The Arc Database Manager extension provides a full-featured IDE for working with Arc:
* **Connection Management**: Multiple saved connections with secure token storage
* **SQL IntelliSense**: Auto-completion for tables, columns, and SQL functions
* **Interactive Results**: Export to CSV/JSON/Markdown with automatic chart visualization
* **Arc Notebooks**: Mix SQL and Markdown in `.arcnb` files with parameterized queries
* **Schema Explorer**: Browse databases and tables with context menus
* **Data Ingestion**: CSV import wizard and bulk data generator
* **Alerting**: Create query-based alerts with desktop notifications
* **Query Management**: Automatic history and saved queries
* **Dark Mode**: Automatic theme detection and adaptation
## Installation [#installation]
### From VS Code Marketplace [#from-vs-code-marketplace]
1. Open VS Code
2. Click **Extensions** (Ctrl+Shift+X / Cmd+Shift+X)
3. Search for **"Arc Database Manager"**
4. Click **Install**
Or install directly from the marketplace:
* **[Arc Database Manager on VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=basekick-labs.arc-db-manager)**
### From command line [#from-command-line]
```bash
code --install-extension basekick-labs.arc-db-manager
```
## Quick start [#quick-start]
### 1. Connect to Arc server [#1-connect-to-arc-server]
**Option A: From Status Bar**
1. Click **"Arc: Not Connected"** in the status bar
2. Enter connection details:
* Name: `My Arc Server`
* Host: `localhost`
* Port: `8000`
* Protocol: `http` or `https`
3. Enter your authentication token
**Option B: From Command Palette**
1. Press `Ctrl+Shift+P` / `Cmd+Shift+P`
2. Type `Arc: Connect to Server`
3. Follow the prompts
### 2. Get your API token [#2-get-your-api-token]
```bash
# Docker - check logs for admin token
docker logs 2>&1 | grep "Admin token"
# Or create a new token
curl -X POST http://localhost:8000/api/v1/auth/tokens \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "vscode-extension",
"description": "VS Code extension access"
}'
```
### 3. Start querying [#3-start-querying]
1. Press `Ctrl+Shift+P` → `Arc: New Query`
2. Write your SQL query
3. Press `Ctrl+Enter` / `Cmd+Enter` to execute
**Example Query:**
```sql
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(usage_idle) * -1 + 100 AS cpu_usage,
host
FROM prod.cpu
WHERE cpu = 'cpu-total'
AND time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, host
ORDER BY bucket ASC;
```
## Features [#features]
### SQL IntelliSense [#sql-intellisense]
Get auto-completion for:
* Database names
* Table names
* Column names
* SQL functions
* SQL keywords
**How to use:**
* Start typing and IntelliSense will suggest completions
* Press `Ctrl+Space` to manually trigger suggestions
* Navigate with arrow keys, press `Enter` to accept
### Interactive results view [#interactive-results-view]
After executing a query, results are displayed with:
**Export Options:**
* CSV format
* JSON format
* Markdown tables
**Automatic Visualizations:**
* Time-series data is automatically charted
* Line charts for temporal data
* Theme-aware (adapts to VS Code theme)
**Table Features:**
* Sort by clicking column headers
* Filter rows with search
* View execution time and row count
### Arc notebooks [#arc-notebooks]
Create analysis documents mixing SQL and Markdown in `.arcnb` files.
**Create a Notebook:**
1. Press `Ctrl+Shift+P` → `Arc: New Notebook`
2. Save with `.arcnb` extension
**Notebook Features:**
* Mix Markdown documentation with SQL queries
* Parameterized queries with variable substitution
* Execute cells individually or all at once
* Export to Markdown with results
* Auto-save functionality
**Example Notebook:**
```markdown
# CPU Performance Analysis
This notebook analyzes CPU usage patterns over time.
Variables:
- interval = 1 HOUR
- threshold = 80
- database = prod
## Average CPU Usage
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(usage_user) as avg_cpu,
host
FROM ${database}.cpu
WHERE time > NOW() - INTERVAL ${interval}
AND usage_user > ${threshold}
GROUP BY bucket, host
ORDER BY bucket DESC;
## Results
The query shows periods where CPU exceeded ${threshold}% in the last ${interval}.
```
**Variable Syntax:**
* Define variables in YAML frontmatter
* Reference with `${variable_name}`
* Variables are replaced before execution
### Schema explorer [#schema-explorer]
Browse your Arc databases and tables in the sidebar.
**Features:**
* Hierarchical view of databases and tables
* Connection status indicator
* Visual refresh button
**Right-Click Context Menus:**
**On Tables:**
* **Show Table Schema** - View column names and types
* **Preview Data** - Show first 100 rows
* **Show Table Statistics** - Row count and size
* **Generate SELECT Query** - Create basic query
* **Query Last Hour** - Filter to recent data
* **Query Today** - Filter to today's data
**Example: Show Table Schema**
```text
Right-click table → Show Table Schema
Result:
┌────────────┬──────────┐
│ Column │ Type │
├────────────┼──────────┤
│ time │ TIMESTAMP│
│ host │ VARCHAR │
│ usage_idle │ DOUBLE │
│ usage_user │ DOUBLE │
└────────────┴──────────┘
```
### Data ingestion [#data-ingestion]
#### CSV import wizard [#csv-import-wizard]
Import CSV files directly into Arc with guided setup.
**Steps:**
1. Press `Ctrl+Shift+P` → `Arc: Import CSV`
2. Select your CSV file
3. Configure import settings:
* Auto-detect delimiter and headers
* Select timestamp column
* Choose target database
* Set batch size
**Performance:**
* Throughput depends on file size, schema, and host hardware
* Uses MessagePack columnar format
* Progress tracking for large files
* Batch processing support
**Example:**
```bash
Import Settings:
- File: metrics.csv
- Delimiter: , (auto-detected)
- Timestamp Column: time
- Database: prod
- Measurement: custom_metrics
- Batch Size: 10,000
Result: 250,000 rows imported in 3.2 seconds
```
#### Bulk data generator [#bulk-data-generator]
Generate test data for development and testing.
**Presets:**
1. **CPU Metrics** - System CPU usage data
2. **Memory Metrics** - Memory usage statistics
3. **Network Metrics** - Network traffic data
4. **IoT Sensor Data** - Temperature, humidity sensors
5. **Custom Schema** - Define your own fields
**Steps:**
1. Press `Ctrl+Shift+P` → `Arc: Generate Test Data`
2. Select preset
3. Configure:
* Number of rows
* Target database
* Time range
**Performance:**
* Throughput depends on the selected dataset and host hardware
* Realistic sample data
* Configurable patterns
### Alerting & monitoring [#alerting--monitoring]
Create alerts based on query results with desktop notifications.
**Create an Alert:**
1. Press `Ctrl+Shift+P` → `Arc: Create Alert`
2. Configure alert:
* Name
* SQL query
* Condition type
* Threshold value
* Check interval
**Condition Types:**
* Greater than
* Less than
* Equals
* Not equals
* Contains
**Example Alert:**
```text
Name: High CPU Usage
Query: SELECT AVG(usage_user) as cpu FROM prod.cpu WHERE time > NOW() - INTERVAL '5 minutes'
Condition: greater_than
Threshold: 80
Interval: 60s
```
**Alert Features:**
* Desktop notifications when triggered
* Alert history tracking
* Enable/disable without deletion
* Minimum check interval: 10 seconds
### Query management [#query-management]
**Query History:**
* Every executed query is automatically saved
* View execution time and row counts
* Quick re-run from history
* Search through past queries
**Saved Queries:**
* Bookmark frequently used queries
* Organize by tags or folders
* Quick access from sidebar
**Access:**
1. Open Arc sidebar
2. Navigate to **Query History** or **Saved Queries**
3. Click query to view or re-run
### Token management [#token-management]
Manage Arc authentication tokens directly from VS Code.
**Features:**
* Create new tokens
* Rotate existing tokens
* Delete tokens
* Verify token validity
* Secure storage in system keychain
**Access:**
1. Press `Ctrl+Shift+P`
2. Type `Arc: Manage Tokens`
3. Select action
## Commands [#commands]
Access all commands via Command Palette (`Ctrl+Shift+P` / `Cmd+Shift+P`):
### Connection [#connection]
* `Arc: Connect to Server`
* `Arc: Disconnect`
* `Arc: Verify Token`
* `Arc: Manage Tokens`
### Queries [#queries]
* `Arc: New Query`
* `Arc: Execute Query` (Ctrl+Enter / Cmd+Enter)
* `Arc: Save Current Query`
* `Arc: Open Query History`
### Notebooks [#notebooks]
* `Arc: New Notebook`
* `Arc: Execute Notebook Cell`
* `Arc: Execute All Cells`
* `Arc: Export Notebook to Markdown`
### Data [#data]
* `Arc: Import CSV`
* `Arc: Generate Test Data`
### Alerts [#alerts]
* `Arc: Create Alert`
* `Arc: View Alerts`
* `Arc: Enable/Disable Alert`
### Explorer [#explorer]
* `Arc: Refresh Explorer`
* `Arc: Show Table Schema`
* `Arc: Preview Table Data`
* `Arc: Show Table Statistics`
## Keyboard shortcuts [#keyboard-shortcuts]
| Command | Windows/Linux | macOS |
| --------------- | ------------------------------- | ------------------------------ |
| Execute Query | `Ctrl+Enter` | `Cmd+Enter` |
| New Query | `Ctrl+Shift+P` → Arc: New Query | `Cmd+Shift+P` → Arc: New Query |
| Command Palette | `Ctrl+Shift+P` | `Cmd+Shift+P` |
| Toggle Sidebar | `Ctrl+B` | `Cmd+B` |
## Extension settings [#extension-settings]
Configure extension defaults in VS Code settings:
```json
{
"arc.defaultHost": "localhost",
"arc.defaultPort": 8000,
"arc.defaultProtocol": "http"
}
```
**Available Settings:**
| Setting | Description | Default |
| --------------------- | ----------------------- | ----------- |
| `arc.defaultHost` | Default Arc server host | `localhost` |
| `arc.defaultPort` | Default Arc server port | `8000` |
| `arc.defaultProtocol` | Default protocol | `http` |
## Use cases [#use-cases]
### Development & testing [#development--testing]
**Generate Test Data:**
```text
1. Arc: Generate Test Data
2. Select: CPU Metrics
3. Rows: 100,000
4. Database: dev
5. Time Range: Last 24 hours
Result: Realistic CPU metrics for testing
```
**Query the Data:**
```sql
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(usage_user) as avg_cpu
FROM dev.cpu
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;
```
### Data analysis [#data-analysis]
Create analysis notebooks (`.arcnb`) with:
* Documentation in Markdown
* Parameterized SQL queries
* Inline results and charts
* Export to Markdown reports
**Example Use Case:**
```markdown
# Weekly Performance Report
Variables:
- week_start = 2025-01-10
- database = prod
## CPU Trends
SELECT
DATE_TRUNC('day', time) as day,
AVG(usage_user) as avg_cpu
FROM ${database}.cpu
WHERE time >= '${week_start}'
GROUP BY day
ORDER BY day;
```
### Production monitoring [#production-monitoring]
**Create Alert:**
```text
Name: High Memory Usage
Query: SELECT AVG(used_percent) FROM prod.mem WHERE time > NOW() - INTERVAL '5 minutes'
Condition: greater_than
Threshold: 90
Interval: 60s
→ Desktop notification when memory exceeds 90%
```
### Data migration [#data-migration]
**Import CSV Files:**
```text
1. Arc: Import CSV
2. Select file: server_metrics.csv
3. Auto-detect: delimiter, headers
4. Set timestamp column: timestamp
5. Target: prod.imported_metrics
6. Batch size: 10,000
→ Import complete with progress tracking
```
## Performance [#performance]
* **Query Results**: Displays up to 1,000 rows instantly
* **CSV Import**: throughput depends on file size and host hardware
* **Data Generator**: throughput depends on the dataset and host hardware
* **Batch Processing**: Handles millions of rows with progress tracking
## Troubleshooting [#troubleshooting]
### Cannot connect to Arc server [#cannot-connect-to-arc-server]
```bash
# 1. Verify Arc is running
curl http://localhost:8000/health
# 2. Check connection details
- Host: localhost
- Port: 8000
- Protocol: http
# 3. Verify token
Arc: Verify Token (from Command Palette)
```
### Query timeout [#query-timeout]
**Solutions:**
1. Add time filters:
```sql
WHERE time > NOW() - INTERVAL '1 hour'
```
2. Add `LIMIT` clause:
```sql
LIMIT 1000
```
3. Check Arc server performance:
```bash
curl http://localhost:8000/api/v1/compaction/trigger \
-H "Authorization: Bearer $ARC_TOKEN"
```
### CSV import fails [#csv-import-fails]
**Common Issues:**
1. **Encoding**: Ensure UTF-8 encoding
2. **Delimiter**: Verify delimiter is correct (auto-detect usually works)
3. **File Size**: Try smaller file first to test
4. **Timestamp Format**: Ensure timestamp column is recognized
**Check Import Settings:**
```bash
File encoding: UTF-8
Delimiter: , (comma)
Headers: First row
Timestamp column: time
Format: ISO 8601 or Unix timestamp
```
### Extension not activating [#extension-not-activating]
1. **Check VS Code version**: Requires 1.85.0 or higher
2. **View Output**: View → Output → Arc Database Manager
3. **Reload Window**: Ctrl+Shift+P → Reload Window
4. **Reinstall**: Uninstall and reinstall extension
### IntelliSense not working [#intellisense-not-working]
1. **Refresh Schema**: Right-click in Arc Explorer → Refresh
2. **Reconnect**: Disconnect and reconnect to server
3. **Check Connection**: Ensure server is connected (status bar)
## Requirements [#requirements]
* **VS Code**: Version 1.85.0 or higher
* **Arc Database**: Running instance (v1.0.0+)
* **Authentication Token**: Valid Arc API token
## Release notes [#release-notes]
### 0.2.0 - Latest [#020---latest]
**New Features:**
* Auto-qualified table names in queries
* Right-click queries include database prefix (e.g., `prod.cpu`)
**Improvements:**
* Fixed query generation to read metadata correctly
* All context menu queries now work without manual editing
### 0.1.9 [#019]
**⚠️ Breaking Changes:**
* Updated all API endpoints to `/api/v1/` prefix
* Requires Arc v1.0.0 or later
* Not compatible with pre-v1.0 Arc servers
**Migration:**
1. Upgrade Arc to v1.0.0+
2. Update extension to v0.1.9
3. Reconnect to Arc server
## Resources [#resources]
* **[VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=basekick-labs.arc-db-manager)**
* **[GitHub Repository](https://github.com/basekick-labs/arc-vscode-extension)**
* **[Arc Documentation](https://docs.basekick.net)**
* **[Report Issues](https://github.com/basekick-labs/arc-vscode-extension/issues)**
## Next steps [#next-steps]
* **[Getting Started with Arc](/arc/getting-started/)** - Install and configure Arc
* **[Query API Reference](/arc/api-reference/overview/)** - Learn Arc SQL
* **[Grafana Integration](/arc/integrations/grafana/)** - Build dashboards
* **[Telegraf Integration](/arc/integrations/telegraf/)** - Collect system metrics
***
**Enjoy using Arc Database Manager!**
Made with ❤️ by [Basekick Labs](https://github.com/basekick-labs)
# Migrate from ClickHouse (/arc/migration/clickhouse)
This guide walks you through moving a ClickHouse workload to Arc: standing up Arc, bulk-loading your table history, moving ongoing writes, and translating your queries. Every command below was tested end-to-end against **ClickHouse 26.6.1** and **Arc 26.06.3**.
ClickHouse and Arc are both fast columnar analytical engines, so this is a migration between peers, not an escape from a slow system. Teams move for three reasons:
* **Operational simplicity.** Arc is a single Go binary with embedded consensus. No ZooKeeper or ClickHouse Keeper, no MergeTree merge tuning, no multi-process cluster to babysit.
* **Ingestion throughput.** Arc sustains higher write throughput than ClickHouse on the same hardware, and accepts small batches (from \~1,000 rows) instead of ClickHouse's 100,000-row minimums.
* **Portable storage.** Arc stores every table as standard Apache Parquet you own and can read with any Parquet tool. MergeTree is readable only by ClickHouse.
On ClickBench with the same storage format (Parquet) and the same hardware, Arc matches or beats ClickHouse and wins every cold run. ClickHouse's **native** MergeTree format is faster on hot analytical queries. Migrate for ingestion throughput, operational simplicity, cold-start performance, and portable storage, not for a raw hot-query speedup over native ClickHouse.
## How ClickHouse concepts map to Arc [#how-clickhouse-concepts-map-to-arc]
| ClickHouse | Arc | Notes |
| --------------------------------------------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| MergeTree table | Measurement | A queryable table. No `ENGINE`, no `ORDER BY`/`PARTITION BY` to port. |
| `CREATE TABLE ... ENGINE = MergeTree` | (none) | Arc is schema-on-write. A measurement and its columns appear on first write; there is no DDL. |
| `DateTime` / `DateTime64` column | `time` column | Arc stores the timestamp as a `time` column. See [Step 3](#step-3-migrate-historical-data) for the clean export form. |
| `Nullable(T)`, `LowCardinality(T)`, `Enum` | plain column of `T` | These are storage/encoding wrappers. They land as ordinary nullable / string / value columns. |
| `toStartOfInterval()`, `toStartOfHour()` | `time_bucket()`, `date_trunc()` | Standard equivalents, tested to return identical results. |
| `argMax(v, t)` / `argMin(v, t)` | `arg_max(v, t)` / `arg_min(v, t)` | Direct rename. |
| `SummingMergeTree` / `AggregatingMergeTree`, materialized views | Scheduled rollups | No table engines and no insert-time materialized views; run a bucketed aggregation on a schedule and write the rollup to a measurement. |
| `MergeTree` proprietary storage | Apache Parquet | Portable Parquet you own, queryable in place by any Parquet tool. |
## Step 0: install and run Arc [#step-0-install-and-run-arc]
Install with Homebrew (Apple Silicon; the query engine is statically linked, so there are no runtime dependencies):
```bash
brew install basekick-labs/tap/arc
```
Start Arc in the foreground:
```bash
arc
```
On first run Arc prints a one-time admin token to **stderr**:
```bash
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save that token, it is not shown again. To set a known token instead of a generated one, export `ARC_AUTH_BOOTSTRAP_TOKEN` (minimum 32 characters) before the first start:
```bash
ARC_AUTH_BOOTSTRAP_TOKEN="your-32-char-or-longer-secret-token" arc
```
Download the latest `.deb` (Debian/Ubuntu) or `.rpm` (RHEL/Fedora) from GitHub Releases and install it. The package registers an `arc` systemd service.
```bash
# Debian/Ubuntu
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc_${LATEST}_amd64.deb
sudo dpkg -i arc_${LATEST}_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
Read the first-run admin token from the journal:
```bash
sudo journalctl -u arc | grep -i "admin"
```
```bash
docker run -d --name arc -p 8000:8000 \
-e ARC_AUTH_BOOTSTRAP_TOKEN="your-32-char-or-longer-secret-token" \
ghcr.io/basekick-labs/arc:latest
```
Read the first-run admin token from the container logs (omit `ARC_AUTH_BOOTSTRAP_TOKEN` above if you want Arc to generate one):
```bash
docker logs arc 2>&1 | grep -i "admin"
```
Arc listens on port **8000**. Confirm it's up:
```bash
curl http://localhost:8000/health
# {"status":"ok",...}
```
Arc reads an optional `arc.toml` (searched in the current directory, then `/etc/arc/`, then `$HOME/.arc/`) and environment variables prefixed with `ARC_` (a TOML key `section.key` maps to `ARC_SECTION_KEY`). To disable anonymous usage telemetry, set `ARC_TELEMETRY_ENABLED=false`. See [Authentication](/arc/configuration/authentication/) and [Native Installation](/arc/installation/native/) for the full reference.
## Step 1: set your Arc token [#step-1-set-your-arc-token]
Every request below authenticates with a bearer token. Export it once:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
Verify it works:
```bash
curl http://localhost:8000/api/v1/auth/verify \
-H "Authorization: Bearer $ARC_TOKEN"
```
For production, create scoped write tokens rather than reusing the admin token, see [Authentication](/arc/configuration/authentication/).
## Step 2: move ongoing writes [#step-2-move-ongoing-writes]
ClickHouse shares no wire protocol with Arc. ClickHouse clients write over the native TCP protocol (port 9000) or HTTP `INSERT` (port 8123); Arc ingests over its own HTTP API. There is no drop-in driver compatibility, so plan for a real change to your write path.
* **If you already ship through Telegraf**, point the [Arc output plugin](/arc/integrations/telegraf/) at Arc and drop the ClickHouse output. Note that ClickHouse deployments are frequently fed directly from application code or Kafka rather than Telegraf, so confirm this applies to you.
* **Otherwise**, re-target your producers at Arc's HTTP write API: the [MessagePack columnar protocol](/arc/api-reference/overview/) is the fastest, and the [Line Protocol endpoints](/arc/data-import/line-protocol/) accept InfluxDB-style writes. Kafka pipelines re-point their sink; application writers change their client.
Keep ClickHouse authoritative while you backfill history into Arc and shadow-read to compare results. Cut over only once you trust Arc.
## Step 3: migrate historical data [#step-3-migrate-historical-data]
Export each table with `clickhouse-client`, then bulk-import into Arc. Both CSV and Parquet work; Parquet is the cleanest because column types carry through.
### Option A: Parquet (recommended) [#option-a-parquet-recommended]
ClickHouse exports `DateTime` as a proper Parquet timestamp, so Arc reads it with no format flags:
```bash
clickhouse-client --output_format_parquet_string_as_string=1 \
--query "SELECT * FROM cpu ORDER BY time FORMAT Parquet" > cpu.parquet
```
```bash
curl -X POST "http://localhost:8000/api/v1/import/parquet?measurement=cpu&time_column=time" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: mydb" \
-F "file=@cpu.parquet"
```
### Option B: CSV [#option-b-csv]
Export with `CSVWithNames` (header row). Force ISO/UTC timestamps so the import is unambiguous:
```bash
clickhouse-client --date_time_output_format=iso \
--query "SELECT * FROM cpu ORDER BY time FORMAT CSVWithNames" > cpu.csv
# time column becomes 2023-11-14T22:13:20Z (RFC 3339, UTC)
```
```bash
curl -X POST "http://localhost:8000/api/v1/import/csv?measurement=cpu&time_column=time" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: mydb" \
-F "file=@cpu.csv"
```
Response (either option):
```json
{
"status": "ok",
"result": {
"database": "mydb",
"measurement": "cpu",
"rows_imported": 10000,
"time_range_min": "2023-11-14T22:13:20Z",
"time_range_max": "2023-11-15T00:59:59Z",
"columns": ["time", "host", "usage_idle", "usage_user"],
"duration_ms": 11
}
}
```
ClickHouse's default CSV output is the space form `2023-11-14 22:13:20` with **no offset**. Arc accepts that form and treats it as UTC, so it imports directly **when your `DateTime` column is UTC**. But ClickHouse renders `DateTime` in the column's declared timezone, so a non-UTC column produces a naive string that Arc would read as UTC and silently shift. Passing `--date_time_output_format=iso` (Option B) forces `...Z` UTC and removes the ambiguity. Alternatively, export epoch seconds with `toUnixTimestamp(time)` and import with `time_format=epoch_s`. All four forms were tested and import cleanly; ISO is the safe default.
Check table sizes before you start:
```sql
SELECT name, total_rows, formatReadableSize(total_bytes) AS size
FROM system.tables WHERE database = currentDatabase();
```
For large tables, export in time ranges (`WHERE time >= '2024-01-01' AND time < '2024-02-01'`) to stay under the 500 MB import limit and to parallelize. Arc auto-detects gzip on CSV uploads. Script export-then-import in a loop over your tables.
**Verify the load** against the source. In Arc, set `X-Arc-Database` so you can use bare table names:
```bash
curl -X POST "http://localhost:8000/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-H "X-Arc-Database: mydb" \
-d '{"sql":"SELECT count(*), min(time), max(time) FROM cpu"}'
```
Compare with ClickHouse: `SELECT count(), min(time), max(time) FROM cpu`. Counts and time range should match.
## Step 4: translate your queries [#step-4-translate-your-queries]
Arc runs standard SQL. ClickHouse's SQL dialect diverges more than most, but the common time-series patterns have direct, tested equivalents.
Set the `X-Arc-Database` header on your query requests and you keep bare table names (`FROM cpu` instead of `FROM mydb.cpu`). This is the recommended pattern, it keeps your SQL closest to the original and takes Arc's leaner query path.
| ClickHouse | Arc |
| --------------------------------------------- | ------------------------------------------------------------- |
| `toStartOfInterval(time, INTERVAL 1 HOUR)` | `time_bucket(INTERVAL '1 hour', time)` |
| `toStartOfHour(time)` / `toStartOfDay(time)` | `date_trunc('hour', time)` / `date_trunc('day', time)` |
| `argMax(value, time)` / `argMin(value, time)` | `arg_max(value, time)` / `arg_min(value, time)` |
| `uniqExact(x)` | `count(DISTINCT x)` |
| `uniq(x)` / `uniqCombined(x)` | `approx_count_distinct(x)` |
| `quantile(0.95)(x)` | `quantile_cont(x, 0.95)` |
| `sumIf(x, cond)` / `countIf(cond)` | `sum(x) FILTER (WHERE cond)` / `count(*) FILTER (WHERE cond)` |
| `FINAL`, `PREWHERE`, `SAMPLE` | drop or rewrite (no equivalent) |
### Downsampling: `toStartOfInterval` to `time_bucket` [#downsampling-tostartofinterval-to-time_bucket]
```sql
-- ClickHouse
SELECT toStartOfInterval(time, INTERVAL 1 HOUR) AS bucket, avg(usage_idle)
FROM cpu GROUP BY bucket ORDER BY bucket;
-- Arc
SELECT time_bucket(INTERVAL '1 hour', time) AS bucket, avg(usage_idle)
FROM cpu GROUP BY 1 ORDER BY 1;
```
Tested identical: same buckets, same averages, same counts.
### Latest per series and conditional aggregates [#latest-per-series-and-conditional-aggregates]
```sql
-- ClickHouse: argMax + sumIf
SELECT host, argMax(usage_idle, time) FROM cpu GROUP BY host;
SELECT sumIf(usage_user, usage_idle > 90) FROM cpu;
-- Arc: arg_max + FILTER
SELECT host, arg_max(usage_idle, time) FROM cpu GROUP BY host;
SELECT sum(usage_user) FILTER (WHERE usage_idle > 90) FROM cpu;
```
Both tested to return identical results.
ClickHouse's `quantile(0.95)(x)` is an **approximate** aggregate (reservoir sampling). Arc's `quantile_cont(x, 0.95)` is exact continuous interpolation, so the two will not match to the last digit. That is expected. Use `quantile_cont` for an exact percentile; if you specifically want to reproduce ClickHouse's exact-quantile behavior, ClickHouse's own `quantileExact(0.95)(x)` is the closer reference.
### ClickHouse-only keywords [#clickhouse-only-keywords]
`FINAL` (query-time dedup of `ReplacingMergeTree`), `PREWHERE` (a read optimization that is semantically just a filter), and `SAMPLE` have no standard equivalent. Drop `PREWHERE` into the regular `WHERE`; drop `FINAL` and do any dedup explicitly with `arg_max` or a `ROW_NUMBER()` window; remove `SAMPLE` or replace it with an explicit sampling predicate.
See the [SQL Querying Guide](/arc/guides/querying/) for the full function reference.
## Differences to know [#differences-to-know]
* **No wire protocol.** Arc ingests over HTTP. Ongoing writes move to Telegraf-to-Arc or the HTTP write API, not a connection-string swap. See [Step 2](#step-2-move-ongoing-writes).
* **No engines or DDL.** Arc is schema-on-write. There is no `MergeTree`, no `ORDER BY`/`PARTITION BY`, no `CREATE TABLE`. A measurement appears on first write.
* **No aggregate combinators or insert-time materialized views.** `-If` maps to `FILTER`; `-State`/`-Merge` and `SummingMergeTree`/`AggregatingMergeTree` rollups become scheduled aggregation jobs.
* **The query API is read-only.** `POST /api/v1/query` rejects write verbs (`INSERT`, `DELETE`, `DROP`, and the file-reading functions). Ingest through the write and import endpoints.
* **Portable storage.** Arc's Parquet files are yours, queryable in place by any Parquet-compatible tool, or moved to S3/MinIO/Azure without an export step.
## Next steps [#next-steps]
* [SQL Querying Guide](/arc/guides/querying/) - full SQL function reference and query patterns
* [CSV Import](/arc/data-import/csv/) - all import parameters and options
* [Parquet Import](/arc/data-import/parquet/) - importing Parquet exports directly
* [Telegraf integration](/arc/integrations/telegraf/) - the native `outputs.arc` output plugin
* [Grafana data source](/arc/integrations/grafana/) - dashboards on your migrated data
# Migration (/arc/migration)
Each guide covers the same three steps: move ongoing writes, bulk-load the history you already have, and translate the queries you run today.
# Migrate from InfluxDB (/arc/migration/influxdb)
This guide walks you through moving an InfluxDB workload to Arc: re-pointing live ingestion, bulk-loading your history, and translating your queries. It covers **InfluxDB 1.x, 2.x, and 3.x**. Every command was tested end-to-end against **InfluxDB 1.8.10** and **Arc 26.06.3**.
InfluxDB is the smoothest migration to Arc, because Arc speaks InfluxDB Line Protocol natively:
* **Live ingestion is a URL change.** Point Telegraf or any InfluxDB client at Arc's `/write` (1.x) or `/api/v2/write` (2.x) endpoint. Dual-write during the migration, cut over when ready. No agent changes, no downtime.
* **History has a purpose-built tool.** [`tsm2arc`](https://github.com/Basekick-Labs/tsm2arc) reads InfluxDB 1.x/2.x TSM files directly off disk and loads Arc, reconstructing multi-field points correctly. Line Protocol export/import works for every version including 3.x.
* **Queries move to standard SQL.** InfluxQL and Flux map to standard SQL. Flux is deprecated in InfluxDB 3 anyway.
## How InfluxDB concepts map to Arc [#how-influxdb-concepts-map-to-arc]
| InfluxDB | Arc | Notes |
| ----------------------------- | -------------- | ----------------------------------------------------------------------------------------- |
| Measurement | Measurement | Same name, a queryable table. |
| Tag / field | Column | Tags and fields both become columns. |
| Bucket (2.x) / database (1.x) | Database | The target Arc database. |
| Organization (2.x) | (ignored) | Arc has no org concept; the `org` parameter is accepted and ignored. |
| `time` | `time` column | Timestamps convert losslessly to Arc's internal microsecond precision. |
| InfluxQL / Flux | Standard SQL | Direct equivalents for the common patterns. See [Step 4](#step-4-translate-your-queries). |
| TSM / Parquet (3.x) storage | Apache Parquet | Portable Parquet you own, queryable by any Parquet tool. |
## Step 0: install and run Arc [#step-0-install-and-run-arc]
Install with Homebrew (Apple Silicon; the query engine is statically linked, so there are no runtime dependencies):
```bash
brew install basekick-labs/tap/arc
```
Start Arc in the foreground:
```bash
arc
```
On first run Arc prints a one-time admin token to **stderr**:
```bash
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save that token, it is not shown again. To set a known token instead of a generated one, export `ARC_AUTH_BOOTSTRAP_TOKEN` (minimum 32 characters) before the first start:
```bash
ARC_AUTH_BOOTSTRAP_TOKEN="your-32-char-or-longer-secret-token" arc
```
Download the latest `.deb` (Debian/Ubuntu) or `.rpm` (RHEL/Fedora) from GitHub Releases and install it. The package registers an `arc` systemd service.
```bash
# Debian/Ubuntu
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc_${LATEST}_amd64.deb
sudo dpkg -i arc_${LATEST}_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
Read the first-run admin token from the journal:
```bash
sudo journalctl -u arc | grep -i "admin"
```
```bash
docker run -d --name arc -p 8000:8000 \
-e ARC_AUTH_BOOTSTRAP_TOKEN="your-32-char-or-longer-secret-token" \
ghcr.io/basekick-labs/arc:latest
```
Read the first-run admin token from the container logs (omit `ARC_AUTH_BOOTSTRAP_TOKEN` above if you want Arc to generate one):
```bash
docker logs arc 2>&1 | grep -i "admin"
```
Arc listens on port **8000**. Confirm it's up:
```bash
curl http://localhost:8000/health
# {"status":"ok",...}
```
Arc reads an optional `arc.toml` (searched in the current directory, then `/etc/arc/`, then `$HOME/.arc/`) and environment variables prefixed with `ARC_` (a TOML key `section.key` maps to `ARC_SECTION_KEY`). To disable anonymous usage telemetry, set `ARC_TELEMETRY_ENABLED=false`. See [Authentication](/arc/configuration/authentication/) and [Native Installation](/arc/installation/native/) for the full reference.
## Step 1: set your Arc token [#step-1-set-your-arc-token]
Every request below authenticates with a bearer token. Export it once:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
Verify it works:
```bash
curl http://localhost:8000/api/v1/auth/verify \
-H "Authorization: Bearer $ARC_TOKEN"
```
For production, create scoped write tokens rather than reusing the admin token, see [Authentication](/arc/configuration/authentication/).
## Step 2: re-point live ingestion (the easy win) [#step-2-re-point-live-ingestion-the-easy-win]
Arc accepts InfluxDB Line Protocol on InfluxDB-compatible endpoints, so live writers move with a URL change. See [InfluxDB Client Compatibility](/arc/integrations/influxdb-clients/) for the full client and endpoint reference.
| InfluxDB endpoint | Arc endpoint | For |
| ----------------- | ----------------------------------------------- | ----------- |
| `/write` | `/write?db=` | 1.x clients |
| `/api/v2/write` | `/api/v2/write?bucket=&org=` | 2.x clients |
```bash
# InfluxDB 1.x style
curl -X POST "http://localhost:8000/write?db=metrics&precision=ns" \
-H "Authorization: Bearer $ARC_TOKEN" \
--data-binary 'cpu,host=server01 usage_idle=95.0,usage_user=3.2 1700000000000000000'
# InfluxDB 2.x style (bucket becomes the database; org is ignored)
curl -X POST "http://localhost:8000/api/v2/write?bucket=metrics&org=myorg&precision=ns" \
-H "Authorization: Token $ARC_TOKEN" \
--data-binary 'cpu,host=server02 usage_idle=88.0,usage_user=5.1 1700000001000000000'
```
**Dual-write with Telegraf** so new data lands in both systems while you migrate history, then drop the InfluxDB output at cutover:
```toml
# Keep InfluxDB during the migration
[[outputs.influxdb_v2]]
urls = ["http://influxdb:8086"]
token = "$INFLUX_TOKEN"
organization = "myorg"
bucket = "metrics"
# New data also goes to Arc
[[outputs.arc]]
url = "http://localhost:8000/api/v1/write/msgpack"
api_key = "$ARC_TOKEN"
content_encoding = "gzip"
database = "metrics"
```
The Arc Telegraf output requires Telegraf 1.33+. See the [Telegraf integration](/arc/integrations/telegraf/).
## Step 3: migrate historical data [#step-3-migrate-historical-data]
Two paths. **Line Protocol export/import** works for every InfluxDB version and is the universal path. For large **1.x/2.x** datasets, **tsm2arc** is faster, resumable, and reconstructs multi-field points correctly.
### Line Protocol export and import [#line-protocol-export-and-import]
Export to a `.lp` file, then import with `POST /api/v1/import/lp`. The export command depends on your InfluxDB version.
Use `influx_inspect export` (reads TSM and WAL directly, works offline):
```bash
influx_inspect export \
-datadir /var/lib/influxdb/data \
-waldir /var/lib/influxdb/wal \
-database mydb -lponly -out mydb.lp
```
Export with a Flux query, pivoting fields back into line-protocol-friendly rows:
```bash
influx query --raw 'from(bucket:"mydb")
|> range(start: 0)
|> pivot(rowKey:["_time"], columnKey:["_field"], valueColumn:"_value")' > mydb.lp
```
Use the `influxdb3` CLI with line-protocol output:
```bash
influxdb3 query --database mydb --format lp \
"SELECT * FROM cpu" > cpu.lp
```
Import into Arc:
```bash
curl -X POST "http://localhost:8000/api/v1/import/lp" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: mydb" \
-F "file=@mydb.lp"
```
`influx_inspect export` writes one Line Protocol line per field, so a two-field point becomes two rows in Arc, each with one field populated and the other null. For single-field measurements this is fine. For multi-field measurements, either use **tsm2arc** (below), which rejoins fields into one point, or export with an InfluxQL `SELECT` per measurement so the fields stay together. See the full mechanics in [Line Protocol Bulk Import](/arc/data-import/line-protocol/).
### Bulk migration with tsm2arc (recommended for 1.x/2.x) [#bulk-migration-with-tsm2arc-recommended-for-1x2x]
[tsm2arc](https://github.com/Basekick-Labs/tsm2arc) (Apache-2.0) reads InfluxDB 1.x/2.x TSM and WAL files **directly off disk**, reconstructs each multi-field point into a single Line Protocol record, and streams it into Arc's `/api/v1/import/lp` endpoint with resumable checkpointing. It does not need a running InfluxDB, which makes it ideal for cold volumes and EBS snapshots. (InfluxDB 3.x stores Parquet, not TSM, so use the Line Protocol path above for 3.x.)
Install:
```bash
go install github.com/basekick-labs/tsm2arc/cmd/tsm2arc@latest
# or a release binary from https://github.com/basekick-labs/tsm2arc/releases
# or: docker run --rm ghcr.io/basekick-labs/tsm2arc:latest --version
```
Dry-run first (extracts and counts, writes nothing):
```bash
tsm2arc --datadir /var/lib/influxdb/data --waldir /var/lib/influxdb/wal \
--dry-run --sample 5
```
The dry-run reports discovered shards and reconstructed points, for example `points: 10000 fields: 20000` with rejoined lines like `cpu,host=server01 usage_idle=50,usage_user=10 1700000000000000000`.
Then the real run:
```bash
export ARC_TOKEN=''
tsm2arc \
--datadir /var/lib/influxdb/data \
--waldir /var/lib/influxdb/wal \
--arc-url http://localhost:8000 \
--token "$ARC_TOKEN" \
--workers 2 \
--checkpoint /var/lib/tsm2arc/migration.checkpoint.db
```
* **Always pass `--waldir`.** InfluxDB does not flush the WAL to TSM on a normal shutdown, so recent shards can live entirely in `.wal` files. Omitting `--waldir` silently misses them.
* **Multi-writer cluster (Arc Enterprise)? Put an L7 load balancer in front.**
A Kubernetes ClusterIP Service balances per TCP connection, and tsm2arc reuses
keep-alive connections — so without an HTTP-aware balancer (ingress, Envoy,
ALB) nearly all import traffic pins to one writer pod while the rest idle. See
[clustering: multi-writer](/arc-enterprise/configuration/clustering/#pattern-2--shared-object-storage-multi-writer).
* **Check the writer's WAL volume before a bulk load.** The WAL absorbs
ingest-rate × flush-lag; a small volume (the chart default is 10Gi) fills in
minutes at bulk rates and a full WAL volume prevents the writer from booting.
Size it for the burst, or disable the WAL for the migration window
(`ARC_WAL_ENABLED=false`) and re-enable it afterwards.
* **Size `--workers` against Arc's RAM, not the migration host.** Arc buffers each import server-side, roughly 1 to 1.3 GB per concurrent worker at the default chunk size. The default of 2 is conservative; raise it if Arc has headroom.
* **Resume by re-running the identical command.** Progress is checkpointed per shard in SQLite; completed shards are skipped. Changing `--chunk-bytes`, `--start`/`--end`, or `--db-map` between runs is refused to keep the checkpoint consistent.
* **Rename databases** with `--db-map old=new`, and filter with `--database-filter` or `--start`/`--end` (RFC3339 UTC).
* Supports InfluxDB 1.7/1.8 and 2.0 to 2.7. For 2.x it reads `influxd.bolt` to recover bucket names.
**Verify the load** against the source. In Arc, set `X-Arc-Database` so you can use bare table names:
```bash
curl -X POST "http://localhost:8000/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-H "X-Arc-Database: mydb" \
-d '{"sql":"SELECT count(*), min(time), max(time) FROM cpu"}'
```
Compare with InfluxDB: `SELECT count(usage_idle), min(time), max(time) FROM cpu`. Counts and time range should match.
## Step 4: translate your queries [#step-4-translate-your-queries]
Arc runs standard SQL. InfluxQL and Flux map to it directly for the common patterns. Set the `X-Arc-Database` header on your query requests and you keep bare table names (`FROM cpu` instead of `FROM mydb.cpu`).
| InfluxQL / Flux | Arc |
| ----------------------------------------------- | ----------------------------------------------------- |
| `GROUP BY time(1h)` | `time_bucket(INTERVAL '1 hour', time)` + `GROUP BY 1` |
| `MEAN(field)` / `SUM` / `MAX` | `avg(field)` / `sum` / `max` |
| `LAST(field)` / `FIRST(field)` | `arg_max(field, time)` / `arg_min(field, time)` |
| `PERCENTILE(field, 95)` | `quantile_cont(field, 0.95)` |
| `GROUP BY "tag"` | `GROUP BY tag` |
| Flux `\|> aggregateWindow(every: 1h, fn: mean)` | `time_bucket(INTERVAL '1 hour', time)` + `avg(...)` |
| Flux (deprecated in InfluxDB 3) | standard SQL |
### Downsampling [#downsampling]
```sql
-- InfluxQL
SELECT MEAN(usage_idle) FROM cpu GROUP BY time(1h);
-- Arc
SELECT time_bucket(INTERVAL '1 hour', time) AS bucket, avg(usage_idle)
FROM cpu GROUP BY 1 ORDER BY 1;
```
Tested identical: same buckets, same averages.
### Last value per series [#last-value-per-series]
```sql
-- InfluxQL
SELECT LAST(usage_idle) FROM cpu GROUP BY host;
-- Arc
SELECT host, arg_max(usage_idle, time) FROM cpu GROUP BY host;
```
Tested identical.
InfluxQL `PERCENTILE(field, 95)` and Arc's `quantile_cont(field, 0.95)` use different interpolation, so they can differ in the last digits (for example 97.49 vs 97.4905). Both are correct percentiles; the small difference is expected.
See the [SQL Querying Guide](/arc/guides/querying/) for the full function reference.
## Differences to know [#differences-to-know]
* **No Flux.** Arc uses standard SQL. Flux is deprecated in InfluxDB 3, so this aligns with InfluxData's own direction.
* **Organizations are ignored.** The 2.x `org` parameter is accepted and ignored; a bucket maps to an Arc database.
* **Retention and tasks.** InfluxDB retention policies map to Arc's retention API; continuous tasks map to Arc continuous queries.
* **The query API is read-only.** `POST /api/v1/query` rejects write verbs. Ingest through the write and import endpoints.
* **Portable storage.** Arc's Parquet files are yours, queryable in place by any Parquet-compatible tool.
## Next steps [#next-steps]
* [InfluxDB Client Compatibility](/arc/integrations/influxdb-clients/) - endpoint mapping, auth, and supported clients
* [Line Protocol Bulk Import](/arc/data-import/line-protocol/) - the `/api/v1/import/lp` reference
* [tsm2arc on GitHub](https://github.com/Basekick-Labs/tsm2arc) - the TSM bulk migration tool
* [SQL Querying Guide](/arc/guides/querying/) - full SQL function reference
* [Telegraf integration](/arc/integrations/telegraf/) - the native `outputs.arc` output plugin
* [Grafana data source](/arc/integrations/grafana/) - dashboards on your migrated data
# Migrate from QuestDB (/arc/migration/questdb)
This guide walks you through moving a QuestDB workload to Arc: standing up Arc, re-pointing your live ingestion, bulk-loading your historical data, and translating your queries. Every command below was tested end-to-end against **QuestDB 9.4.3** and **Arc 26.06.3**.
The migration has three moving parts, and none of them require rewriting your application:
1. **Live ingestion:** QuestDB and Arc both speak InfluxDB Line Protocol (ILP), so Telegraf and ILP clients re-point to Arc with a URL change.
2. **Historical data:** export each QuestDB table to CSV via its `/exp` endpoint and bulk-import it into Arc.
3. **Queries:** QuestDB's time-series SQL extensions map to the standard SQL that Arc runs natively.
## How QuestDB concepts map to Arc [#how-questdb-concepts-map-to-arc]
| QuestDB | Arc | Notes |
| -------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Table | Measurement | Same thing, a queryable table. Queried as `database.measurement`. |
| Designated timestamp | `time` column | Arc names the timestamp column `time`. It's a normal column you filter and order on explicitly, there's no implicit "designated" timestamp. |
| `SYMBOL` | `VARCHAR` | String/tag columns become standard strings. |
| Native column format | Apache Parquet | Arc stores every measurement as compressed Parquet you own and can query in place with any Parquet tool, no export step. |
| QuestDB SQL (`SAMPLE BY`, `LATEST ON`) | Standard SQL (`time_bucket`, `DISTINCT ON`) | Full analytical SQL surface: window functions, CTEs, joins. See [Translate your queries](#step-4-translate-your-queries). |
| Schema declared / evolved on write | Schema inferred on write | A new measurement or column appears on first write; no DDL required. |
## Step 0: install and run Arc [#step-0-install-and-run-arc]
Install with Homebrew (Apple Silicon; the query engine is statically linked, so there are no runtime dependencies):
```bash
brew install basekick-labs/tap/arc
```
Start Arc in the foreground:
```bash
arc
```
On first run Arc prints a one-time admin token to **stderr**:
```bash
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save that token, it is not shown again. To set a known token instead of a generated one, export `ARC_AUTH_BOOTSTRAP_TOKEN` (minimum 32 characters) before the first start:
```bash
ARC_AUTH_BOOTSTRAP_TOKEN="your-32-char-or-longer-secret-token" arc
```
Download the latest `.deb` (Debian/Ubuntu) or `.rpm` (RHEL/Fedora) from GitHub Releases and install it. The package registers an `arc` systemd service.
```bash
# Debian/Ubuntu
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc_${LATEST}_amd64.deb
sudo dpkg -i arc_${LATEST}_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
Read the first-run admin token from the journal:
```bash
sudo journalctl -u arc | grep -i "admin"
```
```bash
docker run -d --name arc -p 8000:8000 \
-e ARC_AUTH_BOOTSTRAP_TOKEN="your-32-char-or-longer-secret-token" \
ghcr.io/basekick-labs/arc:latest
```
Read the first-run admin token from the container logs (omit `ARC_AUTH_BOOTSTRAP_TOKEN` above if you want Arc to generate one):
```bash
docker logs arc 2>&1 | grep -i "admin"
```
Arc listens on port **8000**. Confirm it's up:
```bash
curl http://localhost:8000/health
# {"status":"ok",...}
```
Arc reads an optional `arc.toml` (searched in the current directory, then `/etc/arc/`, then `$HOME/.arc/`) and environment variables prefixed with `ARC_` (a TOML key `section.key` maps to `ARC_SECTION_KEY`). To disable anonymous usage telemetry, set `ARC_TELEMETRY_ENABLED=false`. See [Authentication](/arc/configuration/authentication/) and [Native Installation](/arc/installation/native/) for the full reference.
## Step 1: set your Arc token [#step-1-set-your-arc-token]
Every request below authenticates with a bearer token. Export it once:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
Verify it works:
```bash
curl http://localhost:8000/api/v1/auth/verify \
-H "Authorization: Bearer $ARC_TOKEN"
```
For production, create scoped write tokens rather than reusing the admin token, see [Authentication](/arc/configuration/authentication/).
## Step 2: re-point live ingestion [#step-2-re-point-live-ingestion]
QuestDB ingests via InfluxDB Line Protocol, and so does Arc. Any Telegraf pipeline or ILP client you already run keeps its exact payload, you only change the destination URL and add an Arc token.
Arc exposes an InfluxDB 1.x-compatible endpoint at `POST /write`. The measurement name in each line becomes the Arc measurement; the target database comes from the `db` query parameter (or the `X-Arc-Database` header):
```bash
curl -X POST "http://localhost:8000/write?db=metrics&precision=ns" \
-H "Authorization: Bearer $ARC_TOKEN" \
--data-binary 'cpu,host=server01 usage_idle=95.0,usage_user=3.2 1700000000000000000'
```
`precision` accepts `ns` (default), `us`, `ms`, or `s`.
If you send to QuestDB from Telegraf today, point the InfluxDB output (or the native [`outputs.arc`](/arc/integrations/telegraf/) plugin) at Arc's URL and set the token. No change to your inputs or metric names.
Arc buffers writes and flushes them on an interval. During migration you can force an immediate flush so data is queryable right away:
```bash
curl -X POST "http://localhost:8000/api/v1/write/line-protocol/flush" \
-H "Authorization: Bearer $ARC_TOKEN"
```
Confirm the data landed:
```bash
curl -X POST "http://localhost:8000/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-H "X-Arc-Database: metrics" \
-d '{"sql":"SELECT count(*) FROM cpu"}'
```
Setting the `X-Arc-Database` header lets you keep the same bare table names you use in QuestDB (`FROM cpu`) instead of `FROM metrics.cpu`. This is the recommended pattern, see [Step 4](#step-4-translate-your-queries).
The ILP field/measurement timestamp is stored in Arc's `time` column automatically. Tag values (like `host`) become string columns; numeric fields keep their types.
## Step 3: migrate historical data [#step-3-migrate-historical-data]
Export each QuestDB table to CSV through its REST `/exp` endpoint, then bulk-import the file into Arc.
**Export from QuestDB** (default REST port `9000`):
```bash
curl -G "http://localhost:9000/exp" \
--data-urlencode "query=SELECT * FROM cpu" \
-o cpu.csv
```
QuestDB writes a header row and RFC 3339 timestamps, e.g.:
```csv
"host","usage_idle","usage_user","timestamp"
"server01",75.572,24.311,"2023-11-14T22:13:20.000000Z"
```
Note the timestamp column is named **`timestamp`** (QuestDB's default designated-timestamp name).
**Import into Arc** with `POST /api/v1/import/csv`. Point `time_column` at QuestDB's `timestamp` column, Arc renames it to `time` in the output Parquet and auto-detects the RFC 3339 format:
```bash
curl -X POST "http://localhost:8000/api/v1/import/csv?measurement=cpu&time_column=timestamp" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: metrics" \
-F "file=@cpu.csv"
```
Response:
```json
{
"status": "ok",
"result": {
"database": "metrics",
"measurement": "cpu",
"rows_imported": 10000,
"partitions_created": 3,
"time_range_min": "2023-11-14T22:13:20Z",
"time_range_max": "2023-11-15T00:59:59Z",
"columns": ["host", "usage_idle", "usage_user", "time"],
"duration_ms": 10
}
}
```
Arc auto-detects the timestamp unit. QuestDB's `/exp` produces RFC 3339 strings, which import without a `time_format`. If a column instead holds numeric epochs, pass `time_format=epoch_s`, `epoch_ms`, `epoch_us`, or `epoch_ns` (or omit it to auto-detect by magnitude). Column types are inferred per column. Maximum file size is 500 MB, export large tables in time-ranged chunks with a `WHERE` clause on the timestamp.
For large migrations, script one export + import per table (and per time range for very large tables), then verify each with a `count(*)` and a `min/max(time)` check against QuestDB.
## Step 4: translate your queries [#step-4-translate-your-queries]
Arc runs standard SQL. QuestDB's time-series extensions have direct equivalents.
**Scope your queries with the `X-Arc-Database` header.** Set `X-Arc-Database` on the request and you write `FROM cpu`, the same table names you use in QuestDB, instead of `FROM database.cpu`. This is the recommended pattern: it keeps your SQL identical to QuestDB's and takes Arc's leaner query path, since the engine resolves the measurement directly instead of rewriting a cross-database reference.
```bash
curl -X POST "http://localhost:8000/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-H "X-Arc-Database: metrics" \
-d '{"sql":"SELECT count(*) FROM cpu"}'
```
Without the header, address tables as `database.measurement` (e.g. `FROM metrics.cpu`). The examples below use the header form and bare table names.
| QuestDB | Arc |
| ------------------------------------- | ----------------------------------------------------- |
| `SAMPLE BY 1h` | `time_bucket(INTERVAL '1 hour', time)` + `GROUP BY 1` |
| `SAMPLE BY 5m` | `time_bucket(INTERVAL '5 minutes', time)` |
| `LATEST ON ts PARTITION BY sym` | `DISTINCT ON (sym) ... ORDER BY sym, time DESC` |
| Designated timestamp (implicit order) | Explicit `WHERE time >= ... ORDER BY time` |
| `now()` | `now()` |
| `dateadd('d', -7, now())` | `now() - INTERVAL '7 days'` |
| `to_timezone(ts, 'Europe/Madrid')` | `time AT TIME ZONE 'Europe/Madrid'` |
### Downsampling: `SAMPLE BY` → `time_bucket` [#downsampling-sample-by--time_bucket]
QuestDB:
```sql
SELECT timestamp, avg(usage_idle), count()
FROM cpu
SAMPLE BY 1h
ORDER BY timestamp;
```
Arc (with `X-Arc-Database: metrics`):
```sql
SELECT time_bucket(INTERVAL '1 hour', time) AS bucket,
avg(usage_idle),
count(*)
FROM cpu
GROUP BY 1
ORDER BY 1;
```
Both return identical buckets, averages, and counts.
### Latest row per series: `LATEST ON` → `DISTINCT ON` [#latest-row-per-series-latest-on--distinct-on]
QuestDB:
```sql
SELECT host, timestamp, usage_idle
FROM cpu
LATEST ON timestamp PARTITION BY host;
```
Arc (with `X-Arc-Database: metrics`):
```sql
SELECT DISTINCT ON (host) host, time, usage_idle
FROM cpu
ORDER BY host, time DESC;
```
Or equivalently with a window function:
```sql
SELECT host, time, usage_idle
FROM (
SELECT host, time, usage_idle,
ROW_NUMBER() OVER (PARTITION BY host ORDER BY time DESC) AS rn
FROM cpu
)
WHERE rn = 1
ORDER BY host;
```
See the [SQL Querying Guide](/arc/guides/querying/) for the full function reference.
## Differences to know [#differences-to-know]
* **No proprietary query language.** Arc is standard SQL, there's no `SAMPLE BY`/`LATEST ON` syntax, but every pattern has a direct equivalent (above). In exchange you get the full analytical surface: CTEs, window functions, and complex joins run without a penalty.
* **Schema-on-write inference.** Types are inferred from the first write to a measurement; new columns appear automatically. There's no `CREATE TABLE` step.
* **Partition pruning.** Arc partitions Parquet by time. Filtering on the `time` column (`WHERE time >= ...`) prunes partitions and is the single biggest lever on query speed, keep a time filter on large scans.
* **The query API is read-only.** `POST /api/v1/query` rejects write verbs (`INSERT`, `DELETE`, `DROP`, `COPY`, `ATTACH`, and the file-reading functions). Ingest through the write and import endpoints, not through SQL.
* **Portable storage.** Arc's Parquet files are yours. You can query them directly with any Parquet-compatible tool, or move to S3/MinIO/Azure without an export step.
## Next steps [#next-steps]
* [SQL Querying Guide](/arc/guides/querying/), full SQL function reference and query patterns
* [CSV Import](/arc/data-import/csv/), all import parameters and options
* [Authentication](/arc/configuration/authentication/), creating scoped tokens for production
* [Telegraf integration](/arc/integrations/telegraf/), the native `outputs.arc` output plugin
* [Grafana data source](/arc/integrations/grafana/), dashboards on your migrated data
# Migrate from TimescaleDB (/arc/migration/timescaledb)
This guide walks you through moving a TimescaleDB workload to Arc: standing up Arc, bulk-loading your hypertable history, moving ongoing writes, and translating your queries. Every command below was tested end-to-end against **TimescaleDB 2.28.2 / PostgreSQL 16.14** and **Arc 26.06.3**.
TimescaleDB is PostgreSQL with time-series extensions. Two things make this migration smooth, and one thing takes real work:
* **Queries mostly carry over.** Arc runs standard SQL, and `time_bucket()` has the same name and argument order on both sides. Most dashboards and reports run with little or no change.
* **History is a `\copy` and a `curl`.** Hypertables export to CSV like any table; Arc imports the file directly.
* **Ongoing writes change path.** TimescaleDB apps write over the PostgreSQL wire protocol, which Arc does not speak. Arc ingests over HTTP. If you write through Telegraf this is a one-block config swap; if you write with SQL `INSERT`/`COPY` it is an application change. See [Step 2](#step-2-move-ongoing-writes).
## How TimescaleDB concepts map to Arc [#how-timescaledb-concepts-map-to-arc]
| TimescaleDB | Arc | Notes |
| ---------------------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Hypertable | Measurement | A queryable table. Chunking is transparent to export; in Arc, partitioning is handled by the storage engine. |
| `create_hypertable()`, `add_retention_policy()`, chunk DDL | (none) | Arc is schema-on-write. There is no `CREATE TABLE` or hypertable DDL; a measurement appears on first write. |
| `timestamptz` column | `time` column | Arc stores the timestamp as a `time` column. See [Step 3](#step-3-migrate-historical-data) for the export format that imports cleanly. |
| `time_bucket()` | `time_bucket()` | Same function, same argument order. Queries barely change. |
| `last(v, time)` / `first(v, time)` | `arg_max(v, time)` / `arg_min(v, time)` | Direct, tested equivalents. |
| Continuous aggregate (auto-refresh materialized view) | Scheduled rollup | No auto-incremental primitive; run a bucketed aggregation on a schedule and write the rollup to a measurement. |
| PostgreSQL heap storage | Apache Parquet | Portable Parquet you own, queryable in place by any Parquet tool. No `pg_dump` as your only export path. |
## Step 0: install and run Arc [#step-0-install-and-run-arc]
Install with Homebrew (Apple Silicon; the query engine is statically linked, so there are no runtime dependencies):
```bash
brew install basekick-labs/tap/arc
```
Start Arc in the foreground:
```bash
arc
```
On first run Arc prints a one-time admin token to **stderr**:
```bash
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save that token, it is not shown again. To set a known token instead of a generated one, export `ARC_AUTH_BOOTSTRAP_TOKEN` (minimum 32 characters) before the first start:
```bash
ARC_AUTH_BOOTSTRAP_TOKEN="your-32-char-or-longer-secret-token" arc
```
Download the latest `.deb` (Debian/Ubuntu) or `.rpm` (RHEL/Fedora) from GitHub Releases and install it. The package registers an `arc` systemd service.
```bash
# Debian/Ubuntu
LATEST=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST}/arc_${LATEST}_amd64.deb
sudo dpkg -i arc_${LATEST}_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
Read the first-run admin token from the journal:
```bash
sudo journalctl -u arc | grep -i "admin"
```
```bash
docker run -d --name arc -p 8000:8000 \
-e ARC_AUTH_BOOTSTRAP_TOKEN="your-32-char-or-longer-secret-token" \
ghcr.io/basekick-labs/arc:latest
```
Read the first-run admin token from the container logs (omit `ARC_AUTH_BOOTSTRAP_TOKEN` above if you want Arc to generate one):
```bash
docker logs arc 2>&1 | grep -i "admin"
```
Arc listens on port **8000**. Confirm it's up:
```bash
curl http://localhost:8000/health
# {"status":"ok",...}
```
Arc reads an optional `arc.toml` (searched in the current directory, then `/etc/arc/`, then `$HOME/.arc/`) and environment variables prefixed with `ARC_` (a TOML key `section.key` maps to `ARC_SECTION_KEY`). To disable anonymous usage telemetry, set `ARC_TELEMETRY_ENABLED=false`. See [Authentication](/arc/configuration/authentication/) and [Native Installation](/arc/installation/native/) for the full reference.
## Step 1: set your Arc token [#step-1-set-your-arc-token]
Every request below authenticates with a bearer token. Export it once:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
Verify it works:
```bash
curl http://localhost:8000/api/v1/auth/verify \
-H "Authorization: Bearer $ARC_TOKEN"
```
For production, create scoped write tokens rather than reusing the admin token, see [Authentication](/arc/configuration/authentication/).
## Step 2: move ongoing writes [#step-2-move-ongoing-writes]
Unlike QuestDB, TimescaleDB shares no wire protocol with Arc. TimescaleDB apps write over the PostgreSQL wire protocol; Arc ingests over HTTP (MessagePack, Line Protocol, or bulk import). How much work this is depends entirely on how your data reaches TimescaleDB today.
**If you write through Telegraf**, this is a one-block change. Keep the PostgreSQL output while you migrate history, and add an Arc output so new data lands in both (dual-write):
```toml
# Keep writing to TimescaleDB during the migration
[[outputs.postgresql]]
connection = "host=localhost user=postgres dbname=mydb sslmode=disable"
# New data also goes to Arc
[[outputs.arc]]
url = "http://localhost:8000/api/v1/write/msgpack"
api_key = "$ARC_TOKEN"
content_encoding = "gzip"
database = "mydb"
```
Once you have cut over, drop the `[[outputs.postgresql]]` block. See the [Telegraf integration](/arc/integrations/telegraf/) for the full plugin reference.
**If you write with custom SQL `INSERT`/`COPY`**, the write path is application work: re-target it at Arc's HTTP write API (the [MessagePack columnar protocol](/arc/api-reference/overview/) is the fastest, and the [Line Protocol endpoints](/arc/data-import/line-protocol/) accept InfluxDB-style writes). There is no wire-level shortcut here, so plan for a code change.
Dual-writing means TimescaleDB stays authoritative while you move history and build confidence in Arc. Nothing is lost if you pause.
## Step 3: migrate historical data [#step-3-migrate-historical-data]
TimescaleDB is PostgreSQL, so you export with `\copy` (client-side, no server file access needed, works against managed instances). Hypertables export exactly like ordinary tables.
A plain `\copy (SELECT * FROM cpu ...)` writes PostgreSQL's default `timestamptz` format, `2023-11-14 22:13:20+00`. Arc's importer **rejects** the `+00` offset form. Convert the timestamp in the export query to either epoch seconds or RFC 3339. Both are tested below.
**Recommended, export the timestamp as epoch seconds:**
```bash
psql -h localhost -U postgres -d mydb -c \
"\copy (SELECT EXTRACT(EPOCH FROM time)::bigint AS time, host, usage_idle, usage_user FROM cpu ORDER BY time) TO 'cpu.csv' WITH (FORMAT csv, HEADER true)"
```
Import into Arc with `time_format=epoch_s`:
```bash
curl -X POST "http://localhost:8000/api/v1/import/csv?measurement=cpu&time_column=time&time_format=epoch_s" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: mydb" \
-F "file=@cpu.csv"
```
Response:
```json
{
"status": "ok",
"result": {
"database": "mydb",
"measurement": "cpu",
"rows_imported": 10000,
"partitions_created": 3,
"time_range_min": "2023-11-14T22:13:20Z",
"time_range_max": "2023-11-15T00:59:59Z",
"columns": ["time", "host", "usage_idle", "usage_user"],
"duration_ms": 9
}
}
```
**Alternative, export RFC 3339 timestamps** (leave `time_format` off, Arc auto-detects):
```bash
psql -h localhost -U postgres -d mydb -c \
"\copy (SELECT to_char(time AT TIME ZONE 'UTC', 'YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"') AS time, host, usage_idle, usage_user FROM cpu ORDER BY time) TO 'cpu.csv' WITH (FORMAT csv, HEADER true)"
```
```bash
curl -X POST "http://localhost:8000/api/v1/import/csv?measurement=cpu&time_column=time" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "X-Arc-Database: mydb" \
-F "file=@cpu.csv"
```
Each hypertable becomes a measurement in Arc. The CSV header becomes the columns; Arc infers types.
Check what you are moving before you start:
```sql
SELECT hypertable_name, num_chunks,
pg_size_pretty(hypertable_size(format('%I.%I', hypertable_schema, hypertable_name)::regclass)) AS size
FROM timescaledb_information.hypertables;
```
For large hypertables, export in time ranges (`WHERE time >= '2024-01-01' AND time < '2024-02-01'`) to keep files under the 500 MB import limit and to parallelize. Arc auto-detects gzip, so `gzip cpu.csv` and upload `cpu.csv.gz` for faster transfers. Script export-then-import in a `for` loop over your tables.
**Verify the load** against the source. In Arc, set `X-Arc-Database` so you can use bare table names:
```bash
curl -X POST "http://localhost:8000/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-H "X-Arc-Database: mydb" \
-d '{"sql":"SELECT count(*), min(time), max(time) FROM cpu"}'
```
Compare with TimescaleDB: `SELECT count(*), min(time), max(time) FROM cpu;`. Counts and time range should match.
## Step 4: translate your queries [#step-4-translate-your-queries]
The good news first: **`time_bucket()` is identical on both sides**, same name and argument order. A downsampling query that runs on TimescaleDB runs unchanged on Arc.
Set the `X-Arc-Database` header on your query requests and you keep the same bare table names you use in TimescaleDB (`FROM cpu` instead of `FROM mydb.cpu`). This is the recommended pattern, it keeps your SQL identical and takes Arc's leaner query path.
| TimescaleDB | Arc |
| -------------------------------------------------------- | -------------------------------------------------------- |
| `time_bucket(INTERVAL '5 minutes', time)` | `time_bucket(INTERVAL '5 minutes', time)` (same) |
| `last(value, time)` | `arg_max(value, time)` |
| `first(value, time)` | `arg_min(value, time)` |
| `time_bucket_gapfill()` + `locf()` | `generate_series` spine + LOCF window (see below) |
| `percentile_cont(0.95) WITHIN GROUP (ORDER BY x)` | `percentile_cont(0.95) WITHIN GROUP (ORDER BY x)` (same) |
| `create_hypertable(...)`, retention/compression policies | Not needed (schema-on-write; storage-layer concern) |
### Downsampling: Unchanged [#downsampling-unchanged]
```sql
SELECT time_bucket(INTERVAL '1 hour', time) AS bucket,
avg(usage_idle),
count(*)
FROM cpu
GROUP BY 1
ORDER BY 1;
```
Identical SQL, identical results on both engines.
### Latest / earliest per series: `last`/`first` to `arg_max`/`arg_min` [#latest--earliest-per-series-lastfirst-to-arg_maxarg_min]
```sql
-- TimescaleDB
SELECT host, last(usage_idle, time) FROM cpu GROUP BY host;
-- Arc
SELECT host, arg_max(usage_idle, time) FROM cpu GROUP BY host;
```
`arg_max(v, time)` returns the value of `v` at the row with the maximum `time`. `arg_min` does the same for the minimum. Both match TimescaleDB's `last`/`first` exactly.
### Gap-filling: The one real porting cost [#gap-filling-the-one-real-porting-cost]
TimescaleDB's `time_bucket_gapfill()` with `locf()`/`interpolate()` has no drop-in equivalent. Reconstruct it with a generated bucket spine, a `LEFT JOIN`, and a window function for last-observation-carried-forward:
```sql
WITH spine AS (
SELECT unnest(generate_series(
TIMESTAMPTZ '2023-11-14 22:00:00Z',
TIMESTAMPTZ '2023-11-15 00:00:00Z',
INTERVAL '1 hour')) AS bucket
),
agg AS (
SELECT time_bucket(INTERVAL '1 hour', time) AS bucket, avg(usage_idle) AS avg_idle
FROM cpu GROUP BY 1
)
SELECT s.bucket,
a.avg_idle,
last_value(a.avg_idle IGNORE NULLS) OVER (ORDER BY s.bucket) AS locf_idle
FROM spine s
LEFT JOIN agg a ON s.bucket = a.bucket
ORDER BY s.bucket;
```
The `locf_idle` column carries the last non-null value forward across empty buckets. For linear interpolation, use `lead`/`lag` over the same window.
`time_bucket()` returns a timestamp *with* time zone, so the spine must be `TIMESTAMPTZ` for the join to match. A plain `TIMESTAMP` spine joins to nothing and every bucket comes back null.
See the [SQL Querying Guide](/arc/guides/querying/) for the full function reference.
## Differences to know [#differences-to-know]
* **No PostgreSQL wire protocol.** Arc ingests over HTTP. Ongoing writes move to Telegraf-to-Arc or the HTTP write API, not a connection-string swap. See [Step 2](#step-2-move-ongoing-writes).
* **No hypertable DDL.** Arc is schema-on-write, there is no `create_hypertable()`, no chunk sizing, no retention/compression policy SQL. A measurement and its columns appear on first write.
* **Gap-filling is manual.** `time_bucket_gapfill`/`locf`/`interpolate` are reconstructed with `generate_series` and window functions (above).
* **The query API is read-only.** `POST /api/v1/query` rejects write verbs (`INSERT`, `DELETE`, `DROP`, `COPY`, and the file-reading functions). Ingest through the write and import endpoints.
* **Portable storage.** Arc's Parquet files are yours, queryable in place by any Parquet-compatible tool, or moved to S3/MinIO/Azure without an export step.
## Next steps [#next-steps]
* [SQL Querying Guide](/arc/guides/querying/) - full SQL function reference and query patterns
* [CSV Import](/arc/data-import/csv/) - all import parameters and options
* [Telegraf integration](/arc/integrations/telegraf/) - the native `outputs.arc` output plugin
* [Authentication](/arc/configuration/authentication/) - creating scoped tokens for production
* [Grafana data source](/arc/integrations/grafana/) - dashboards on your migrated data
# Backup & Restore (/arc/operations/backup-restore)
Arc includes a full backup and restore system via REST API. Backups capture parquet data files, SQLite metadata (auth, audit, MQTT config), and the `arc.toml` configuration file -- with async operations, real-time progress tracking, and selective restore.
Backup & Restore is available starting Arc v26.03.1 (March 2026).
All backup and restore endpoints require admin authentication.
## Configuration [#configuration]
```toml
[backup]
enabled = true # default: true
local_path = "./data/backups" # default: ./data/backups
```
## API endpoints [#api-endpoints]
| Method | Endpoint | Description |
| -------- | ------------------------ | ----------------------------- |
| `POST` | `/api/v1/backup` | Trigger a full backup (async) |
| `GET` | `/api/v1/backup` | List all available backups |
| `GET` | `/api/v1/backup/status` | Progress of active operation |
| `GET` | `/api/v1/backup/:id` | Get backup manifest |
| `DELETE` | `/api/v1/backup/:id` | Delete a backup |
| `POST` | `/api/v1/backup/restore` | Restore from a backup (async) |
## Creating a backup [#creating-a-backup]
```bash
curl -X POST "http://localhost:8000/api/v1/backup" \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response (202 Accepted):**
```json
{
"message": "Backup started",
"status": "running"
}
```
The backup runs asynchronously in the background. Poll the status endpoint to monitor progress.
### Polling progress [#polling-progress]
```bash
curl "http://localhost:8000/api/v1/backup/status" \
-H "Authorization: Bearer $ARC_TOKEN"
```
```json
{
"operation": "backup",
"backup_id": "backup-20260211-143022-a1b2c3d4",
"status": "running",
"total_files": 1200,
"processed_files": 450,
"total_bytes": 5368709120,
"processed_bytes": 2147483648
}
```
## Listing backups [#listing-backups]
```bash
curl "http://localhost:8000/api/v1/backup" \
-H "Authorization: Bearer $ARC_TOKEN"
```
## Viewing a backup manifest [#viewing-a-backup-manifest]
```bash
curl "http://localhost:8000/api/v1/backup/backup-20260211-143022-a1b2c3d4" \
-H "Authorization: Bearer $ARC_TOKEN"
```
### Backup structure [#backup-structure]
```text
{backup_id}/
manifest.json # metadata: databases, measurements, file counts, sizes
data/ # parquet files preserving partition layout
metadata/arc.db # SQLite database snapshot
config/arc.toml # configuration file
```
## Restoring from a backup [#restoring-from-a-backup]
Restore overwrites existing data. Existing SQLite and config files are preserved with a `.before-restore` suffix before overwriting.
```bash
curl -X POST "http://localhost:8000/api/v1/backup/restore" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"backup_id": "backup-20260211-143022-a1b2c3d4",
"restore_data": true,
"restore_metadata": true,
"restore_config": false,
"confirm": true
}'
```
### Restore options [#restore-options]
| Field | Type | Default | Description |
| ------------------ | ------ | ------------ | ------------------------------------------- |
| `backup_id` | string | *(required)* | ID of the backup to restore |
| `restore_data` | bool | `true` | Restore parquet data files |
| `restore_metadata` | bool | `true` | Restore SQLite database (auth, audit, MQTT) |
| `restore_config` | bool | `false` | Restore `arc.toml` configuration |
| `confirm` | bool | *(required)* | Must be `true` to proceed |
### Selective restore examples [#selective-restore-examples]
```bash
# Restore only data (keep current auth tokens and config)
curl -X POST "http://localhost:8000/api/v1/backup/restore" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"backup_id": "backup-20260211-143022-a1b2c3d4",
"restore_data": true,
"restore_metadata": false,
"restore_config": false,
"confirm": true
}'
# Restore everything including config
curl -X POST "http://localhost:8000/api/v1/backup/restore" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"backup_id": "backup-20260211-143022-a1b2c3d4",
"restore_data": true,
"restore_metadata": true,
"restore_config": true,
"confirm": true
}'
```
## Deleting a backup [#deleting-a-backup]
```bash
curl -X DELETE "http://localhost:8000/api/v1/backup/backup-20260211-143022-a1b2c3d4" \
-H "Authorization: Bearer $ARC_TOKEN"
```
Deletion is refused with `409 Conflict` while a backup or restore is running -- deleting the backup a restore is reading would tear files out from under it. Retry once the operation finishes.
## Key behaviors [#key-behaviors]
* **Async operations** -- backup and restore run in background goroutines with a 2-hour timeout. Clients poll `/status` for progress.
* **Serialized operations** -- only one backup, restore, or delete can run at a time. Attempting a concurrent operation returns `409 Conflict`.
* **Pre-restore safety** -- existing SQLite and config files are copied with `.before-restore` suffix before overwriting.
* **Destructive restore protection** -- restore requires explicit `confirm: true` in the request body.
* **What gets backed up** -- parquet data files, SQLite database (with WAL checkpoint for consistency), and `arc.toml` config.
* **All storage backends** -- works with local filesystem, S3, and Azure Blob Storage.
## Error responses [#error-responses]
| Status | Description |
| ------ | ------------------------------------------------------------------------------ |
| `401` | Authentication required |
| `403` | Admin role required |
| `404` | Backup not found |
| `409` | Another operation is already running (returned by backup, restore, and delete) |
| `500` | Backup or restore execution error |
# Operations (/arc/operations)
Day-two concerns for an Arc deployment.
# Profiling with pprof (/arc/operations/profiling)
Arc exposes Go's built-in `net/http/pprof` profiler — heap, goroutine, CPU, allocations, blocking, mutex, and execution-trace endpoints — for diagnosing memory pressure, hot CPU paths, goroutine leaks, and deadlocks in production. The endpoints are **opt-in** and bound to `localhost` by default; exposing them anywhere else requires a deliberate two-step configuration.
The opt-in pprof listener ships in Arc v26.06.1 ([PR #443](https://github.com/Basekick-Labs/arc/pull/443), [GHSA-j93g-rp6m-j32m](https://github.com/Basekick-Labs/arc/security/advisories/GHSA-j93g-rp6m-j32m)). Prior versions registered pprof on the public API port without authentication — upgrade and adopt the env-var gate below.
A reachable `/debug/pprof/*` endpoint leaks process internals: in-flight SQL strings and msgpack records (via heap dumps), goroutine stacks, environment variables on some Go versions, and lets any caller pin a CPU core for arbitrary seconds via `/debug/pprof/profile?seconds=N`. Treat the pprof listener like a root shell — bind to loopback, restrict by firewall, and turn it off when you're done debugging.
## Why pprof is off by default [#why-pprof-is-off-by-default]
Pre-v26.06.1, `/debug/pprof/*` was mounted on Arc's public Fiber app — no token, no allowlist. An unauthenticated network caller could fetch heap dumps containing recent query text and ingested records. The hardening PR removed pprof from the public app entirely and moved it to a separate listener that only starts when the `ARC_DEBUG_PPROF` env var is set.
The new design has three properties:
1. **Off by default** — `ARC_DEBUG_PPROF` unset means no socket is opened, no goroutine is spawned, the endpoints don't exist on Arc's process.
2. **Loopback-bound by default** — even with `ARC_DEBUG_PPROF=1`, the listener binds to `127.0.0.1:6060` unless you explicitly override.
3. **Two-step opt-in for non-loopback** — binding to any non-loopback address (`0.0.0.0:6060`, a public IP, etc.) requires both `ARC_DEBUG_PPROF_ADDR` AND `ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1`, so a typo in the bind address can't accidentally expose the endpoint cross-host.
## Configuration [#configuration]
All configuration is via environment variables — pprof is a debugging surface, not a runtime feature, so there's no `[debug]` block in `arc.toml`.
| Variable | Default | Description |
| ------------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ARC_DEBUG_PPROF` | unset (off) | Set to `1`, `true`, `yes`, or `on` to enable the pprof listener. Any other value (including unset) leaves it off. |
| `ARC_DEBUG_PPROF_ADDR` | `127.0.0.1:6060` | Bind address for the pprof listener. Accepts any form `net.Listen("tcp", …)` accepts — `127.0.0.1:6060`, `localhost:6060`, `[::1]:6060`, `0.0.0.0:6060`, etc. |
| `ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK` | unset (off) | Required when `ARC_DEBUG_PPROF_ADDR` is non-loopback. Set to `1`/`true`/`yes`/`on`. Without it, Arc logs an error and refuses to start the pprof listener. |
## Enabling pprof on a single node [#enabling-pprof-on-a-single-node]
The common case — investigate a single production node from the same host via SSH and a local port-forward:
```bash
# On the node you want to profile:
ARC_DEBUG_PPROF=1 ./arc
```
Arc emits a startup warning:
```bash
WARN ARC_DEBUG_PPROF is set — pprof endpoints are exposed on this address.
Restrict access via firewall or unset ARC_DEBUG_PPROF in production.
addr=127.0.0.1:6060
```
From your laptop, SSH-tunnel the port:
```bash
ssh -L 6060:127.0.0.1:6060 user@node
```
Then point `go tool pprof` at `localhost:6060` on your laptop. See [Profiling Workflows](#profiling-workflows) below.
### With docker-compose [#with-docker-compose]
```yaml
services:
arc-writer:
image: basekick/arc:latest
environment:
ARC_DEBUG_PPROF: "1"
# No host port mapping for 6060 — the listener stays inside the container.
# Use `docker exec` or a sidecar to reach it.
```
To reach the in-container listener:
```bash
docker exec -it arc-writer wget -qO heap.pprof http://127.0.0.1:6060/debug/pprof/heap
docker cp arc-writer:/heap.pprof ./
go tool pprof -http=:8080 heap.pprof
```
### With Kubernetes [#with-kubernetes]
```yaml
env:
- name: ARC_DEBUG_PPROF
value: "1"
```
Then port-forward:
```bash
kubectl port-forward arc-writer-0 6060:6060
```
`kubectl port-forward` only listens on the local machine, so the pprof endpoint stays loopback-bound on the Arc pod AND on your laptop simultaneously. No cluster-network exposure.
## Exposing pprof cross-host (discouraged) [#exposing-pprof-cross-host-discouraged]
There are cases where loopback isn't enough — for example, a remote profiler that can't open an SSH tunnel, or a multi-tenant box where the operator workstation isn't on the Arc host. Arc supports this with a deliberate two-step opt-in:
```bash
ARC_DEBUG_PPROF=1 \
ARC_DEBUG_PPROF_ADDR=0.0.0.0:6060 \
ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1 \
./arc
```
Without `ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1`, Arc logs an **error** and refuses to start the pprof listener — the rest of Arc continues to run normally, but pprof stays off:
```bash
ERROR ARC_DEBUG_PPROF=1 with a non-loopback ARC_DEBUG_PPROF_ADDR requires
ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1; refusing to start pprof listener
addr=0.0.0.0:6060
```
When the second opt-in IS set and Arc binds to a non-loopback address, the startup log line is escalated to **error** level (instead of warn) so default alerting policies notice the cross-host exposure on this node:
```bash
ERROR ARC_DEBUG_PPROF is set — pprof endpoints are exposed on this address.
Restrict access via firewall or unset ARC_DEBUG_PPROF in production.
addr=0.0.0.0:6060
```
The pprof listener has no authentication. Anyone who can reach `0.0.0.0:6060` (or whatever address you bound) can fetch heap dumps containing recent query text and ingested records, dump goroutine stacks, and pin CPU cores. Restrict by network ACL, security group, or iptables before turning this on. Unset all three env vars the moment you're done.
## Profiling workflows [#profiling-workflows]
Once the listener is reachable at `http://localhost:6060` (whether direct or via SSH/kubectl port-forward), `go tool pprof` does the rest. The recipes below assume Go 1.20+.
### Heap (memory) [#heap-memory]
The most common case — Arc's RSS is high and you want to know what's holding it.
```bash
# Live snapshot:
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/heap
# Save for later analysis:
curl -o heap.pprof http://localhost:6060/debug/pprof/heap
go tool pprof -http=:8080 heap.pprof
```
The `-http=:8080` flag launches the interactive web UI at `http://localhost:8080` — flame graph, top callers, source view. Without it you get the CLI prompt.
Common starting commands at the pprof CLI prompt:
```text
(pprof) top20 # 20 largest in-use allocations by bytes
(pprof) top20 -cum # 20 largest by cumulative (function + callees)
(pprof) list # source-level breakdown of one function
```
### CPU profile [#cpu-profile]
Capture 30 seconds of CPU activity:
```bash
go tool pprof -http=:8080 'http://localhost:6060/debug/pprof/profile?seconds=30'
```
The `seconds` parameter is configurable — 30s is a reasonable default. **Don't go above \~300s** unless you know what you're doing: each in-flight capture holds a connection open and consumes scheduler overhead. Arc's pprof listener has a 10-minute write timeout as the hard ceiling.
### Goroutines [#goroutines]
Diagnose a goroutine leak or deadlock:
```bash
# Summary (top goroutine call sites + counts):
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=1' | head -50
# Full stacks for every goroutine (text):
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=2' > goroutines.txt
# Or via pprof for the UI:
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/goroutine
```
A healthy idle Arc writer typically has \~50–200 goroutines (Fiber workers, WAL writer, ingest shards, compaction scheduler, Raft loops). Thousands of goroutines stuck on the same `chan receive` or `sync.Mutex.Lock` is the diagnostic signature of a stall.
### Execution trace [#execution-trace]
Captures every scheduler event for `N` seconds — useful for diagnosing latency spikes:
```bash
curl -o trace.out 'http://localhost:6060/debug/pprof/trace?seconds=5'
go tool trace -http=:8080 trace.out
```
The trace UI shows per-goroutine timelines, GC pauses, and network/syscall waits. Use sparingly — even 5 seconds of trace produces \~10–50 MB of data on a busy writer.
### Block & mutex profiles [#block--mutex-profiles]
By default these profiles are zero-rate (Go runtime samples nothing). To enable, you'd need to call `runtime.SetBlockProfileRate` / `runtime.SetMutexProfileFraction` from inside Arc — currently not exposed via env var. If you need block/mutex profiles, open an issue describing the problem you're chasing and we'll add the knobs.
## Operational notes [#operational-notes]
### Startup logging [#startup-logging]
When `ARC_DEBUG_PPROF` is unset, Arc emits nothing at startup about pprof. The listener is genuinely absent — no port, no handlers, no log noise.
When set, a single warn-level (loopback) or error-level (non-loopback) line names the bind address and reminds you to restrict access. Grep for `ARC_DEBUG_PPROF is set` in your logs to find nodes that left it on accidentally.
### Shutdown behavior [#shutdown-behavior]
Arc registers pprof with the same shutdown priority as the main HTTP server. On `SIGTERM` / `SIGINT`, the pprof listener closes **immediately** — in-flight captures (especially long `/debug/pprof/profile?seconds=N` requests) are aborted. This is deliberate: a long pprof capture would otherwise hold the cluster's shared shutdown budget and risk skipping downstream hooks (WAL flush, storage close, auth close), which is a data-loss path on what the operator expected to be a graceful exit.
If your capture was killed by shutdown, just re-run it after Arc restarts.
### Port conflicts [#port-conflicts]
If the configured bind address is already in use, Arc logs an **error** and continues without the pprof listener — Arc itself doesn't fail to start. Look for:
```bash
ERROR ARC_DEBUG_PPROF=1 but failed to bind pprof listener; continuing without pprof
addr=127.0.0.1:6060 error="listen tcp 127.0.0.1:6060: bind: address already in use"
```
Common causes:
* A previous Arc process didn't release the port (`lsof -nP -iTCP:6060`).
* Another Go service on the host already runs pprof on `:6060` (the Go-runtime convention).
* A non-Arc service grabbed the port.
Resolve the conflict and restart Arc, or set `ARC_DEBUG_PPROF_ADDR` to a different port.
## Security checklist [#security-checklist]
Before enabling pprof on a production node:
* [ ] `ARC_DEBUG_PPROF_ADDR` is loopback (default) **or** the host is firewalled to allow only your jumphost / operator workstation.
* [ ] If non-loopback, `ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1` is set deliberately (not by env-var inheritance from a parent process).
* [ ] You have a plan to unset `ARC_DEBUG_PPROF` when the investigation is done — pprof should not be left on indefinitely.
* [ ] On Kubernetes / docker-compose, the pprof port is **not** in the service's port list or compose `ports:` block — only reachable via `kubectl port-forward` or `docker exec`.
* [ ] Heap dumps you save (`heap.pprof`, `goroutines.txt`, `trace.out`) are treated as sensitive: they contain in-flight query text and ingested records. Don't paste them into public issues; share via your team's secure channel.
## Reference [#reference]
* Source: [`cmd/arc/debug_pprof.go`](https://github.com/Basekick-Labs/arc/blob/main/cmd/arc/debug_pprof.go) — the listener and the two-step gate.
* PR that introduced the gate: [#443](https://github.com/Basekick-Labs/arc/pull/443).
* Advisory: [GHSA-j93g-rp6m-j32m](https://github.com/Basekick-Labs/arc/security/advisories/GHSA-j93g-rp6m-j32m).
* Upstream Go docs: [`net/http/pprof`](https://pkg.go.dev/net/http/pprof) and [`runtime/pprof`](https://pkg.go.dev/runtime/pprof).
# Telemetry (/arc/operations/telemetry)
Arc sends anonymous usage telemetry to help improve the project. This page explains what data is collected, how it's used, and how to opt out.
## Overview [#overview]
Arc collects minimal, anonymous usage statistics to help the development team understand:
* How Arc is being deployed (operating systems, hardware configurations)
* Which Arc versions are in active use
* Basic system characteristics for optimization and testing
Arc does not collect any personally identifiable information, user data, database contents, queries, or performance metrics.
## What is collected [#what-is-collected]
Arc sends the following anonymous data every 24 hours:
### Instance information [#instance-information]
* **instance\_id**: A random UUID generated on first run
* Stored in `./data/.instance_id`
* Unique per Arc installation
* Not linked to any personal information
* **timestamp**: When the telemetry report was generated (UTC)
* **arc\_version**: The running version number (e.g., `0.1.0`)
### System information [#system-information]
* **os**: Operating system details
* Name (e.g., "Linux", "macOS", "Windows")
* Version (e.g., "Ubuntu 22.04", "macOS 14.0")
* Architecture (e.g., "x86\_64", "arm64")
* Platform (e.g., "linux", "darwin")
* **cpu**: CPU characteristics
* Physical cores
* Logical cores (threads)
* Frequency in MHz
* **memory**: System memory
* Total RAM in gigabytes
### Example payload [#example-payload]
```json
{
"instance_id": "550e8400-e29b-41d4-a716-446655440000",
"timestamp": "2024-01-20T10:30:00Z",
"arc_version": "0.1.0",
"os": {
"name": "Linux",
"version": "Ubuntu 22.04",
"architecture": "x86_64",
"platform": "linux"
},
"cpu": {
"physical_cores": 8,
"logical_cores": 16,
"frequency_mhz": 3400
},
"memory": {
"total_gb": 32
}
}
```
## What is NOT collected [#what-is-not-collected]
Arc explicitly avoids collecting:
* **User Data**: No usernames, emails, or personal information
* **Database Contents**: No table names, schemas, or data
* **Query Information**: No SQL queries or query patterns
* **Network Information**: No IP addresses or hostnames
* **Credentials**: No API keys, passwords, or tokens
* **File Paths**: No directory structures or file names
* **Performance Metrics**: No query times, throughput, or resource usage
* **Custom Configuration**: No application-specific settings
## How it works [#how-it-works]
### Telemetry schedule [#telemetry-schedule]
1. **First Transmission**: 1 minute after Arc starts
2. **Subsequent Transmissions**: Every 24 hours
3. **Primary Worker Only**: Only the primary worker process sends telemetry (multi-worker deployments send one report)
### Endpoint [#endpoint]
Telemetry is sent to: `telemetry.basekick.net`
### Network behavior [#network-behavior]
* If the telemetry endpoint is unreachable, Arc logs a warning but continues operating normally
* Failed transmissions are retried during the next scheduled transmission
* No telemetry data is queued or persisted locally
### Startup logging [#startup-logging]
Arc logs telemetry status on startup:
**When Enabled**:
```text
INFO: Telemetry enabled. Sending anonymous usage data to telemetry.basekick.net every 24 hours.
```
**When Disabled**:
```text
INFO: Telemetry disabled via configuration.
```
## Disabling telemetry [#disabling-telemetry]
You can opt out of telemetry in two ways:
### Option 1: configuration file [#option-1-configuration-file]
Edit your `arc.toml` file and add:
```toml
[telemetry]
enabled = false
```
**Full Example**:
```toml
[server]
host = "0.0.0.0"
port = 8000
[telemetry]
enabled = false
```
### Option 2: environment variable [#option-2-environment-variable]
Set the environment variable before starting Arc:
```bash
export ARC_TELEMETRY_ENABLED=false
```
**With Docker**:
```bash
docker run -e ARC_TELEMETRY_ENABLED=false arc:latest
```
**With Docker Compose**:
```yaml
services:
arc:
image: arc:latest
environment:
- ARC_TELEMETRY_ENABLED=false
```
### Verification [#verification]
After configuring, start Arc and check the logs:
```text
INFO: Telemetry disabled via configuration.
```
If you see this message, telemetry is successfully disabled.
## Why telemetry? [#why-telemetry]
### Benefits to the project [#benefits-to-the-project]
Anonymous telemetry helps the Arc team:
1. **Prioritize Platform Support**: Understand which operating systems and architectures to focus on
2. **Test on Real Hardware**: Know what CPU and memory configurations are common
3. **Track Version Adoption**: See how quickly users upgrade to new releases
4. **Plan Deprecations**: Identify when old versions are no longer in use
### Privacy considerations [#privacy-considerations]
Arc's telemetry is designed with privacy as a priority:
* **Anonymous**: No linkage to individuals or organizations
* **Minimal**: Only essential system characteristics
* **Transparent**: Full disclosure of what is collected
* **Optional**: Easy opt-out with no functionality loss
* **No Tracking**: No cookies, fingerprinting, or cross-site tracking
## Frequently asked questions [#frequently-asked-questions]
### Does telemetry affect performance? [#does-telemetry-affect-performance]
No. Telemetry runs asynchronously and has negligible performance impact:
* Transmission occurs once per 24 hours
* Payload is \~500 bytes
* Network timeout is short (5 seconds)
* Failed transmissions don't block Arc operations
### Can I verify what's being sent? [#can-i-verify-whats-being-sent]
Yes. You can inspect the telemetry payload by:
1. **Network Inspection**: Use tools like Wireshark or tcpdump to capture the request
2. **Source Code**: Review the telemetry implementation in the Arc repository
3. **Logging**: Enable debug logging to see telemetry payloads (future feature)
### What happens to the data? [#what-happens-to-the-data]
Telemetry data is:
* Stored securely on Basekick infrastructure
* Aggregated for statistical analysis
* Not shared with third parties
* Not used for commercial purposes
* Retained for a limited time (90 days)
### Will Arc work if telemetry is blocked? [#will-arc-work-if-telemetry-is-blocked]
Yes. Arc functions identically whether telemetry is enabled or disabled. If the telemetry endpoint is unreachable (firewall, network issues), Arc logs a warning and continues normally.
### Why not make it opt-in? [#why-not-make-it-opt-in]
We believe in transparency and easy opt-out rather than opt-in because:
* Telemetry helps improve the product for everyone
* Data collected is truly anonymous and minimal
* Opt-out is simple and clearly documented
* Many users don't discover opt-in options
However, we respect your choice and make opting out straightforward.
### Does Arc Enterprise have different telemetry? [#does-arc-enterprise-have-different-telemetry]
No. Both Arc OSS and Arc Enterprise use identical telemetry collection. Arc Enterprise customers can request custom telemetry configurations for their deployments.
## Privacy policy [#privacy-policy]
For detailed information about how Basekick handles data, see our [Privacy Policy](https://basekick.net/privacy?utm_source=docs\&utm_medium=referral\&utm_campaign=arc) (Coming Soon).
## Support [#support]
If you have questions or concerns about telemetry:
* [Discord Community](https://discord.gg/nxnWfUxsdm)
* [GitHub Issues](https://github.com/basekick-labs/arc/issues)
* Email: [privacy@basekick.net](mailto:privacy@basekick.net)
# Performance Benchmarks (/arc/performance/benchmarks)
Benchmark results are published on the Basekick blog rather than duplicated here. A number in the docs goes stale the moment the hardware, the dataset, or the release changes; the blog posts keep each result attached to the machine, the dataset, and the Arc version that produced it.
## ClickBench [#clickbench]
Arc's ClickBench results, including the test hardware, the dataset, and the full per-query timings:
* **[Arc on ClickBench](https://basekick.net/blog/arc-fastest-timeseries-database-clickbench?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)** — the headline results and how the runs were performed.
* **[Cold-run results](https://basekick.net/blog/arc-clickbench-cold-runs?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)** — query performance with caches flushed.
Every ClickBench submission is independently verifiable at [benchmark.clickhouse.com](https://benchmark.clickhouse.com).
## Comparisons with other databases [#comparisons-with-other-databases]
Each post runs the same benchmark on the same instance type against one other system:
* [Arc vs InfluxDB](https://basekick.net/blog/arc-clickbench-vs-influxdb?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)
* [Arc vs TimescaleDB](https://basekick.net/blog/arc-clickbench-vs-timescaledb?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)
* [Arc vs DuckDB](https://basekick.net/blog/arc-clickbench-vs-duckdb?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)
* [Arc vs CrateDB](https://basekick.net/blog/arc-clickbench-vs-cratedb?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)
* [Arc vs StarRocks](https://basekick.net/blog/arc-clickbench-vs-starrocks?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)
* [Arc vs Elasticsearch](https://basekick.net/blog/arc-clickbench-vs-elasticsearch?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)
## Log workloads [#log-workloads]
* **[Arc log benchmark](https://basekick.net/blog/arc-log-benchmark-2026?utm_source=docs\&utm_medium=referral\&utm_campaign=arc)** — ingestion and query performance on log-shaped data.
## Reproducing a run [#reproducing-a-run]
The benchmark harnesses are public, so you can re-run any of these against your own hardware:
* [ClickBench](https://github.com/ClickHouse/ClickBench) — the upstream harness and dataset.
* [Arc repository](https://github.com/basekick-labs/arc) — Arc itself, plus the configuration used in the published runs.
Measuring on hardware that resembles your production deployment is worth more than any published figure. Throughput depends heavily on batch size, ingestion protocol, storage backend, and whether the [WAL](/arc/advanced/wal/) is enabled.
## Related [#related]
* [Storage file format](/arc/configuration/storage-file-format/) — Arc's on-disk Parquet format and its compression.
* [File compaction](/arc/advanced/compaction/) — why small files slow queries down.
* [Query caching](/arc/advanced/caching/) — what Arc memoizes on the query path.
# Performance (/arc/performance)
Benchmark results are published on the Basekick blog rather than duplicated here, so the numbers stay attached to the hardware and methodology that produced them.
# SDKs (/arc/sdks)
Client libraries wrap Arc's REST API with typed helpers for writing, querying, and administration.
# Query Caching (/arc-enterprise/advanced/caching)
Arc implements multiple caching layers to optimize query performance, particularly for dashboard and monitoring use cases where the same queries are executed repeatedly.
## Cache layers [#cache-layers]
Arc uses three complementary caches that work together:
| Cache | TTL | Purpose | Savings |
| -------------------- | --- | -------------------------------------- | --------------------------------- |
| SQL Transform Cache | 60s | Caches SQL-to-storage-path conversions | Avoids re-parsing per query |
| Partition Path Cache | 60s | Caches `OptimizeTablePath()` results | Avoids repeated path resolution |
| Glob Cache | 30s | Caches filesystem glob results | Avoids repeated directory listing |
## SQL transform cache [#sql-transform-cache]
When you execute a query like:
```sql
SELECT * FROM mydb.cpu WHERE time > now() - INTERVAL '1 hour'
```
Arc converts the table reference `mydb.cpu` to a `read_parquet()` call:
```sql
SELECT * FROM read_parquet('./data/mydb/cpu/**/*.parquet') WHERE time > now() - INTERVAL '1 hour'
```
This string transformation uses regex matching and happens on every query. The SQL Transform Cache stores the result so repeated queries skip this processing.
### Performance impact [#performance-impact]
| Scenario | Time | Speedup |
| --------------------------- | ------- | ------- |
| Without cache (first query) | 13-37μs | - |
| With cache (repeated query) | \~300ns | 49-104x |
### When it helps [#when-it-helps]
The SQL Transform Cache is most beneficial for:
* **Dashboard refresh**: Same queries every 30s-5min
* **Monitoring alerts**: Repeated threshold checks
* **API integrations**: Clients polling the same metrics
* **Multi-user dashboards**: Shared queries across users
### Cache behavior [#cache-behavior]
* **Key**: SHA256 hash of the raw SQL string
* **TTL**: 60 seconds (matches partition cache)
* **Max entries**: 10,000 queries
* **Eviction**: Expired entries removed first, then oldest
## Partition path cache [#partition-path-cache]
After SQL transformation, Arc optimizes the storage path by applying time-based partition pruning. This cache stores the optimized paths.
### Example [#example]
Query with time filter:
```sql
SELECT * FROM mydb.cpu WHERE time > 1704067200000000
```
Without cache: Scans partition metadata to find relevant directories.
With cache: Returns pre-computed path like `./data/mydb/cpu/2024/01/**/*.parquet`.
### Performance impact [#performance-impact-1]
Saves 50-100ms per query on large datasets with many partitions.
## Glob cache [#glob-cache]
After determining the partition path, Arc uses filesystem globs to find matching Parquet files. The Glob Cache stores these file listings.
### Performance impact [#performance-impact-2]
Saves 5-10ms per query by avoiding repeated filesystem operations.
## Cache statistics [#cache-statistics]
Monitor cache performance via the pruner stats:
```go
stats := pruner.GetAllCacheStats()
```
Returns:
```json
{
"partition_cache": {
"size": 150,
"hits": 12847,
"misses": 423,
"hit_rate_percent": 96.8
},
"glob_cache": {
"size": 89,
"hits": 8234,
"misses": 312,
"hit_rate_percent": 96.3
}
}
```
## Best practices [#best-practices]
### 1. Use consistent query strings [#1-use-consistent-query-strings]
Cache keys are based on exact SQL text. These are different cache entries:
```sql
SELECT * FROM mydb.cpu WHERE time > 1704067200000000
SELECT * FROM mydb.cpu WHERE time > 1704067200000000 -- extra space
select * from mydb.cpu where time > 1704067200000000 -- lowercase
```
Normalize your queries for better cache hit rates.
### 2. Use parameterized time ranges [#2-use-parameterized-time-ranges]
For dashboard queries, use relative time:
```sql
-- Good: Same query text on every refresh
SELECT * FROM mydb.cpu WHERE time > now() - INTERVAL '1 hour'
-- Less efficient: Different timestamp each time
SELECT * FROM mydb.cpu WHERE time > 1704067200000000
```
### 3. Monitor hit rates [#3-monitor-hit-rates]
Healthy dashboards should see 60-80%+ cache hit rates. Low hit rates may indicate:
* Too many unique queries
* Query text variations
* TTL too short for your refresh interval
## Configuration [#configuration]
Cache parameters are currently fixed but tuned for typical workloads:
| Parameter | Value | Rationale |
| ---------------------- | ------ | -------------------------------------- |
| SQL Transform TTL | 60s | Covers 1-2 dashboard refresh cycles |
| SQL Transform Max Size | 10,000 | Handles large multi-tenant deployments |
| Partition Cache TTL | 60s | Balance freshness vs. performance |
| Glob Cache TTL | 30s | Files change less frequently |
## Cache invalidation [#cache-invalidation]
Caches automatically expire based on TTL. Manual invalidation happens when:
* New data is ingested (invalidates partition/glob caches for affected measurements)
* Compaction runs (file paths change)
The SQL Transform Cache is not invalidated by data changes since the transformation logic doesn't depend on data content.
## Technical details [#technical-details]
### Thread safety [#thread-safety]
All caches use `sync.RWMutex` for concurrent access:
* Multiple readers allowed
* Exclusive write lock for updates
* Lock-free atomic counters for hit/miss tracking
### Memory usage [#memory-usage]
Approximate memory per cache:
| Cache | Entry Size | Max Entries | Max Memory |
| -------------- | ----------- | ----------- | ---------- |
| SQL Transform | \~500 bytes | 10,000 | \~5MB |
| Partition Path | \~200 bytes | 1,000 | \~200KB |
| Glob | \~1KB | 1,000 | \~1MB |
Total cache overhead: \~6MB typical, \~10MB maximum.
## Next steps [#next-steps]
* **[Compaction](/arc-enterprise/advanced/compaction/)** - Optimize query performance through file merging
* **[WAL](/arc-enterprise/advanced/wal/)** - Write-ahead log for durability
* **[Performance Benchmarks](/arc-enterprise/performance/benchmarks/)** - Benchmark methodology and results
# File Compaction (/arc-enterprise/advanced/compaction)
Arc's automatic compaction system merges small Parquet files into larger, optimized files for dramatically faster queries.
## Overview [#overview]
Compaction is Arc's file optimization system that **merges small files into larger ones**, improving query performance by 10-50x.
**Key Features:**
* **Automatic** - Runs on schedule (default: hourly at :05)
* **Safe** - Locked partitions prevent concurrent compaction
* **Efficient** - Parallel, sorted merging by the query engine
* **Non-blocking** - Queries work during compaction
* **Enabled by default** - Essential for production
Compaction is **enabled by default** and runs automatically every hour.
## Why compaction matters [#why-compaction-matters]
### The small file problem [#the-small-file-problem]
Arc's high-performance ingestion creates many small files:
```text
At a sustained high ingest rate with a 5-second flush:
→ one file per flush interval per measurement
→ 12 files per minute per measurement
→ 720 files per hour per measurement
→ 17,280 files per day per measurement
```
**Impact on Queries:**
* **Slow queries** - The query engine must open/scan hundreds of files
* **High costs** - More S3/MinIO API calls
* **Poor compression** - Small files compress less efficiently
* **Reduced pruning** - Less effective partition elimination
### After compaction [#after-compaction]
**Real Production Test Results:**
```yaml
Before: 2,704 small files (Snappy) = 3.7 GB
After: 3 compacted files (ZSTD) = 724 MB
Compression: 80.4% space savings
File reduction: 901x fewer files (2,704 → 3)
Compaction time: 5 seconds
```
**Per-Measurement Breakdown:**
* **mem**: 888 files → 1 file, 1,213 MB → 239 MB (80.3% compression)
* **disk**: 906 files → 1 file, 1,237 MB → 242 MB (80.4% compression)
* **cpu**: 910 files → 1 file, 1,246 MB → 243 MB (80.5% compression)
**Query Performance:**
* **10-50x faster** - Single file scan vs hundreds
* **99% fewer API calls** - Massive cost reduction (2,704 → 3 LIST operations)
* **80.4% compression** - ZSTD compaction vs Snappy writes
* **Effective pruning** - The query engine can skip entire files
## How it works [#how-it-works]
### Compaction flow [#compaction-flow]
```text
1. Scheduler wakes up (cron: "5 * * * *")
↓
2. Scan storage for eligible partitions
↓
3. For each partition:
- Check age (>1 hour old?)
- Check file count (≥10 files?)
- Check if already compacted?
↓
4. Acquire partition lock (SQLite)
↓
5. Download small files to temp directory
↓
6. Compact via the query engine (parallel, sorted)
↓
7. Upload compacted file to storage
↓
8. Delete old small files
↓
9. Release lock & cleanup temp files
↓
10. Repeat for next partition
```
### Partition structure [#partition-structure]
Data is organized by hour:
```text
arc/ # Bucket
├── default/ # Database
│ └── cpu/ # Measurement
│ └── 2025/10/08/ # Date
│ ├── 14/ # Hour (2 PM) - Eligible for compaction
│ │ ├── file1.parquet (50 MB)
│ │ ├── file2.parquet (48 MB)
│ │ └── ...
│ ├── 15/ # Hour (3 PM) - Eligible for compaction
│ └── 16/ # Hour (4 PM) - CURRENT, skip!
```
Compaction merges all files in a partition (e.g., `2025/10/08/14/`) into one optimized file.
## Configuration [#configuration]
### Default configuration [#default-configuration]
Compaction is **enabled by default** in `arc.toml`:
```toml
[compaction]
enabled = true
# Hourly tier
hourly_enabled = true
hourly_schedule = "5 * * * *" # Cron schedule: every hour at :05
hourly_min_age_hours = 1 # Wait 1 hour before compacting (let the hour complete)
hourly_min_files = 10 # Only compact if >=10 files exist
# Daily tier
daily_enabled = true
daily_schedule = "0 3 * * *" # Cron schedule: 3 AM daily
daily_min_age_hours = 24 # Wait 24 hours
daily_min_files = 12 # Only compact if >=12 files exist
max_concurrent = 2 # Run 2 compactions in parallel
```
### Configuration options [#configuration-options]
#### Schedule [#schedule]
```toml
[compaction]
hourly_schedule = "5 * * * *" # Every hour at :05 (default)
daily_schedule = "0 3 * * *" # 3 AM daily (default)
# hourly_schedule = "0 */2 * * *" # Every 2 hours at :00
```
**Cron format:** `minute hour day month weekday`
#### Minimum age [#minimum-age]
```toml
[compaction]
hourly_min_age_hours = 1 # Don't compact the current hour (default)
daily_min_age_hours = 24 # Daily tier waits a full day (default)
# hourly_min_age_hours = 2 # Wait 2 hours (more conservative)
# hourly_min_age_hours = 0 # Compact immediately (aggressive)
```
Setting `hourly_min_age_hours = 0` can compact the current hour while data is still being written, potentially creating many compacted files.
#### Minimum files [#minimum-files]
```toml
[compaction]
hourly_min_files = 10 # Only compact if >=10 files (default)
daily_min_files = 12 # Daily tier threshold (default)
# hourly_min_files = 50 # Only compact with many files
# hourly_min_files = 5 # Compact more aggressively
```
#### Concurrent jobs [#concurrent-jobs]
```toml
[compaction]
max_concurrent = 2 # Run 2 compactions in parallel (default)
# max_concurrent = 4 # More parallelism (uses more CPU/memory)
# max_concurrent = 1 # Sequential (lower resource usage)
```
#### Memory limit and threads (per subprocess) [#memory-limit-and-threads-per-subprocess]
`memory_limit` and `threads` are configurable starting in Arc **v26.09.1**. On earlier versions each compaction subprocess inherits the full `database.memory_limit` and uses all CPU cores.
Each compaction job runs in an isolated subprocess with its own query engine instance. These keys bound that instance's resources:
```toml
[compaction]
memory_limit = "" # Per-subprocess engine memory limit; "" (default) = auto
threads = 0 # Per-subprocess engine threads; 0 (default) = auto
# memory_limit = "2GB" # Explicit cap
# threads = 4 # Explicit thread count
```
Env vars: `ARC_COMPACTION_MEMORY_LIMIT`, `ARC_COMPACTION_THREADS`.
**Auto behavior:**
* `memory_limit` derives as `database.memory_limit / max_concurrent`, so all concurrent compaction jobs together stay within roughly one `database.memory_limit`. With `database.memory_limit = "8GB"` and the default concurrency of 2, each subprocess gets `4GB`.
* `threads` defaults to half the CPU cores (minimum 1), so the default two concurrent jobs together use about one machine's worth of cores, leaving headroom for ingest and queries.
Accepted `memory_limit` forms are absolute sizes with a unit: `"8GB"`, `"512MB"`, `"0.5GB"`. Percent and unit-less forms are rejected at startup (DuckDB's `SET memory_limit` does not support them), as are other invalid values. The effective values appear in the startup log (`subprocess_memory_limit`, `subprocess_threads`).
When a job exceeds its memory limit, DuckDB spills to a `duckdb-spill/` directory inside the job's temp directory (under `compaction.temp_directory`) — size that volume for your largest partitions. Spill files are removed by normal job cleanup and by the crash sweeps on startup.
On a dedicated compactor node these can be raised well above the defaults, since compaction is not competing with ingest or queries for RAM and cores on that host.
#### Files per batch [#files-per-batch]
`max_files_per_batch` is configurable starting in Arc **v26.09.1**. On earlier versions the batch size is fixed at 30 files and this setting has no effect.
A partition with more files than this is split into several batches, each compacted as an independent job producing its own output file.
```toml
[compaction]
max_files_per_batch = 30 # Files per compaction job (default)
# max_files_per_batch = 5 # Smaller outputs, more jobs per partition
# max_files_per_batch = 60 # Fewer, larger outputs
```
Valid range is **2–500**. Values outside it fall back to the default with a startup warning; `1` is rejected because compaction's adaptive retry cannot process a single-file batch.
This bounds the **file count** per job, not the output size in bytes — compacted file size tracks input file size, which follows your ingest buffer settings. The upper bound exists because a single `read_parquet()` call spanning too many files can abort.
#### Compression [#compression]
Compaction always writes its output with ZSTD, which is why compacted files are
substantially smaller than the freshly-ingested files they replace. This is not
configurable per tier.
The compression used for **incoming** writes is separate, and is set by
`ingest.compression` (default `snappy`) — see the
[configuration overview](/arc-enterprise/configuration/overview/).
### Disable compaction [#disable-compaction]
```toml
[compaction]
enabled = false
```
**When to disable:**
* Testing ingestion performance
* Very low write volume (\<10 files/hour)
* Debugging compaction issues
Disabling compaction will cause queries to slow down significantly as files accumulate.
## Monitoring [#monitoring]
### Check compaction status [#check-compaction-status]
```bash
curl http://localhost:8000/api/compaction/status \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response:**
```json
{
"enabled": true,
"running": false,
"last_run": "2025-10-08T14:05:00Z",
"next_run": "2025-10-08T15:05:00Z",
"stats": {
"total_jobs": 42,
"successful_jobs": 40,
"failed_jobs": 2,
"total_files_compacted": 12580,
"total_bytes_saved": 8589934592
}
}
```
### Get detailed statistics [#get-detailed-statistics]
```bash
curl http://localhost:8000/api/compaction/stats \
-H "Authorization: Bearer $ARC_TOKEN"
```
### List eligible partitions [#list-eligible-partitions]
```bash
curl http://localhost:8000/api/compaction/candidates \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response:**
```json
{
"candidates": [
{
"partition": "default/cpu/2025/10/08/14",
"file_count": 150,
"total_size_mb": 7500,
"age_hours": 2.5,
"eligible": true
},
{
"partition": "default/mem/2025/10/08/14",
"file_count": 120,
"total_size_mb": 6000,
"age_hours": 2.5,
"eligible": true
}
],
"total_candidates": 2
}
```
### Manually trigger compaction [#manually-trigger-compaction]
```bash
curl -X POST http://localhost:8000/api/compaction/trigger \
-H "Authorization: Bearer $ARC_TOKEN"
```
### View active jobs [#view-active-jobs]
```bash
curl http://localhost:8000/api/compaction/jobs \
-H "Authorization: Bearer $ARC_TOKEN"
```
### View job history [#view-job-history]
```bash
curl http://localhost:8000/api/compaction/history \
-H "Authorization: Bearer $ARC_TOKEN"
```
## Performance impact [#performance-impact]
### Compaction performance [#compaction-performance]
**Test Environment:** Apple M3 Max (14 cores, 36GB RAM)
| Files | Size | Compaction Time | Final Size | Compression |
| ----- | ------ | --------------- | ---------- | ----------- |
| 888 | 1.2 GB | 2.1s | 239 MB | 80.3% |
| 906 | 1.2 GB | 2.2s | 242 MB | 80.4% |
| 910 | 1.2 GB | 2.3s | 243 MB | 80.5% |
**Total:** 2,704 files (3.7 GB) → 3 files (724 MB) in **6.6 seconds**
### Query performance [#query-performance]
**Before Compaction:**
```sql
SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL 1 HOUR;
-- 5.2 seconds (scan 720 files)
```
**After Compaction:**
```sql
SELECT * FROM default.cpu WHERE time > NOW() - INTERVAL 1 HOUR;
-- 0.05 seconds (scan 1 file) - 104x faster!
```
### Storage savings [#storage-savings]
```text
Original files (Snappy): 3.7 GB
Compacted files (ZSTD): 724 MB
Space saved: 80.4%
```
## Best practices [#best-practices]
### 1. Let compaction run automatically [#1-let-compaction-run-automatically]
The default schedule (hourly) works well for most use cases:
```toml
[compaction]
enabled = true
hourly_schedule = "5 * * * *"
```
### 2. Monitor compaction jobs [#2-monitor-compaction-jobs]
Set up alerts for:
* Failed compaction jobs
* Partitions with >1000 files
* Compaction taking >10 minutes
### 3. Adjust based on write volume [#3-adjust-based-on-write-volume]
**High write volume:**
```toml
[compaction]
hourly_min_files = 100 # Wait for more files
max_concurrent = 4 # More parallelism
```
**Low write volume:**
```toml
[compaction]
hourly_min_files = 5 # Compact with fewer files
hourly_schedule = "0 */6 * * *" # Every 6 hours
```
### 4. Tune files per batch [#4-tune-files-per-batch]
```toml
[compaction]
max_files_per_batch = 30 # Files per compaction job (default)
# max_files_per_batch = 60 # Fewer, larger outputs
# max_files_per_batch = 5 # Smaller outputs, more jobs per partition
```
### 5. Reduce file generation at source [#5-reduce-file-generation-at-source]
**Best practice:** Increase buffer sizes to generate fewer files:
```toml
[ingest]
max_buffer_size = 200000 # Up from 50,000 (4x fewer files)
max_buffer_age_ms = 10000 # Up from 5000 (2x fewer files)
```
**Impact:**
* Files generated: 2,000/hour → 250/hour (8x reduction)
* Compaction time: substantially reduced
* Memory usage: +300MB per worker
This is the **most effective optimization** - fewer files means faster compaction AND faster queries.
## Troubleshooting [#troubleshooting]
### Compaction not running [#compaction-not-running]
**Check status:**
```bash
curl http://localhost:8000/api/compaction/status
```
**Verify configuration:**
```bash
# Check if enabled
grep "enabled" arc.toml
# Check schedule
grep "schedule" arc.toml
```
**Check logs:**
```bash
# Docker
docker logs arc | grep compaction
# Native
sudo journalctl -u arc | grep compaction
```
### Compaction taking too long [#compaction-taking-too-long]
**Symptoms:** Compaction jobs running for >30 minutes
**Solutions:**
1. **Reduce files per batch:**
```toml
[compaction]
max_files_per_batch = 10 # Smaller compaction jobs
```
2. **Increase parallelism:**
```toml
[compaction]
max_concurrent = 4
```
3. **Reduce files at source:**
```toml
[ingest]
max_buffer_size = 200000
```
### Out of disk space during compaction [#out-of-disk-space-during-compaction]
**Symptoms:** Compaction fails with disk space errors
**Solutions:**
1. **Use temp directory on larger disk:**
```bash
export TMPDIR=/mnt/large-disk/tmp
```
2. **Reduce concurrent jobs:**
```toml
[compaction]
max_concurrent = 1
```
3. **Clean up old compacted files manually:**
```bash
# Remove small files that were already compacted
find ./data -name "*.parquet" -size -10M -delete
```
### Compaction locks not releasing [#compaction-locks-not-releasing]
**Symptoms:** Partitions stuck in "locked" state
**Check locks:**
```bash
# View active locks
sqlite3 ./data/arc.db "SELECT * FROM compaction_locks;"
```
**Clear stale locks:**
```bash
# Locks expire automatically after 2 hours
# Or manually clear:
sqlite3 ./data/arc.db "DELETE FROM compaction_locks WHERE expires_at < datetime('now');"
```
## API reference [#api-reference]
### GET /api/v1/compaction/status [#get-apiv1compactionstatus]
Get current compaction status.
**Response:**
```json
{
"enabled": true,
"running": false,
"last_run": "2025-10-08T14:05:00Z",
"next_run": "2025-10-08T15:05:00Z"
}
```
### GET /api/v1/compaction/stats [#get-apiv1compactionstats]
Get detailed compaction statistics.
### GET /api/v1/compaction/candidates [#get-apiv1compactioncandidates]
List partitions eligible for compaction.
### POST /api/v1/compaction/trigger [#post-apiv1compactiontrigger]
Manually trigger compaction.
**Response:**
```json
{
"message": "Compaction triggered",
"job_id": "comp_1696775400"
}
```
### GET /api/v1/compaction/jobs [#get-apiv1compactionjobs]
View active compaction jobs.
### GET /api/v1/compaction/history [#get-apiv1compactionhistory]
View compaction job history.
## Summary [#summary]
Compaction is essential for production deployments:
**Benefits:**
* 10-50x faster queries
* 80% storage savings
* 99% fewer API calls
* Automatic and safe
**Default configuration works for most cases:**
```toml
[compaction]
enabled = true
hourly_schedule = "5 * * * *"
hourly_min_age_hours = 1
hourly_min_files = 10
```
**Monitor regularly:**
* Check `/api/v1/compaction/status`
* Alert on failed jobs
* Watch for partitions with >1000 files
## Next steps [#next-steps]
* **[Monitor Compaction](/arc-enterprise/operations/telemetry/)** - Set up health checks
* **[Configure WAL](/arc-enterprise/advanced/wal/)** - Add durability guarantees
* **[Tune Performance](/arc-enterprise/performance/benchmarks/)** - Optimize for your workload
# Data-Time Partitioning (/arc-enterprise/advanced/data-time-partitioning)
Arc organizes Parquet files by the data's timestamp rather than ingestion time, enabling proper backfill of historical data and optimal query performance.
## Overview [#overview]
Data-time partitioning ensures that your data lands in the correct time-based partitions based on when the events actually occurred, not when they were ingested into Arc.
**Key Features:**
* **Historical backfill** - Past data lands in correct partitions (e.g., December 2024 data goes to `2024/12/` folders)
* **Sorted files** - Data is sorted by timestamp within each Parquet file
* **Automatic splitting** - Batches spanning multiple hours are split into separate files
* **Partition pruning** - Enables accurate time-range query optimization
Data-time partitioning is **enabled by default** and requires no configuration.
## Why it matters [#why-it-matters]
### The problem with ingestion-time partitioning [#the-problem-with-ingestion-time-partitioning]
Traditional ingestion-time partitioning creates problems when backfilling historical data:
```yaml
Scenario: Ingesting December 2024 sensor data on January 4, 2025
❌ Ingestion-time partitioning:
data/mydb/cpu/2025/01/04/... (wrong - today's partition)
✅ Data-time partitioning:
data/mydb/cpu/2024/12/01/14/... (correct - data's timestamp)
data/mydb/cpu/2024/12/01/15/...
```
**Impact:**
* **Broken queries** - Time-range queries can't find historical data
* **No partition pruning** - The query engine must scan all files, not just relevant partitions
* **Mixed data** - Historical and current data mixed in same partition
* **Poor compaction** - Files with mixed timestamps don't compact efficiently
### After data-time partitioning [#after-data-time-partitioning]
With data-time partitioning, your data is always organized correctly:
```sql
-- Query for December 2024 data only scans December partitions
SELECT * FROM mydb.cpu
WHERE time >= '2024-12-01' AND time < '2025-01-01'
→ Arc scans only: data/mydb/cpu/2024/12/**/*.parquet
→ Skips all 2025 partitions entirely
```
**Benefits:**
* **Faster queries** - Partition pruning eliminates irrelevant files
* **Accurate historical analysis** - Data lives where it belongs
* **Efficient compaction** - Files with similar timestamps compact together
* **Predictable storage** - Easy to manage retention by date folders
## How it works [#how-it-works]
### Single-hour batches [#single-hour-batches]
When all records in a batch fall within the same hour:
```text
Incoming batch (all records from 2024-12-15 14:xx):
┌─────────────────────────┬────────┬───────┐
│ time │ host │ value │
├─────────────────────────┼────────┼───────┤
│ 2024-12-15T14:05:00.000 │ srv01 │ 45.2 │
│ 2024-12-15T14:32:00.000 │ srv01 │ 47.8 │
│ 2024-12-15T14:58:00.000 │ srv01 │ 44.1 │
└─────────────────────────┴────────┴───────┘
Result: Single sorted file
→ data/mydb/cpu/2024/12/15/14/abc123.parquet
(records sorted by timestamp)
```
### Multi-hour batches [#multi-hour-batches]
When a batch spans multiple hours, Arc automatically splits it:
```text
Incoming batch (records spanning 14:00-16:00):
┌─────────────────────────┬────────┬───────┐
│ time │ host │ value │
├─────────────────────────┼────────┼───────┤
│ 2024-12-15T14:30:00.000 │ srv01 │ 45.2 │
│ 2024-12-15T15:15:00.000 │ srv01 │ 47.8 │
│ 2024-12-15T15:45:00.000 │ srv01 │ 46.3 │
│ 2024-12-15T16:10:00.000 │ srv01 │ 44.1 │
└─────────────────────────┴────────┴───────┘
Result: Three separate sorted files
→ data/mydb/cpu/2024/12/15/14/abc123.parquet (1 record)
→ data/mydb/cpu/2024/12/15/15/def456.parquet (2 records)
→ data/mydb/cpu/2024/12/15/16/ghi789.parquet (1 record)
```
### Partition structure [#partition-structure]
Data is organized hierarchically by time:
```text
data/ # Storage root
├── default/ # Database
│ └── cpu/ # Measurement
│ ├── 2024/ # Year
│ │ └── 12/ # Month
│ │ ├── 01/ # Day
│ │ │ ├── 14/ # Hour (2 PM)
│ │ │ │ └── abc123.parquet
│ │ │ └── 15/ # Hour (3 PM)
│ │ │ └── def456.parquet
│ │ └── 15/ # Day 15
│ │ └── ...
│ └── 2025/ # Year
│ └── 01/ # Month
│ └── ...
```
## Sorting within files [#sorting-within-files]
Each Parquet file contains data sorted by timestamp in ascending order:
```sql
-- Data is pre-sorted, enabling efficient scans
-- Sorted file metadata enables:
-- - Early termination on LIMIT queries
-- - Efficient MIN/MAX aggregations
-- - Optimized range scans
SELECT * FROM mydb.cpu
WHERE time >= '2024-12-15T14:00:00'
AND time < '2024-12-15T14:30:00'
ORDER BY time
LIMIT 100
```
**Performance benefits:**
* **No runtime sorting** - Data already ordered
* **Efficient LIMIT** - Stop scanning after N rows
* **Fast aggregations** - MIN/MAX read file metadata
* **Optimal compression** - Similar timestamps compress better
## UTC consistency [#utc-consistency]
All partition paths use UTC time, regardless of server timezone:
```text
Server in New York (UTC-5):
Local time: 2024-12-15 10:00 EST
UTC time: 2024-12-15 15:00 UTC
→ Data written to: data/mydb/cpu/2024/12/15/15/...
(UTC hour, not local hour)
```
Using UTC ensures consistent partitioning across servers in different timezones and prevents partition misalignment during timezone changes (DST).
## Query partition pruning [#query-partition-pruning]
Arc's query engine automatically prunes partitions based on time predicates:
```sql
-- This query only scans December 2024 partitions
SELECT host, AVG(value) as avg_value
FROM mydb.cpu
WHERE time >= '2024-12-01T00:00:00Z'
AND time < '2025-01-01T00:00:00Z'
GROUP BY host
```
**What happens:**
1. Arc parses the time range from the WHERE clause
2. Converts range to partition paths: `2024/12/**/*.parquet`
3. The query engine receives only the relevant file list
4. Files outside the range are never opened
**Performance impact:**
* Querying 1 month in a year of data → \~92% fewer files scanned
* Querying 1 day in a month of data → \~97% fewer files scanned
* Querying 1 hour in a day of data → \~96% fewer files scanned
## Backfilling historical data [#backfilling-historical-data]
Data-time partitioning makes historical backfill straightforward:
```python
from arc_client import ArcClient
# Backfill sensor data from December 2024
# (even though we're ingesting in January 2025)
historical_data = {
"time": [
1701388800000000, # 2024-12-01T00:00:00Z
1701475200000000, # 2024-12-02T00:00:00Z
1701561600000000, # 2024-12-03T00:00:00Z
],
"sensor_id": ["temp-01", "temp-01", "temp-01"],
"value": [22.5, 23.1, 21.8],
}
with ArcClient(host="localhost", token="your-token") as client:
client.write.write_columnar(
measurement="sensors",
columns=historical_data,
)
# Data lands in correct partitions:
# → data/default/sensors/2024/12/01/00/...
# → data/default/sensors/2024/12/02/00/...
# → data/default/sensors/2024/12/03/00/...
```
## Interaction with compaction [#interaction-with-compaction]
Data-time partitioning works seamlessly with [file compaction](/arc-enterprise/advanced/compaction/):
1. **Ingestion** - Small files written to correct hourly partitions
2. **Compaction** - Files within each partition merged into larger files
3. **Result** - Each hour has one large, sorted, optimized file
```text
Before compaction:
data/mydb/cpu/2024/12/15/14/
├── file1.parquet (5 MB, 100K records)
├── file2.parquet (4 MB, 80K records)
├── file3.parquet (6 MB, 120K records)
└── ... (100 more small files)
After compaction:
data/mydb/cpu/2024/12/15/14/
└── compacted_abc123.parquet (450 MB, 10M records, sorted)
```
## Best practices [#best-practices]
### Timestamp requirements [#timestamp-requirements]
Ensure your timestamps are accurate:
```python
# ✅ Good: Microsecond Unix timestamps (UTC)
"time": [1701388800000000, 1701388801000000]
# ✅ Good: Nanosecond Unix timestamps (UTC)
"time": [1701388800000000000, 1701388801000000000]
# ❌ Bad: String timestamps (require parsing)
"time": ["2024-12-01T00:00:00Z", "2024-12-01T00:00:01Z"]
```
### Bulk imports [#bulk-imports]
When importing large historical datasets:
1. **Sort by time first** - Pre-sorted data writes faster
2. **Batch by hour** - Reduces file splitting overhead
3. **Use columnar format** - MessagePack columnar is fastest
4. **Trigger compaction after** - Consolidate small files
```bash
# After bulk import, trigger compaction
curl -X POST http://localhost:8000/api/v1/compaction/hourly \
-H "Authorization: Bearer $TOKEN"
```
### Monitoring partition distribution [#monitoring-partition-distribution]
Check that data is landing in expected partitions:
```sql
-- View partition distribution
SELECT
EXTRACT(YEAR FROM time) as year,
EXTRACT(MONTH FROM time) as month,
COUNT(*) as records
FROM mydb.sensors
GROUP BY year, month
ORDER BY year, month
```
## Next steps [#next-steps]
* [File Compaction](/arc-enterprise/advanced/compaction/) - Optimize partitioned files
* [Retention Policies](/arc-enterprise/data-lifecycle/retention-policies/) - Manage data by partition age
* [Query Performance](/arc-enterprise/performance/benchmarks/) - Benchmark partition pruning benefits
# Advanced (/arc-enterprise/advanced)
These pages describe mechanisms rather than features. They matter in an Enterprise deployment because each one lands on a specific node role: the WAL is a writer concern, compaction usually runs on a dedicated compactor, and query caching lives on readers.
# Write-Ahead Log (WAL) (/arc-enterprise/advanced/wal)
Arc's Write-Ahead Log (WAL) provides **zero data loss guarantees** on system crashes.
There are **two different WAL features** in Arc:
1. **SQLite WAL mode** (always enabled) - Internal mode for Arc's metadata database (`arc.db`). This enables concurrent access to connection settings, export jobs, and compaction locks. You'll see the log message `"SQLite WAL mode enabled for concurrent access"` on startup - this is expected and not related to data ingestion.
2. **Arc's WAL feature** (disabled by default) - Optional durability feature for **data ingestion** that provides zero data loss guarantees. This page documents the Arc WAL feature, controlled by the `WAL_ENABLED` environment variable.
**TL;DR**: The startup log `"SQLite WAL mode enabled"` is normal and does NOT mean Arc's data ingestion WAL is enabled.
## Overview [#overview]
WAL is an optional durability feature that persists all incoming data to disk **before** acknowledging writes. When enabled, Arc guarantees that data can be recovered even if the instance crashes.
WAL is **disabled by default** to maximize ingest throughput. Enable it when zero data loss is required.
### When to enable WAL [#when-to-enable-wal]
Enable WAL if you need:
* **Zero data loss** on system crashes
* **Guaranteed durability** for regulatory compliance (finance, healthcare)
* **Recovery from unexpected failures** (power loss, OOM kills)
Keep WAL disabled if you:
* **Prioritize maximum ingest throughput**
* **Can tolerate 0-5 seconds data loss** on rare crashes
* **Have client-side retry logic** or message queue upstream
### Performance vs durability tradeoff [#performance-vs-durability-tradeoff]
| Configuration | Throughput | Data Loss Risk |
| -------------------- | ---------------- | -------------- |
| **No WAL (default)** | Highest | 0-5 seconds |
| **WAL + async** | Slightly reduced | \<1 second |
| **WAL + fdatasync** | Slightly reduced | Near-zero |
| **WAL + fsync** | Slightly reduced | Zero |
**Tradeoff**: \~20% throughput reduction for near-zero data loss (fdatasync mode)
## Architecture [#architecture]
### Data flow with WAL [#data-flow-with-wal]
```text
┌──────────────────────────────────────────────────────────┐
│ HTTP Request (MessagePack or Line Protocol) │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 1. WAL.append(records) │
│ - Serialize to MessagePack binary │
│ - Calculate CRC32 checksum │
│ - Write to disk │
│ - fdatasync() ← Force physical disk sync │
└──────────────────┬───────────────────────────────────────┘
│
▼ Data is DURABLE (on disk)
┌──────────────────────────────────────────────────────────┐
│ 2. HTTP 202 Accepted ← Response to client │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 3. Buffer.write(records) │
│ - Add to in-memory buffer │
│ - Flush when 50K records or 5 seconds │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 4. Parquet Writer │
│ - Convert to Arrow columnar format │
│ - Write Parquet file │
│ - Upload to S3/MinIO │
└──────────────────┬───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────┐
│ 5. WAL.mark_completed() ← Can now delete WAL entry │
└──────────────────────────────────────────────────────────┘
```
Once WAL confirms the write (step 1), the data is **guaranteed durable** even if Arc crashes before step 4 completes.
### WAL files [#wal-files]
Arc uses a single WAL writer with goroutines for concurrent access:
```text
./data/wal/
├── arc-20251008_140530.wal
└── arc-20251008_150530.wal
```
**Benefits:**
* Simple implementation
* Automatic rotation
* Parallel recovery on startup
## Configuration [#configuration]
### Enable WAL [#enable-wal]
Edit `arc.toml`:
```toml
[wal]
enabled = true
sync_mode = "fdatasync" # Recommended for production
directory = "./data/wal"
max_size_mb = 500 # Rotate at 500MB
max_age_seconds = 3600 # Rotate after 1 hour
```
Or via environment variables:
```bash
ARC_WAL_ENABLED=true
ARC_WAL_DIRECTORY=./data/wal
ARC_WAL_SYNC_MODE=fdatasync
ARC_WAL_MAX_SIZE_MB=500
ARC_WAL_MAX_AGE_SECONDS=3600
```
### Sync modes [#sync-modes]
Arc supports three sync modes with different durability/performance tradeoffs:
#### fdatasync (recommended) [#fdatasync-recommended]
```toml
[wal]
sync_mode = "fdatasync"
```
**How it works:**
* Syncs data to disk (file contents)
* Skips metadata sync (file size, modified time)
* 50% faster than `fsync`, nearly same durability
**Guarantees:**
* Data is on physical disk
* Can recover all data on crash
* File metadata may be stale (not critical)
**Use case**: Production deployments (recommended)
#### fsync (maximum safety) [#fsync-maximum-safety]
```toml
[wal]
sync_mode = "fsync"
```
**How it works:**
* Syncs both data AND metadata to disk
* Slowest, but absolute guarantee
**Use when:**
* Regulatory compliance requires it
* Zero tolerance for any data loss
* Performance is secondary
#### async (performance-first) [#async-performance-first]
```toml
[wal]
sync_mode = "async"
```
**How it works:**
* Writes to OS buffer cache
* No explicit sync (OS flushes periodically)
* Very fast, but small risk window
**Use when:**
* Need 90% of original throughput
* Can tolerate \~1 second data loss
* Have upstream retry mechanisms
### Rotation settings [#rotation-settings]
Control when WAL files rotate:
```toml
[wal]
max_size_mb = 100 # Rotate when file reaches 100MB
max_age_seconds = 3600 # Rotate after 1 hour (even if file is small)
```
**Why rotation matters:**
* Prevents unbounded growth
* Faster recovery (smaller files)
* Automatic cleanup of old WALs
## Operations [#operations]
### Recovery on startup [#recovery-on-startup]
Arc automatically recovers from WAL files on startup:
```text
2025-10-08 14:30:00 [INFO] WAL recovery started: 4 files
2025-10-08 14:30:01 [INFO] Recovering WAL: worker-1-20251008_143000.wal
2025-10-08 14:30:01 [INFO] WAL read complete: 1000 entries, 5242880 bytes, 0 corrupted
2025-10-08 14:30:02 [INFO] Recovering WAL: worker-2-20251008_143000.wal
...
2025-10-08 14:30:05 [INFO] WAL recovery complete: 4000 batches, 200000 entries, 0 corrupted
2025-10-08 14:30:05 [INFO] WAL archived: worker-1-20251008_143000.wal.recovered
```
**Process:**
1. Find all `*.wal` files in `WAL_DIR`
2. Read and validate each entry (checksum verification)
3. Replay records into buffer system
4. Archive recovered WAL as `*.wal.recovered`
5. Continue normal operations
**Recovery time:**
* \~5 seconds per 100MB WAL file
* Parallel recovery across workers
* Corrupted entries are skipped (logged)
## Monitoring [#monitoring]
### WAL status [#wal-status]
```bash
curl http://localhost:8000/api/wal/status \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response:**
```json
{
"enabled": true,
"configuration": {
"sync_mode": "fdatasync",
"worker_id": 1,
"current_file": "./data/wal/worker-1-20251008_143000.wal"
},
"stats": {
"current_size_mb": 45.2,
"current_age_seconds": 1850,
"total_entries": 5000,
"total_bytes": 47382528,
"total_syncs": 5000,
"total_rotations": 2
}
}
```
### WAL files [#wal-files-1]
```bash
curl http://localhost:8000/api/wal/files \
-H "Authorization: Bearer $ARC_TOKEN"
```
**Response:**
```json
{
"active": [
{
"name": "worker-1-20251008_143000.wal",
"size_mb": 45.2,
"modified": 1696775400
}
],
"recovered": [
{
"name": "worker-1-20251008_120000.wal.recovered",
"size_mb": 98.5,
"modified": 1696768800
}
],
"total_size_mb": 143.7
}
```
### Health check [#health-check]
```bash
curl http://localhost:8000/api/wal/health \
-H "Authorization: Bearer $ARC_TOKEN"
```
### Cleanup old WAL files [#cleanup-old-wal-files]
```bash
# Cleanup files older than 24 hours (default)
curl -X POST http://localhost:8000/api/wal/cleanup \
-H "Authorization: Bearer $ARC_TOKEN"
# Custom age (in hours)
curl -X POST "http://localhost:8000/api/wal/cleanup?max_age_hours=48" \
-H "Authorization: Bearer $ARC_TOKEN"
```
## Troubleshooting [#troubleshooting]
### WAL recovery taking too long [#wal-recovery-taking-too-long]
**Symptoms:**
```text
2025-10-08 14:30:00 [INFO] WAL recovery started: 50 files
... (minutes pass) ...
```
**Solutions:**
1. **Adjust rotation settings:**
```toml
[wal]
max_size_mb = 50 # Smaller files, faster recovery
max_age_seconds = 1800 # Rotate more frequently
```
2. **Use faster disks for WAL:**
```toml
[wal]
directory = "/mnt/nvme/arc-wal" # NVMe SSD
```
3. **Use faster storage:**
* NVMe SSD for WAL directory
* Separate disk from data storage
### WAL disk space growing [#wal-disk-space-growing]
**Symptoms:**
```bash
$ du -sh ./data/wal
5.2G ./data/wal
```
**Solutions:**
1. **Manual cleanup:**
```bash
rm -f ./data/wal/*.wal.recovered
```
2. **Reduce retention:**
```toml
[wal]
max_size_mb = 50 # Rotate sooner
max_age_seconds = 1800 # 30 minutes
```
3. **Add cron job for cleanup:**
```bash
# Cleanup recovered WALs older than 24 hours
0 2 * * * find /path/to/data/wal -name "*.wal.recovered" -mtime +1 -delete
```
### WAL write failures [#wal-write-failures]
**Symptoms:**
```text
2025-10-08 14:30:00 [ERROR] WAL append failed: [Errno 28] No space left on device
```
**Solutions:**
1. **Check disk space:**
```bash
df -h /path/to/WAL_DIR
```
2. **Check permissions:**
```bash
ls -ld ./data/wal
chmod 755 ./data/wal
```
3. **Move WAL to larger disk:**
```toml
[wal]
directory = "/mnt/large-disk/arc-wal"
```
### Performance degradation with WAL [#performance-degradation-with-wal]
**Symptoms:**
* Ingest throughput dropped sharply after enabling WAL
* High CPU usage from fsync calls
**Solutions:**
1. **Verify sync mode:**
```toml
[wal]
sync_mode = "fdatasync" # Should be fdatasync, not fsync
```
2. **Check disk I/O wait:**
```bash
iostat -x 1
# Look for %iowait > 50%
```
3. **Move WAL to faster disk:**
```toml
[wal]
directory = "/mnt/nvme/arc-wal"
```
4. **Consider disabling WAL if durability isn't critical:**
```toml
[wal]
enabled = false
```
## Best practices [#best-practices]
### Production deployment [#production-deployment]
**Recommended configuration:**
```toml
[wal]
enabled = true
sync_mode = "fdatasync"
directory = "/mnt/fast-ssd/arc-wal"
max_size_mb = 100
max_age_seconds = 3600
```
**Monitoring setup:**
1. Monitor WAL disk usage
2. Alert on write failures
3. Track recovery time during restarts
4. Log rotation metrics
**Backup strategy:**
* WAL files are ephemeral (deleted after recovery)
* Don't backup WAL files directly
* Backup final Parquet files in S3/MinIO instead
### Development/testing [#developmenttesting]
**Recommended configuration:**
```toml
[wal]
enabled = false # WAL disabled for maximum speed
```
**Or if testing WAL:**
```toml
[wal]
enabled = true
sync_mode = "async"
max_size_mb = 10 # Small files for testing
```
## Summary [#summary]
**Enable WAL if:**
* Zero data loss is required
* Regulated industry (finance, healthcare)
* Can accept 19% throughput reduction
**Disable WAL if:**
* Maximum throughput is priority
* Can tolerate 0-5s data loss risk
* Have upstream retry/queue mechanisms
**Recommended settings:**
```toml
[wal]
enabled = true
sync_mode = "fdatasync" # Best balance
directory = "/mnt/nvme/arc-wal" # Fast disk
```
## Next steps [#next-steps]
* **[Configure Compaction](/arc-enterprise/advanced/compaction/)** - Optimize query performance
* **[Monitor Arc](/arc-enterprise/operations/telemetry/)** - Set up health checks
* **[Performance Tuning](/arc-enterprise/performance/benchmarks/)** - Maximize throughput
# API Reference (/arc-enterprise/api-reference)
Arc Enterprise serves the full Arc REST API plus the administrative routes that the licensed features add. Ingestion and query endpoints are identical to OSS, so existing clients keep working against a cluster endpoint unchanged.
What differs is authorization and routing. Tokens carry [RBAC](/arc-enterprise/security/rbac/) scopes, requests may be forwarded between nodes according to role, and [query governance](/arc-enterprise/query/query-governance/) can reject a request that exceeds a token's quota.
# API reference overview (/arc-enterprise/api-reference/overview)
Arc provides a comprehensive REST API for data ingestion, querying, and management.
## Base URL [#base-url]
```text
http://localhost:8000
```
## Authentication [#authentication]
All endpoints (except public ones) require authentication. Arc supports multiple authentication methods for compatibility with various clients:
### Bearer token (standard) [#bearer-token-standard]
```bash
curl -H "Authorization: Bearer $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### Token header (InfluxDB 2.x style) [#token-header-influxdb-2x-style]
```bash
curl -H "Authorization: Token $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### API key header [#api-key-header]
```bash
curl -H "x-api-key: $ARC_TOKEN" http://localhost:8000/api/v1/query
```
### Query parameter (InfluxDB 1.x style) [#query-parameter-influxdb-1x-style]
For InfluxDB 1.x client compatibility, tokens can be passed via the `p` query parameter:
```bash
curl "http://localhost:8000/write?db=mydb&p=$ARC_TOKEN" -d 'cpu,host=server01 usage=45.2'
```
### Public endpoints (no auth required) [#public-endpoints-no-auth-required]
* `GET /health` - Health check
* `GET /ready` - Readiness probe
* `GET /metrics` - Prometheus metrics
* `GET /api/v1/auth/verify` - Token verification
## Quick examples [#quick-examples]
### Write data (MessagePack) [#write-data-messagepack]
```python
import os
import msgpack
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
data = {
"m": "cpu",
"columns": {
"time": [1697472000000],
"host": ["server01"],
"usage": [45.2]
}
}
response = requests.post(
"http://localhost:8000/api/v1/write/msgpack",
headers={
"Authorization": f"Bearer {ARC_TOKEN}",
"Content-Type": "application/msgpack",
"x-arc-database": "default"
},
data=msgpack.packb(data)
)
```
### Query data (JSON) [#query-data-json]
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM default.cpu LIMIT 10", "format": "json"}'
```
### Query data (Apache Arrow) [#query-data-apache-arrow]
For large result sets, use Arrow format for the highest sustained row throughput:
```python
import os
import requests
import pyarrow as pa
ARC_TOKEN = os.environ["ARC_TOKEN"]
response = requests.post(
"http://localhost:8000/api/v1/query/arrow",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"sql": "SELECT * FROM default.cpu LIMIT 100000"}
)
reader = pa.ipc.open_stream(response.content)
arrow_table = reader.read_all()
```
### Health check [#health-check]
```bash
curl http://localhost:8000/health
```
***
## Health & monitoring [#health--monitoring]
### GET /health [#get-health]
Health check endpoint.
**Response:**
```json
{
"status": "ok",
"time": "2024-12-02T10:30:00Z",
"uptime": "1h 23m 45s",
"uptime_sec": 5025
}
```
### GET /ready [#get-ready]
Kubernetes readiness probe.
**Response:**
```json
{
"status": "ready",
"time": "2024-12-02T10:30:00Z",
"uptime_sec": 5025
}
```
### GET /metrics [#get-metrics]
Prometheus-format metrics.
**Response:** `text/plain` (Prometheus format)
Or request JSON:
```bash
curl -H "Accept: application/json" http://localhost:8000/metrics
```
### GET /api/v1/metrics [#get-apiv1metrics]
All metrics in JSON format.
### GET /api/v1/metrics/memory [#get-apiv1metricsmemory]
Detailed memory statistics including the Go runtime and the query engine.
### GET /api/v1/metrics/query-pool [#get-apiv1metricsquery-pool]
Query engine connection pool statistics.
### GET /api/v1/metrics/endpoints [#get-apiv1metricsendpoints]
Per-endpoint request statistics.
### GET /api/v1/metrics/timeseries/:type [#get-apiv1metricstimeseriestype]
Timeseries metrics data.
**Parameters:**
* `:type` - `system`, `application`, or `api`
* `?duration_minutes=30` - Time range (default: 30, max: 1440)
### GET /api/v1/logs [#get-apiv1logs]
Recent application logs. **Requires an admin token** (`Authorization: Bearer `) when authentication is enabled.
**Query Parameters:**
* `?limit=100` - Number of logs (default: 100, max: 1000)
* `?level=error` - Filter by level (error, warn, info, debug)
* `?since_minutes=60` - Time range (default: 60, max: 1440)
***
## Data ingestion [#data-ingestion]
### POST /api/v1/write/msgpack [#post-apiv1writemsgpack]
High-performance MessagePack binary writes (recommended).
**Headers:**
* `Authorization: Bearer TOKEN`
* `Content-Type: application/msgpack`
* `Content-Encoding: gzip` (optional)
* `x-arc-database: default` (optional)
**Body (MessagePack):**
```json
{
"m": "measurement_name",
"columns": {
"time": [1697472000000, 1697472001000],
"host": ["server01", "server02"],
"value": [45.2, 67.8]
}
}
```
**Response:** `204 No Content`
### GET /api/v1/write/msgpack/stats [#get-apiv1writemsgpackstats]
MessagePack ingestion statistics.
### GET /api/v1/write/msgpack/spec [#get-apiv1writemsgpackspec]
MessagePack format specification.
### POST /write [#post-write]
InfluxDB 1.x Line Protocol compatible endpoint. This path matches InfluxDB's native API for drop-in client compatibility.
**Query Parameters:**
* `db` - Target database name (required)
* `rp` - Retention policy (optional, ignored)
* `precision` - Timestamp precision: `ns`, `us`, `ms`, `s` (default: `ns`)
* `p` - Authentication token (InfluxDB 1.x style)
**Headers:**
* `Content-Type: text/plain`
* `Authorization: Bearer TOKEN` (or use `p` query param)
**Body:**
```text
cpu,host=server01 usage=45.2 1697472000000000000
mem,host=server01 used=8.2,total=16.0 1697472000000000000
```
**Example:**
```bash
curl -X POST "http://localhost:8000/write?db=mydb&p=$ARC_TOKEN" \
-d 'cpu,host=server01 usage=45.2'
```
### POST /api/v2/write [#post-apiv2write]
InfluxDB 2.x compatible endpoint. This path matches InfluxDB's native API for drop-in client compatibility.
**Query Parameters:**
* `bucket` - Target database/bucket name (required)
* `org` - Organization (optional, ignored)
* `precision` - Timestamp precision: `ns`, `us`, `ms`, `s` (default: `ns`)
**Headers:**
* `Content-Type: text/plain`
* `Authorization: Token ARC_TOKEN` (InfluxDB 2.x style)
**Example:**
```bash
curl -X POST "http://localhost:8000/api/v2/write?bucket=mydb&org=myorg" \
-H "Authorization: Token $ARC_TOKEN" \
-d 'cpu,host=server01 usage=45.2'
```
### POST /api/v1/write/line-protocol [#post-apiv1writeline-protocol]
Arc-native Line Protocol endpoint. Uses headers instead of query parameters.
**Headers:**
* `Content-Type: text/plain`
* `Authorization: Bearer TOKEN`
* `x-arc-database: default` - Target database
### POST /api/v1/write/line-protocol/flush [#post-apiv1writeline-protocolflush]
Force buffer flush to disk.
### GET /api/v1/write/line-protocol/stats [#get-apiv1writeline-protocolstats]
Line Protocol ingestion statistics.
### GET /api/v1/write/line-protocol/health [#get-apiv1writeline-protocolhealth]
Line Protocol handler health.
***
## Querying [#querying]
### POST /api/v1/query [#post-apiv1query]
Execute SQL queries with JSON response.
**Request:**
```json
{
"sql": "SELECT * FROM default.cpu LIMIT 10",
"format": "json"
}
```
**Response:**
```json
{
"columns": ["time", "host", "usage"],
"types": ["TIMESTAMP", "VARCHAR", "DOUBLE"],
"data": [
[1697472000000, "server01", 45.2],
[1697472001000, "server02", 67.8]
],
"row_count": 2,
"execution_time_ms": 12
}
```
### POST /api/v1/query/arrow [#post-apiv1queryarrow]
Execute SQL queries with Apache Arrow IPC response.
**Request:**
```json
{
"sql": "SELECT * FROM default.cpu LIMIT 10000"
}
```
**Response:** `application/vnd.apache.arrow.stream`
### POST /api/v1/query/estimate [#post-apiv1queryestimate]
Estimate query cost before execution.
**Request:**
```json
{
"sql": "SELECT * FROM default.cpu WHERE time > now() - INTERVAL '1 hour'"
}
```
### GET /api/v1/measurements [#get-apiv1measurements]
List all measurements across databases.
### GET /api/v1/query/:measurement [#get-apiv1querymeasurement]
Query a specific measurement directly.
***
## Authentication [#authentication-1]
### GET /api/v1/auth/verify [#get-apiv1authverify]
Verify token validity (public endpoint).
**Response:**
```json
{
"valid": true,
"token_id": "abc123",
"name": "my-token",
"is_admin": false
}
```
### GET /api/v1/auth/tokens [#get-apiv1authtokens]
List all tokens (admin only).
### POST /api/v1/auth/tokens [#post-apiv1authtokens]
Create a new token (admin only).
**Request:**
```json
{
"name": "my-service",
"description": "Token for my service",
"is_admin": false
}
```
**Response:**
```json
{
"id": "abc123",
"name": "my-service",
"token": "arc_xxxxxxxxxxxxxxxxxxxxxxxx",
"is_admin": false,
"created_at": "2024-12-02T10:30:00Z"
}
```
### GET /api/v1/auth/tokens/:id [#get-apiv1authtokensid]
Get token details (admin only).
### DELETE /api/v1/auth/tokens/:id [#delete-apiv1authtokensid]
Delete/revoke a token (admin only).
### POST /api/v1/auth/tokens/:id/rotate [#post-apiv1authtokensidrotate]
Rotate a token (admin only).
### POST /api/v1/auth/tokens/:id/revoke [#post-apiv1authtokensidrevoke]
Revoke a token (admin only).
### GET /api/v1/auth/cache/stats [#get-apiv1authcachestats]
Token cache statistics (admin only).
### POST /api/v1/auth/cache/invalidate [#post-apiv1authcacheinvalidate]
Invalidate token cache (admin only).
***
## Compaction [#compaction]
### GET /api/v1/compaction/status [#get-apiv1compactionstatus]
Current compaction status.
**Response:**
```json
{
"enabled": true,
"running": false,
"last_run": "2024-12-02T10:00:00Z",
"next_run": "2024-12-02T11:00:00Z"
}
```
### GET /api/v1/compaction/stats [#get-apiv1compactionstats]
Compaction statistics.
### GET /api/v1/compaction/candidates [#get-apiv1compactioncandidates]
List files eligible for compaction.
### POST /api/v1/compaction/trigger [#post-apiv1compactiontrigger]
Manually trigger compaction.
**Request:**
```json
{
"database": "default",
"measurement": "cpu"
}
```
### GET /api/v1/compaction/jobs [#get-apiv1compactionjobs]
List active compaction jobs.
### GET /api/v1/compaction/history [#get-apiv1compactionhistory]
Compaction job history.
***
## Delete operations [#delete-operations]
### POST /api/v1/delete [#post-apiv1delete]
Delete data matching conditions.
**Request:**
```json
{
"database": "default",
"measurement": "cpu",
"where": "host = 'server01' AND time < '2024-01-01'",
"confirm": true
}
```
**Response:**
```json
{
"deleted_rows": 1523,
"deleted_files": 3
}
```
### GET /api/v1/delete/config [#get-apiv1deleteconfig]
Get delete operation configuration.
***
## Database management [#database-management]
Endpoints for managing databases programmatically.
### GET /api/v1/databases [#get-apiv1databases]
List all databases with measurement counts.
**Response:**
```json
{
"databases": [
{"name": "default", "measurement_count": 5},
{"name": "production", "measurement_count": 12}
],
"count": 2
}
```
### POST /api/v1/databases [#post-apiv1databases]
Create a new database.
**Request:**
```json
{
"name": "my_database"
}
```
**Response (201 Created):**
```json
{
"name": "my_database",
"measurement_count": 0,
"created_at": "2024-12-21T10:30:00Z"
}
```
**Validation rules:**
* Must start with a letter (a-z, A-Z)
* Can contain letters, numbers, underscores, and hyphens
* Maximum 64 characters
* Reserved names blocked: `system`, `internal`, `_internal`
**Error Response (400):**
```json
{
"error": "Invalid database name: must start with a letter and contain only alphanumeric characters, underscores, or hyphens"
}
```
### GET /api/v1/databases/:name [#get-apiv1databasesname]
Get information about a specific database.
**Response:**
```json
{
"name": "production",
"measurement_count": 12
}
```
**Error Response (404):**
```json
{
"error": "Database 'nonexistent' not found"
}
```
### GET /api/v1/databases/:name/measurements [#get-apiv1databasesnamemeasurements]
List all measurements in a database.
**Response:**
```json
{
"database": "production",
"measurements": [
{"name": "cpu"},
{"name": "memory"},
{"name": "disk"}
],
"count": 3
}
```
### DELETE /api/v1/databases/:name [#delete-apiv1databasesname]
Delete a database and all its data.
This operation is destructive and cannot be undone. Requires:
- `delete.enabled = true` in configuration
- `?confirm=true` query parameter
**Request:**
```bash
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/databases/old_data?confirm=true"
```
**Response:**
```json
{
"message": "Database 'old_data' deleted successfully",
"files_deleted": 47
}
```
**Error Responses:**
*Delete disabled (403):*
```json
{
"error": "Delete operations are disabled. Set delete.enabled=true in arc.toml to enable."
}
```
*Missing confirmation (400):*
```json
{
"error": "Confirmation required. Add ?confirm=true to delete the database."
}
```
***
## Retention policies [#retention-policies]
### POST /api/v1/retention [#post-apiv1retention]
Create a retention policy.
**Request:**
```json
{
"name": "30-day-retention",
"database": "default",
"measurement": "cpu",
"duration": "30d",
"schedule": "0 2 * * *"
}
```
### GET /api/v1/retention [#get-apiv1retention]
List all retention policies.
### GET /api/v1/retention/:id [#get-apiv1retentionid]
Get a specific policy.
### PUT /api/v1/retention/:id [#put-apiv1retentionid]
Update a retention policy.
### DELETE /api/v1/retention/:id [#delete-apiv1retentionid]
Delete a retention policy.
### POST /api/v1/retention/:id/execute [#post-apiv1retentionidexecute]
Execute a policy manually.
### GET /api/v1/retention/:id/executions [#get-apiv1retentionidexecutions]
Get policy execution history.
***
## Continuous queries [#continuous-queries]
### POST /api/v1/continuous\_queries [#post-apiv1continuous_queries]
Create a continuous query.
**Request:**
```json
{
"name": "hourly-rollup",
"source_database": "default",
"source_measurement": "cpu",
"destination_database": "default",
"destination_measurement": "cpu_hourly",
"query": "SELECT time_bucket('1 hour', time) as time, host, AVG(usage) as avg_usage FROM default.cpu GROUP BY 1, 2",
"schedule": "0 * * * *"
}
```
### GET /api/v1/continuous\_queries [#get-apiv1continuous_queries]
List all continuous queries.
### GET /api/v1/continuous\_queries/:id [#get-apiv1continuous_queriesid]
Get a specific continuous query.
### PUT /api/v1/continuous\_queries/:id [#put-apiv1continuous_queriesid]
Update a continuous query.
### DELETE /api/v1/continuous\_queries/:id [#delete-apiv1continuous_queriesid]
Delete a continuous query.
### POST /api/v1/continuous\_queries/:id/execute [#post-apiv1continuous_queriesidexecute]
Execute a continuous query manually.
### GET /api/v1/continuous\_queries/:id/executions [#get-apiv1continuous_queriesidexecutions]
Get execution history.
***
## MQTT subscriptions [#mqtt-subscriptions]
MQTT subscription management is available starting Arc v26.02.1.
Manage MQTT broker subscriptions for direct IoT data ingestion. See the [MQTT Integration Guide](/arc-enterprise/integrations/mqtt/) for detailed usage.
### POST /api/v1/mqtt/subscriptions [#post-apiv1mqttsubscriptions]
Create a new MQTT subscription.
**Request:**
```json
{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"qos": 1,
"auto_start": true
}
```
**Response (201 Created):**
```json
{
"id": "sub_abc123",
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"status": "running",
"created_at": "2026-02-01T10:00:00Z"
}
```
**Full options:**
| Field | Type | Required | Default | Description |
| --------------- | ------ | -------- | ------- | ------------------------------------ |
| `name` | string | Yes | - | Unique subscription name |
| `broker` | string | Yes | - | Broker URL (tcp\://, ssl://, ws\://) |
| `topics` | array | Yes | - | Topics to subscribe |
| `database` | string | Yes | - | Target Arc database |
| `qos` | int | No | 1 | QoS level: 0, 1, or 2 |
| `client_id` | string | No | auto | MQTT client ID |
| `username` | string | No | - | MQTT username |
| `password` | string | No | - | MQTT password (encrypted at rest) |
| `tls_enabled` | bool | No | false | Enable TLS/SSL |
| `tls_cert_path` | string | No | - | Client certificate path |
| `tls_key_path` | string | No | - | Client key path |
| `tls_ca_path` | string | No | - | CA certificate path |
| `topic_mapping` | object | No | \{} | Topic-to-measurement mapping |
| `auto_start` | bool | No | true | Start on creation and server restart |
### GET /api/v1/mqtt/subscriptions [#get-apiv1mqttsubscriptions]
List all MQTT subscriptions.
**Response:**
```json
{
"subscriptions": [
{
"id": "sub_abc123",
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"status": "running"
}
],
"count": 1
}
```
### GET /api/v1/mqtt/subscriptions/:id [#get-apiv1mqttsubscriptionsid]
Get subscription details.
### PUT /api/v1/mqtt/subscriptions/:id [#put-apiv1mqttsubscriptionsid]
Update a subscription. Subscription must be stopped first.
### DELETE /api/v1/mqtt/subscriptions/:id [#delete-apiv1mqttsubscriptionsid]
Delete a subscription. Subscription must be stopped first.
### POST /api/v1/mqtt/subscriptions/:id/start [#post-apiv1mqttsubscriptionsidstart]
Start a stopped subscription.
**Response:**
```json
{
"id": "sub_abc123",
"status": "running",
"message": "Subscription started"
}
```
### POST /api/v1/mqtt/subscriptions/:id/stop [#post-apiv1mqttsubscriptionsidstop]
Stop a running subscription.
### POST /api/v1/mqtt/subscriptions/:id/restart [#post-apiv1mqttsubscriptionsidrestart]
Restart a subscription (stop + start).
### GET /api/v1/mqtt/subscriptions/:id/stats [#get-apiv1mqttsubscriptionsidstats]
Get statistics for a specific subscription.
**Response:**
```json
{
"id": "sub_abc123",
"messages_received": 15420,
"bytes_received": 2458320,
"decode_errors": 0,
"last_message_at": "2026-02-01T10:30:15Z",
"topics": {
"sensors/temperature": 8500,
"sensors/humidity": 6920
}
}
```
### GET /api/v1/mqtt/stats [#get-apiv1mqttstats]
Aggregate statistics across all running subscriptions.
**Response:**
```json
{
"status": "success",
"running_count": 2,
"subscriptions_stats": {
"sub_abc123": { ... },
"sub_def456": { ... }
}
}
```
### GET /api/v1/mqtt/health [#get-apiv1mqtthealth]
MQTT service health check.
**Response:**
```json
{
"status": "healthy",
"healthy": true,
"running_count": 2,
"connected_count": 2,
"disconnected_count": 0,
"service": "mqtt_subscriptions"
}
```
***
## Response formats [#response-formats]
### Success response [#success-response]
```json
{
"status": "success",
"data": [...],
"count": 10
}
```
### Error response [#error-response]
```json
{
"error": "Error message"
}
```
### HTTP status codes [#http-status-codes]
* `200` - Success
* `204` - No Content (successful write)
* `400` - Bad Request
* `401` - Unauthorized
* `403` - Forbidden (requires admin)
* `404` - Not Found
* `500` - Internal Server Error
***
## Rate limiting [#rate-limiting]
Arc does not enforce rate limiting by default. For production deployments, consider:
* Reverse proxy rate limiting (Nginx, Traefik)
* API Gateway (AWS API Gateway, Kong)
* Application-level throttling
## CORS [#cors]
CORS is enabled by default with permissive settings. Configure via reverse proxy for production.
## Best practices [#best-practices]
### 1. Use MessagePack for writes [#1-use-messagepack-for-writes]
MessagePack is considerably faster than Line Protocol:
```python
# Fast: MessagePack columnar
data = {"m": "cpu", "columns": {...}}
requests.post(url, data=msgpack.packb(data))
# Slower: Line Protocol text
data = "cpu,host=server01 usage=45.2"
requests.post(url, data=data)
```
### 2. Batch your writes [#2-batch-your-writes]
Send multiple records per request:
```python
# Good: Batch write
data = {
"m": "cpu",
"columns": {
"time": [t1, t2, t3, ...],
"host": [h1, h2, h3, ...],
"usage": [u1, u2, u3, ...]
}
}
```
### 3. Use Arrow for large queries [#3-use-arrow-for-large-queries]
For 10K+ rows, use the Arrow endpoint:
```python
response = requests.post(url + "/api/v1/query/arrow", ...)
table = pa.ipc.open_stream(response.content).read_all()
df = table.to_pandas() # Zero-copy conversion
```
### 4. Enable gzip compression [#4-enable-gzip-compression]
```python
import gzip
compressed = gzip.compress(msgpack.packb(data))
requests.post(
url,
data=compressed,
headers={"Content-Encoding": "gzip", ...}
)
```
## Client libraries [#client-libraries]
### Python (official SDK) [#python-official-sdk]
```bash
pip install arc-tsdb-client[all]
```
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
client.write.write_columnar(
measurement="cpu",
columns={"time": [...], "host": [...], "usage": [...]},
)
df = client.query.query_pandas("SELECT * FROM default.cpu LIMIT 10")
```
See [Python SDK Documentation](/arc-enterprise/sdks/python/) for full details.
## Enterprise API endpoints [#enterprise-api-endpoints]
The following endpoints are available with an Arc Enterprise license.
### Clustering [#clustering]
| Method | Endpoint | Description |
| ------ | --------------------------- | ------------------ |
| `GET` | `/api/v1/cluster` | Cluster status |
| `GET` | `/api/v1/cluster/nodes` | List cluster nodes |
| `GET` | `/api/v1/cluster/nodes/:id` | Get specific node |
| `GET` | `/api/v1/cluster/local` | Local node info |
| `GET` | `/api/v1/cluster/health` | Health check |
See [Clustering & High Availability](/arc-enterprise/configuration/clustering/) for detailed API documentation.
### RBAC [#rbac]
| Method | Endpoint | Description |
| ----------------------- | ------------------------------------------ | ----------------------- |
| `POST/GET/PATCH/DELETE` | `/api/v1/rbac/organizations` | Organization management |
| `POST/GET/PATCH/DELETE` | `/api/v1/rbac/organizations/:org_id/teams` | Team management |
| `POST/GET/PATCH/DELETE` | `/api/v1/rbac/teams/:team_id/roles` | Role management |
| `POST/GET/DELETE` | `/api/v1/rbac/roles/:role_id/measurements` | Measurement permissions |
See [RBAC](/arc-enterprise/security/rbac/) for detailed API documentation.
### Tiered storage [#tiered-storage]
| Method | Endpoint | Description |
| --------------------- | -------------------------- | --------------------- |
| `GET` | `/api/v1/tiering/status` | Tiering status |
| `GET` | `/api/v1/tiering/files` | List files by tier |
| `POST` | `/api/v1/tiering/migrate` | Trigger migration |
| `GET` | `/api/v1/tiering/stats` | Migration statistics |
| `POST/GET/PUT/DELETE` | `/api/v1/tiering/policies` | Per-database policies |
See [Tiered Storage](/arc-enterprise/data-lifecycle/tiered-storage/) for detailed API documentation.
### Audit logging [#audit-logging]
| Method | Endpoint | Description |
| ------ | --------------------- | ---------------- |
| `GET` | `/api/v1/audit/logs` | Query audit logs |
| `GET` | `/api/v1/audit/stats` | Audit statistics |
See [Audit Logging](/arc-enterprise/security/audit-logging/) for detailed API documentation.
### Query governance [#query-governance]
| Method | Endpoint | Description |
| --------------------- | ------------------------------------ | ----------------- |
| `POST/GET/PUT/DELETE` | `/api/v1/governance/policies` | Policy management |
| `GET` | `/api/v1/governance/usage/:token_id` | Usage monitoring |
See [Query Governance](/arc-enterprise/query/query-governance/) for detailed API documentation.
### Query management [#query-management]
| Method | Endpoint | Description |
| -------- | ------------------------- | -------------- |
| `GET` | `/api/v1/queries/active` | Active queries |
| `GET` | `/api/v1/queries/history` | Query history |
| `GET` | `/api/v1/queries/:id` | Query details |
| `DELETE` | `/api/v1/queries/:id` | Cancel query |
See [Query Management](/arc-enterprise/query/query-management/) for detailed API documentation.
## Next steps [#next-steps]
* **[Python SDK](/arc-enterprise/sdks/python/)** - Official Python client
* **[Getting Started](/arc-enterprise/getting-started/)** - Quick start guide
* **[Configuration](/arc-enterprise/configuration/overview/)** - Server configuration
# Clustering & High Availability (/arc-enterprise/configuration/clustering)
Scale Arc horizontally with multi-node clusters. Separate write, read, and compaction workloads across dedicated nodes with automatic failover.
Arc Enterprise supports two cluster topologies: **shared object storage** and **local storage with peer replication**. See [Deployment Patterns](/arc-enterprise/configuration/deployment-patterns/) to choose the right one for your environment before configuring a cluster.
## Overview [#overview]
Arc Enterprise clustering uses a role-based architecture where each node in the cluster serves a specific purpose:
## Node roles [#node-roles]
| Role | Purpose | Capabilities |
| -------------- | ---------------------------------- | ------------------ |
| **writer** | Handles data ingestion and WAL | Ingest, coordinate |
| **reader** | Serves queries from shared storage | Query |
| **compactor** | Runs background file optimization | Compact |
| **standalone** | Single-node mode (default) | All capabilities |
* **Writers** receive data via the ingestion API, buffer it, and flush Parquet files to shared storage. WAL replication ensures durability.
* **Readers** query Parquet directly from shared storage. Scale readers horizontally to handle more concurrent queries.
* **Compactors** run hourly and daily file compaction in the background without impacting write or read performance.
## Configuration [#configuration]
### TOML configuration [#toml-configuration]
```toml
[cluster]
enabled = true
node_id = "writer-01" # Unique identifier for this node
role = "writer" # writer, reader, compactor, standalone
cluster_name = "production" # Cluster identifier
seeds = ["10.0.1.10:9000", "10.0.1.11:9000"] # Seed nodes for discovery
coordinator_addr = ":9000" # Address for inter-node communication
health_check_interval = 10 # Health check interval (seconds)
heartbeat_interval = 5 # Heartbeat interval (seconds)
replication_enabled = true # Enable WAL replication to readers
query_gate_on_catchup = false # See "Query gating during replication catch-up" below
```
### Environment variables [#environment-variables]
```bash
ARC_CLUSTER_ENABLED=true
ARC_CLUSTER_NODE_ID=writer-01
ARC_CLUSTER_ROLE=writer
ARC_CLUSTER_CLUSTER_NAME=production
ARC_CLUSTER_SEEDS=10.0.1.10:9000,10.0.1.11:9000
ARC_CLUSTER_COORDINATOR_ADDR=:9000
ARC_CLUSTER_HEALTH_CHECK_INTERVAL=10
ARC_CLUSTER_HEARTBEAT_INTERVAL=5
ARC_CLUSTER_REPLICATION_ENABLED=true
ARC_CLUSTER_QUERY_GATE_ON_CATCHUP=false
```
## Query gating during replication catch-up [#query-gating-during-replication-catch-up]
In a [local-storage cluster](/arc-enterprise/configuration/deployment-patterns/) with peer replication, a reader node may serve queries before its background puller has finished pulling all the Parquet files the cluster manifest references. Without gating, those queries silently return partial results: the manifest knows about the missing files, but `read_parquet()` globs against local storage and only finds what's already on disk. WAL replication (added in 26.05.1) closes part of this gap for unflushed writer data, but flushed Parquet files still depend on the asynchronous puller.
`cluster.query_gate_on_catchup` (added in 26.06.1, off by default) closes the remaining gap. When enabled, all user-facing read endpoints return `503 Service Unavailable` until peer file replication has fully converged on this node.
Turn this on if you'd rather a reader return 503 for a few seconds at startup than serve incomplete results. Leave it off if your application can tolerate eventual consistency during catch-up and you'd rather queries always succeed (the existing pre-26.06.1 behavior). Either choice is defensible; this is a correctness-vs-availability knob.
### What "fully converged" means [#what-fully-converged-means]
The gate is scoped to the **startup catch-up batch only** — not to all pull activity on the node. This distinction matters: in a busy cluster, steady-state ingest constantly puts new files in flight, and a naive "wait for everything to settle" predicate would mean the reader returns 503 every few seconds in normal operation. The gate's job is *"the reader has finished bootstrapping its view of the manifest as of startup,"* not *"no pulls are happening anywhere right now."*
A node is considered ready when **all** of the following are true:
1. The startup catch-up walker has finished its pass over the manifest.
2. No paths the walker tagged are still in flight (`catchup_inflight == 0`). Steady-state pulls from reactive FSM callbacks are deliberately excluded.
3. No catch-up-batch pulls failed after retries (`catchup_failed == 0`).
4. No catch-up-batch pulls were dropped due to queue saturation (`catchup_dropped == 0`).
Failures and drops outside the catch-up window do **not** keep the gate red. They're operational concerns surfaced via puller stats but not correctness blockers — by the time the catch-up batch has settled, the reader has reconciled its view of the manifest as of walker start. Steady-state failures are handled by reactive FSM callbacks (which re-enqueue), the Phase 5 reconciler, and operator alerting via the cumulative `failed` / `dropped` counters.
**Self-heal**: catch-up failures and drops both clear without a process restart. When a later pull succeeds for a previously-affected path (a reactive FSM callback re-enqueueing after the underlying issue resolves, or a subsequent catch-up scan), the corresponding scoped counter decrements and the gate re-opens automatically. The puller tracks affected paths in dedicated sets so it can attribute a successful pull back to the original failure or drop.
Both `catchup_failed` and `catchup_dropped` are surfaced in the 503 body so operators see exactly what happened. The `/api/v1/cluster/status` endpoint also exposes the cumulative `failed` / `dropped` / `pulled` / `skipped_dup` keys with their original whole-puller-lifetime semantics (preserved for dashboards landed before #392), alongside the new `catchup_*` keys for gate-relevant numbers. So dashboards can distinguish "the catch-up batch had a hiccup" (catchup\_failed > 0) from "the puller has been having steady-state problems for hours" (failed >> catchup\_failed).
If you set `cluster.replication_catchup_enabled=false` (the emergency off-switch for pathologically large manifests), the catch-up walker never runs and the gate would never clear. Arc detects this combination at startup, logs a `WARN`, and **auto-disables the gate** so the node isn't permanently 503'd. Operators see a clear log line and can fix the configuration at their leisure. Don't enable the gate if you've also disabled the walker.
### Endpoints affected [#endpoints-affected]
When the gate is enabled and the node is still catching up, these endpoints return 503:
* `POST /api/v1/query`
* `POST /api/v1/query/arrow`
* `POST /api/v1/query/estimate`
* `GET /api/v1/query/:measurement`
* `GET /api/v1/measurements`
Internal endpoints (cache invalidation, cluster status, replication-control APIs) are deliberately **not** gated — peer nodes need them to fire during catch-up.
### 503 response shape [#503-response-shape]
```json
{
"success": false,
"error": "replication_catch_up_in_progress",
"message": "Reader is still catching up on replicated files. Retry shortly or check /api/v1/cluster for catch-up progress.",
"catchup_status": {
"started_at": 1714912800,
"completed_at": 0,
"entries_walked": 1287,
"enqueued": 1287,
"catchup_inflight": 2,
"catchup_failed": 0,
"catchup_dropped": 0,
"queue_depth": 7,
"inflight_count": 2,
"pulled": 1278
}
}
```
A `Retry-After: 5` header is also set so HTTP-aware load balancers and clients can back off automatically.
`completed_at = 0` means the catch-up walker is still enumerating; once it flips non-zero, watch `queue_depth + inflight_count` go to zero. Non-zero `failed` or `dropped` means the gate will not clear without a node restart or a follow-up FSM callback.
### Observability [#observability]
* **Cumulative gate fires**: `QueryHandler.QueryGate503Total()` is exposed for Prometheus / metrics scrapes. Alert on a non-zero rate to detect that the gate is firing without inferring from generic HTTP error logs.
* **Sampled log line**: while the gate is active, Arc emits at most one `WARN` log per second with the gate counter and request path. Avoids flooding under sustained catch-up while still surfacing the degraded state.
* **Live status**: the `/api/v1/cluster` endpoint exposes `replication_catchup_status` with the same fields shown in the 503 body, so dashboards can show catch-up progress without waiting for a query to fail.
### Known limitation [#known-limitation]
There is a sub-millisecond window between the Raft FSM committing a `RegisterFile` entry and the puller's `Enqueue` callback firing. A query landing in that window can observe `ReplicationReady() == true` while a manifest entry from the same Raft commit is not yet in the in-flight set. Closing this gap requires a per-query Raft `LastApplied()` barrier on the query path, which is out of scope for this gate.
The gate's contract is *"every file the puller has observed has been pulled,"* not *"every file the manifest currently contains has been pulled."* In practice this means the gate may unblock a fraction of a second before the very last files committed before the gate-clear are queryable. This is a tracked follow-up.
### Pattern A vs. Pattern B [#pattern-a-vs-pattern-b]
* **Shared object storage** (Pattern A): the puller is disabled (`replication_enabled = false`), so `query_gate_on_catchup` is effectively a no-op — readers see the bucket directly and don't need to catch up. Safe to leave the flag at any value.
* **Local storage with peer replication** (Pattern B): this is where the gate matters. Enable it on readers whose application cannot tolerate partial results during cold start or after a network partition.
## Deployment example [#deployment-example]
A minimal 3-node cluster with one writer and two readers using Docker Compose:
```yaml
# docker-compose.yml
version: "3.8"
services:
# Shared storage (MinIO as S3-compatible backend)
minio:
image: minio/minio
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin123
ports:
- "9001:9001"
# Writer node (primary)
arc-writer:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: minio
ARC_STORAGE_S3_BUCKET: arc-data
ARC_STORAGE_S3_ENDPOINT: minio:9000
ARC_STORAGE_S3_ACCESS_KEY: minioadmin
ARC_STORAGE_S3_SECRET_KEY: minioadmin123
ARC_STORAGE_S3_USE_SSL: "false"
ARC_STORAGE_S3_PATH_STYLE: "true"
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: writer-01
ARC_CLUSTER_ROLE: writer
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_COORDINATOR_ADDR: ":9000"
ARC_CLUSTER_REPLICATION_ENABLED: "true"
ARC_AUTH_ENABLED: "true"
ports:
- "8000:8000"
# Reader node 1
arc-reader-1:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: minio
ARC_STORAGE_S3_BUCKET: arc-data
ARC_STORAGE_S3_ENDPOINT: minio:9000
ARC_STORAGE_S3_ACCESS_KEY: minioadmin
ARC_STORAGE_S3_SECRET_KEY: minioadmin123
ARC_STORAGE_S3_USE_SSL: "false"
ARC_STORAGE_S3_PATH_STYLE: "true"
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: reader-01
ARC_CLUSTER_ROLE: reader
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_SEEDS: arc-writer:9000
ARC_AUTH_ENABLED: "true"
ports:
- "8001:8000"
# Reader node 2
arc-reader-2:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: minio
ARC_STORAGE_S3_BUCKET: arc-data
ARC_STORAGE_S3_ENDPOINT: minio:9000
ARC_STORAGE_S3_ACCESS_KEY: minioadmin
ARC_STORAGE_S3_SECRET_KEY: minioadmin123
ARC_STORAGE_S3_USE_SSL: "false"
ARC_STORAGE_S3_PATH_STYLE: "true"
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: reader-02
ARC_CLUSTER_ROLE: reader
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_SEEDS: arc-writer:9000
ARC_AUTH_ENABLED: "true"
ports:
- "8002:8000"
```
## High availability [#high-availability]
Arc Enterprise's HA model depends on which [deployment pattern](/arc-enterprise/configuration/deployment-patterns/) you chose. Both deliver "writer crash recovered without operator intervention," but the mechanics are very different.
### Pattern 2 — shared object storage (multi-writer) [#pattern-2--shared-object-storage-multi-writer]
When all nodes share an object-storage backend (S3, Azure Blob, MinIO), Arc Enterprise runs in **multi-writer** mode: N writer nodes accept writes concurrently behind a load balancer. There is no "primary writer" to fail over to — every writer is a peer, and the load balancer handles writer-crash recovery via its own health-check.
**Key characteristics:**
* **Recovery time**: immediate — the next request lands on a surviving writer via the LB
* **Health-based detection**: load balancer polls each writer's `/ready` endpoint (Traefik, nginx, HAProxy, and cloud ALBs all support this out of the box)
* **No "promotion" step**: all writers are equivalent for ingestion; nothing in the cluster has to elect a new "primary"
* **Singleton background tasks** (retention, continuous queries, deletes) run on whichever node holds the cluster Raft leadership at the time. Raft re-election on leader death is sub-second and the new leader's next scheduler tick picks up the work.
**Enable** by setting `cluster.shared_storage_mode = true` (env: `ARC_CLUSTER_SHARED_STORAGE_MODE`). The Helm chart sets this automatically when `storage.mode=shared`. Requires an Enterprise license that includes the `shared_storage_multi_writer` feature.
**Deploy 3 writers** for full HA: 1 failure is tolerated on both the ingestion side (LB routes around it) and the cluster-Raft side (quorum-of-2 still elects a leader for singleton tasks). 2 writers is not recommended — ingestion stays HA via the LB but Raft cannot elect on a single failure, so singleton tasks pause until quorum is restored. 1 writer is fine for development but has no HA.
A ClusterIP `Service` balances per **TCP connection**, not per HTTP request.
Well-behaved clients (`tsm2arc`, Telegraf, most SDKs) reuse keep-alive
connections, so a heavy writer — a bulk migration, a fat Telegraf aggregate —
rides a handful of long-lived connections that stay pinned to whichever pods
they first dialed. The symptom is one writer near its limits while the others
sit idle, even though "there are 3 writers behind the Service."
Put an **L7 load balancer** in front of the writers — an ingress controller
(nginx, Traefik), Envoy/HAProxy, or a cloud ALB — and send all write clients
through it. Each request is then balanced independently and write traffic
spreads evenly across writers. Use the plain `Service` only as the backend the
L7 layer targets, never as the client-facing endpoint of a multi-writer
cluster.
When a writer crashes:
1. The writer's `/ready` endpoint stops responding (or returns 503).
2. The load balancer marks the writer unhealthy and stops routing to it within one poll cycle (\~5–10 s).
3. New writes route to the surviving writer(s). **No in-cluster failover action required.**
4. In-flight buffer on the crashed writer (records that arrived in memory but had not yet been flushed to S3) is lost. Records that completed the S3 PUT before the crash are durable.
5. On writer restart, the local WAL replays any un-flushed entries into the new Arrow buffer before `/ready` flips back to 200 and the load balancer resumes routing.
### Pattern 1 — local storage with peer replication [#pattern-1--local-storage-with-peer-replication]
When each node has its own local storage, Arc Enterprise runs in **single-writer + multi-reader** mode: one writer takes all ingest, the readers replicate the WAL in real time, and on writer failure one of the readers is promoted via Arc's in-cluster failover controller.
**Key characteristics:**
* **Recovery time**: less than 30 seconds
* **Health-based detection**: continuous health monitoring with configurable thresholds
* **Automatic promotion**: a reader (acting as standby) is promoted to writer via Raft consensus (`CommandPromoteWriter` FSM apply)
* **Cooldown protection**: prevents rapid failover flapping
**Enable** by setting `cluster.failover.enabled = true` (env: `ARC_CLUSTER_FAILOVER_ENABLED`). Requires the `writer_failover` license feature.
**Deploy 1 writer + 2+ readers** with `cluster.replication_enabled=true` on the readers. The readers are the failover pool — each receives a real-time copy of the writer's WAL and can be promoted on writer failure.
When the writer fails:
1. Health checks detect the failure.
2. The Raft leader selects the most caught-up reader (by replication LSN).
3. The selected reader is promoted to writer via `CommandPromoteWriter` Raft apply.
4. Write traffic re-routes to the new writer (clients reconnect or the LB picks up the new writer's `/ready=200`).
- **Cloud-native deployments** (EKS/GKE/AKS, anywhere managed S3 is available) → **Pattern 2 multi-writer**. Simpler operationally, scales writes horizontally, the LB does failover.
- **Bare metal, on-prem, edge** without easy access to S3-compatible storage → **Pattern 1 with writer failover**. Single-writer ceiling on throughput; HA via promotion.
See [Deployment Patterns](/arc-enterprise/configuration/deployment-patterns/) for the full trade-off comparison.
## API reference [#api-reference]
All cluster endpoints require admin authentication.
### Get cluster status [#get-cluster-status]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/cluster
```
**Response:**
```json
{
"success": true,
"data": {
"cluster_name": "production",
"node_count": 3,
"healthy_nodes": 3,
"roles": {
"writer": 1,
"reader": 2,
"compactor": 0
}
}
}
```
### List cluster nodes [#list-cluster-nodes]
```bash
# All nodes
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/cluster/nodes
# Filter by role
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/cluster/nodes?role=reader"
# Filter by state
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/cluster/nodes?state=healthy"
```
**Response:**
```json
{
"success": true,
"data": [
{
"id": "writer-01",
"role": "writer",
"state": "healthy",
"address": "10.0.1.10:9000",
"last_heartbeat": "2026-02-13T10:30:00Z"
},
{
"id": "reader-01",
"role": "reader",
"state": "healthy",
"address": "10.0.1.11:9000",
"last_heartbeat": "2026-02-13T10:30:01Z"
}
]
}
```
### Get specific node [#get-specific-node]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/cluster/nodes/writer-01
```
### Get local node info [#get-local-node-info]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/cluster/local
```
### Cluster health check [#cluster-health-check]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/cluster/health
```
**Response:**
```json
{
"success": true,
"data": {
"status": "healthy",
"node_id": "writer-01",
"role": "writer",
"cluster_name": "production"
}
}
```
## Best practices [#best-practices]
1. **Pick a deployment pattern** — Use [shared object storage](/arc-enterprise/configuration/deployment-patterns/) (S3, MinIO, Azure) for cloud-native deployments, or [local storage with peer replication](/arc-enterprise/configuration/deployment-patterns/) for bare metal, VMs, and edge. Don't mix the two in the same cluster.
2. **Size for HA based on pattern**:
* **Pattern 2 (shared storage)**: 3 writers behind a load balancer — tolerates 1 failure on both the ingestion path (LB routes around it) and the Raft singleton-task path (quorum still elects a leader). Single writer is fine for dev; 2 writers is not recommended (Raft quorum gap).
* **Pattern 1 (local storage)**: 1 writer + 2 readers with `cluster.replication_enabled=true` on the readers. The readers are the failover pool — one is promoted to writer if the primary fails. Don't run 2+ writers in Pattern 1; the current single-writer model is the supported topology.
3. **Scale readers independently** — Add reader nodes to handle increased query load without affecting write performance.
4. **Use one dedicated compactor** — Run compaction on a single dedicated node to avoid duplicate outputs. Enable `ARC_CLUSTER_FAILOVER_ENABLED=true` for automatic compactor failover.
5. **Configure seed nodes** — Reader and compactor nodes should list writer nodes as seeds for cluster discovery.
6. **Always set a shared secret** — `ARC_CLUSTER_SHARED_SECRET` is required for peer authentication. Arc refuses to start replication without it.
7. **Monitor cluster health** — Use the `/api/v1/cluster/health` endpoint with your monitoring system (Prometheus, Grafana) to detect issues early.
## Next steps [#next-steps]
* [RBAC](/arc-enterprise/security/rbac/) — Secure your cluster with role-based access control
* [Tiered Storage](/arc-enterprise/data-lifecycle/tiered-storage/) — Optimize storage costs with hot/cold tiering
* [Audit Logging](/arc-enterprise/security/audit-logging/) — Track all operations for compliance
# Deployment Patterns (/arc-enterprise/configuration/deployment-patterns)
Arc Enterprise supports two clustering topologies, each optimized for a different operational environment. The choice is about **where the Parquet files live** — and that decision shapes durability, cost, and the operational model of your cluster.
## The two patterns [#the-two-patterns]
### Pattern A: Shared object storage [#pattern-a-shared-object-storage]
All nodes read and write to the **same object store** — S3, MinIO, or Azure Blob. The bucket is the source of truth for Parquet files. Nodes are stateless from a data perspective: any reader can serve any query because every file is one API call away.
**Best for:**
* Cloud deployments (AWS, GCP, Azure)
* Teams that already operate object storage
* Workloads where scaling readers elastically matters more than query latency
* Kubernetes-native deployments with object storage
### Pattern B: Local storage with peer replication [#pattern-b-local-storage-with-peer-replication]
Each node has its own **local disks** (NVMe, SSD, or attached block storage). Parquet files are replicated peer-to-peer over the cluster protocol, verified via SHA-256, and kept on every node that needs them. A Raft-backed file manifest is the cluster-wide source of truth for which files exist.
**Best for:**
* Bare metal and virtual machine deployments
* Edge, on-premises, and air-gapped environments
* Defense, aerospace, industrial, and regulated workloads where shared object storage is not available
* Deployments that need the lowest possible query latency (local NVMe beats network-attached storage every time)
## Side-by-side comparison [#side-by-side-comparison]
| Aspect | Shared Object Storage | Local Storage + Peer Replication |
| ------------------------ | -------------------------------------- | ---------------------------------------------------- |
| **Storage layout** | Single bucket, all nodes read/write | Per-node local disks, replicated peer-to-peer |
| **Source of truth** | The bucket itself | Raft-backed file manifest (FSM) |
| **Durability** | Relies on S3/MinIO/Azure replication | Replicated across N cluster nodes |
| **Query latency** | Network fetch from object store | Local disk I/O |
| **New-node bootstrap** | Instant (no data transfer needed) | Startup catch-up pulls bytes from peers |
| **Compactor outputs** | Written once to bucket, visible to all | Compactor writes locally, Raft announces, peers pull |
| **Compactor failover** | Any healthy node can take over | Any healthy node can take over |
| **Best deployment** | Kubernetes, cloud-native | Bare metal, VMs, edge |
| **Cost model** | Object storage API calls + egress | Local disk capacity × nodes |
| **Network requirements** | Reliable path to object store | Reliable path between cluster nodes |
## Choosing a pattern [#choosing-a-pattern]
Start here:
1. **Do you already run S3/MinIO/Azure in production?** → Pattern A (shared).
2. **Do your nodes have fast local disks and you want minimum query latency?** → Pattern B (local).
3. **Is shared object storage unavailable (edge, air-gap, defense)?** → Pattern B (local).
4. **Do you expect to scale readers elastically based on demand?** → Pattern A (shared).
5. **Do you need a single-digit-ms query path?** → Pattern B (local).
You can also mix — a cluster can use shared object storage for cold data (tiered storage to S3 Glacier) while keeping hot data on local disks. See [Tiered Storage](/arc-enterprise/data-lifecycle/tiered-storage/).
## Pattern A — shared storage setup [#pattern-a--shared-storage-setup]
### Minimal 3-node cluster (1 writer, 1 reader, 1 compactor) on MinIO [#minimal-3-node-cluster-1-writer-1-reader-1-compactor-on-minio]
```yaml
# docker-compose.yml
services:
minio:
image: minio/minio
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin123
ports: ["9001:9001"]
arc-writer:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: minio
ARC_STORAGE_S3_BUCKET: arc-data
ARC_STORAGE_S3_ENDPOINT: minio:9000
ARC_STORAGE_S3_ACCESS_KEY: minioadmin
ARC_STORAGE_S3_SECRET_KEY: minioadmin123
ARC_STORAGE_S3_USE_SSL: "false"
ARC_STORAGE_S3_PATH_STYLE: "true"
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: writer-01
ARC_CLUSTER_ROLE: writer
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_RAFT_BOOTSTRAP: "true"
ARC_CLUSTER_SHARED_SECRET: "your-cluster-secret"
ARC_CLUSTER_REPLICATION_ENABLED: "false" # not needed on shared storage
ports: ["8001:8000"]
arc-reader:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: minio
ARC_STORAGE_S3_BUCKET: arc-data
ARC_STORAGE_S3_ENDPOINT: minio:9000
ARC_STORAGE_S3_ACCESS_KEY: minioadmin
ARC_STORAGE_S3_SECRET_KEY: minioadmin123
ARC_STORAGE_S3_USE_SSL: "false"
ARC_STORAGE_S3_PATH_STYLE: "true"
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: reader-01
ARC_CLUSTER_ROLE: reader
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_SEEDS: arc-writer:9200
ARC_CLUSTER_SHARED_SECRET: "your-cluster-secret"
ports: ["8002:8000"]
arc-compactor:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: minio
ARC_STORAGE_S3_BUCKET: arc-data
ARC_STORAGE_S3_ENDPOINT: minio:9000
ARC_STORAGE_S3_ACCESS_KEY: minioadmin
ARC_STORAGE_S3_SECRET_KEY: minioadmin123
ARC_STORAGE_S3_USE_SSL: "false"
ARC_STORAGE_S3_PATH_STYLE: "true"
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: compactor-01
ARC_CLUSTER_ROLE: compactor
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_SEEDS: arc-writer:9200
ARC_CLUSTER_SHARED_SECRET: "your-cluster-secret"
ARC_CLUSTER_FAILOVER_ENABLED: "true"
ARC_COMPACTION_ENABLED: "true"
ports: ["8003:8000"]
```
### Key points [#key-points]
* **All nodes point to the same bucket.** The writer flushes to the bucket; readers query directly from it; the compactor reads source files, writes compacted outputs back, and deletes the sources.
* **`ARC_CLUSTER_REPLICATION_ENABLED=false`** is the right choice on shared storage — there's no peer-to-peer file transfer needed because the bucket is already shared.
* **Exactly one compactor node.** Multiple compactors against a shared bucket produce duplicate outputs. Arc warns you via the cluster health check if it sees more than one.
* **Compactor failover** (`ARC_CLUSTER_FAILOVER_ENABLED=true`) lets the Raft leader automatically reassign the compactor lease to another healthy node if the current compactor dies. No restart required.
## Pattern B — local storage setup [#pattern-b--local-storage-setup]
### Minimal 3-node cluster (1 writer, 1 reader, 1 compactor) on local disks [#minimal-3-node-cluster-1-writer-1-reader-1-compactor-on-local-disks]
```yaml
# docker-compose.yml
services:
arc-writer:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: local
ARC_STORAGE_LOCAL_PATH: /app/data
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: writer-01
ARC_CLUSTER_ROLE: writer
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_RAFT_BOOTSTRAP: "true"
ARC_CLUSTER_SHARED_SECRET: "your-cluster-secret"
ARC_CLUSTER_REPLICATION_ENABLED: "true" # CRITICAL for local storage
volumes:
- writer-data:/app/data
ports: ["8001:8000"]
arc-reader:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: local
ARC_STORAGE_LOCAL_PATH: /app/data
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: reader-01
ARC_CLUSTER_ROLE: reader
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_SEEDS: arc-writer:9200
ARC_CLUSTER_SHARED_SECRET: "your-cluster-secret"
ARC_CLUSTER_REPLICATION_ENABLED: "true"
volumes:
- reader-data:/app/data
ports: ["8002:8000"]
arc-compactor:
image: basekick/arc:latest
environment:
ARC_LICENSE_KEY: "ARC-XXXX-XXXX-XXXX-XXXX"
ARC_STORAGE_BACKEND: local
ARC_STORAGE_LOCAL_PATH: /app/data
ARC_CLUSTER_ENABLED: "true"
ARC_CLUSTER_NODE_ID: compactor-01
ARC_CLUSTER_ROLE: compactor
ARC_CLUSTER_CLUSTER_NAME: production
ARC_CLUSTER_SEEDS: arc-writer:9200
ARC_CLUSTER_SHARED_SECRET: "your-cluster-secret"
ARC_CLUSTER_REPLICATION_ENABLED: "true"
ARC_CLUSTER_FAILOVER_ENABLED: "true"
ARC_COMPACTION_ENABLED: "true"
volumes:
- compactor-data:/app/data
ports: ["8003:8000"]
volumes:
writer-data:
reader-data:
compactor-data:
```
### How peer replication works [#how-peer-replication-works]
1. **Writer flushes a Parquet file locally.** The file hash (SHA-256) is computed and included in the flush.
2. **The writer registers the file in the Raft manifest** via a `CommandRegisterFile` entry. This commits cluster-wide — every node now knows the file exists and where to find it.
3. **Readers and compactors observe the FSM callback.** A background puller enqueues a byte-level pull from the origin peer (or any healthy peer that has a copy).
4. **The puller fetches over the cluster protocol**, streams bytes, verifies the SHA-256 against the manifest, and writes to local storage. Checksum mismatches trigger retries; failed pulls fall back to other peers.
5. **On node restart**, a startup catch-up walker reconciles the local manifest against the Raft FSM and pulls any files the node missed.
### Key points [#key-points-1]
* **`ARC_CLUSTER_REPLICATION_ENABLED=true`** is required — this enables the file manifest and peer puller.
* **Each node has its own volume.** No shared volume, no NFS, no clustered filesystem — the replication is the primary data-plane mechanism.
* **Shared secret is mandatory.** Peer fetch requests are HMAC-authenticated with the shared secret; Arc refuses to start replication without one.
* **Raft leader is the writer by default.** `ARC_CLUSTER_RAFT_BOOTSTRAP=true` on the writer makes it bootstrap Raft; other nodes join via the seed. Non-leader nodes forward manifest commands to the leader transparently.
### Compacted file distribution [#compacted-file-distribution]
Compaction on local storage works the same way as ingest:
1. The compactor reads source Parquet files (from local storage, pulling from peers if missing).
2. It produces a compacted output, writes it to its own local disk.
3. It registers the new file in the Raft manifest and marks the source files as deleted.
4. Every other node sees the manifest change: readers pull the compacted bytes from the compactor, and delete their local copies of the source files.
## Security notes [#security-notes]
Both patterns share the same security posture:
* **Shared secret authentication** (`cluster.shared_secret`) — required for peer discovery and, in Pattern B, for all peer file fetches. Arc refuses to boot if replication is enabled without a shared secret.
* **TLS encryption** (`cluster.tls_enabled`) — optional but recommended. Encrypts the inter-node coordinator protocol, Raft transport, and peer file transfers.
* **Role-based authorization on manifest mutations** — only nodes with `CanIngest` (writers) or `CanCompact` (compactors) can forward `RegisterFile` / `DeleteFile` commands to the leader. Reader nodes are rejected.
See [Cluster Security](/arc-enterprise/security/cluster-security/) for full details.
## Common mistakes [#common-mistakes]
* **Multiple compactor nodes on shared storage.** This produces duplicate compacted outputs and double-counted query results. Use exactly one `ARC_CLUSTER_ROLE=compactor` and enable `ARC_CLUSTER_FAILOVER_ENABLED=true` for automatic failover.
* **Mixing shared and local storage in the same cluster.** All nodes must agree on the storage model. Pick one per cluster.
* **Forgetting `ARC_CLUSTER_REPLICATION_ENABLED=true` on local storage.** Without it, readers will query empty local directories.
* **Using a shared volume (NFS, EFS) as "local" storage.** Don't — the concurrent-write semantics of a shared POSIX filesystem aren't what Arc expects, and you lose the durability guarantees of either pattern. Either go full shared object storage or full per-node local disks.
## Next steps [#next-steps]
* [Clustering Configuration Reference](/arc-enterprise/configuration/clustering/) — full list of cluster config options
* [Tiered Storage](/arc-enterprise/data-lifecycle/tiered-storage/) — combine local hot storage with cold object storage
* [Cluster Security](/arc-enterprise/security/cluster-security/) — TLS and shared secret configuration
# Edge Sync with a Clustered Hub (/arc-enterprise/configuration/edge-sync-clustered)
[Edge sync](/arc/advanced/edge-sync/) ships Parquet files from an edge Arc (the *spoke*) to a central Arc (the *hub*). The spoke is normally a single node. The hub, in an Enterprise deployment, is usually not — and the two transports touch a cluster at different points.
This page covers what changes when the hub is a cluster. Everything in the [OSS edge sync guide](/arc/advanced/edge-sync/) still applies.
## The short version [#the-short-version]
| | Network transport | Air-gap transport |
| --------------------- | ------------------------------------------ | ----------------------------------------------------------------------- |
| Which node handles it | **Any node** | **One designated node** |
| Why | Manifest writes forward to the Raft leader | The received-files index and dedup ledger are node-local |
| If you get it wrong | Nothing — it just works | Files are still correct, but replay protection and history are per-node |
## Network transport: Any node works [#network-transport-any-node-works]
A spoke has a single `hub_url`, and every request goes to whatever that resolves to. There is **no writer-role gate** on the receive path — unlike retention or continuous queries, which check `IsPrimaryWriter()` on every tick, a node receiving a file simply accepts it.
That is safe because the manifest write forwards:
```text
receiving node is the Raft leader → applies locally
receiving node is a follower → forwards the command to the current leader
```
So a spoke can push at any node and the file still lands in the shared Raft manifest, becoming queryable from your readers.
### Which node should a spoke point at? [#which-node-should-a-spoke-point-at]
Any of them. In practice:
* **A writer** is the natural choice — it already holds storage credentials and is sized for write throughput.
* **A load balancer across the writers** works and survives a node going down. The spoke retries on the next pass regardless, and re-delivery of an already-received file is a no-op, so a mid-transfer failover costs one repeated transfer at worst.
* **A reader** works too, though it means a follower forwarding every manifest write to the leader — an extra hop for no benefit.
The **compactor is invisible** to edge sync. Synced files land as ordinary Parquet under `{spoke_id}/{database}/{measurement}/...` and are compacted like anything else.
### Cluster TLS [#cluster-tls]
If `cluster.tls_enabled` is set, that governs Raft RPC and peer fetch — **not** the spoke's connection. A spoke connects over the public API listener, so its transport security keys off `server.tls_enabled`. These are independent flags; see [Cluster Security](/arc-enterprise/security/cluster-security/).
## Air-gap transport: Designate one import node [#air-gap-transport-designate-one-import-node]
This is the part that needs an operational decision.
Two pieces of edge-sync state live in **node-local SQLite** (the shared metadata database), not in the Raft manifest:
* **`sync_received`** — the index reconcile answers from, so it can respond without re-hashing storage.
* **`sync_imported_bundles`** — the air-gap dedup ledger, keyed `(spoke_id, bundle_id)`.
Neither is replicated. So on a multi-node hub:
| Scenario | Outcome |
| -------------------------------------- | ----------------------------------------------------------------------------------------- |
| Every drive imported on node A | Correct. Dedup works, import history is complete |
| Bundle 1 on node A, bundle 2 on node B | Both import correctly — **the files are fine** — but each node only knows its own imports |
| Bundle 1 re-imported on B after A | **Not refused.** B has no dedup row for it |
That last row is worth being precise about: the re-import is **safe, not harmful**. Every file resolves to `already_present`, the [§6.1 identity rule](/arc/advanced/edge-sync/) still refuses to overwrite differing content, and nothing is corrupted. What you lose is the refusal — and `GET /api/v1/bundle-import/history/{spoke_id}` on node B will not show what node A imported.
Designate **one** node as the drive-import node — a writer, since it already has storage credentials — and put it in your runbook. Nothing in Arc enforces this today, so it is an operational convention rather than a configured one.
### Raft proposal batching [#raft-proposal-batching]
Import registers manifest entries in **batches of 1000 operations per Raft proposal**.
This matters more than it might sound. The network path is naturally rate-limited to one proposal per HTTP request; an import is a tight loop over a whole drive. Without batching, a 10,000-file bundle would fire 10,000 individual proposals at your leader as fast as the disk allows. Batching makes that 10 proposals.
If a batch fails — a quorum loss, say — the import **aborts** rather than continuing. Files already committed to storage stay there, but the bundle is not recorded as imported, so re-importing it re-registers everything. That is the intended recovery.
### One import at a time [#one-import-at-a-time]
Imports are serialized per node by a mutex. A second concurrent import returns `409` with `"reason": "import in progress"` rather than queuing behind the first — an import can legitimately run for hours, and a request hanging that long is indistinguishable from a dead hub.
## Failover [#failover]
Nothing in edge sync pins to a particular writer, so a writer failover needs no spoke-side change:
* **Network** — if the spoke's `hub_url` points at a load balancer, the next pass simply lands on a healthy node. If it points at a specific node that goes down, the pass fails and retries on the next trigger; the ledger keeps its state, so nothing is lost or double-sent.
* **Air gap** — if your designated import node is down, import on another node. You lose dedup and history *for that bundle*, per the table above; the data itself is correct.
## What is not cluster-aware yet [#what-is-not-cluster-aware-yet]
Stated plainly so it is not a surprise:
* **The received-files index and dedup ledger are node-local.** Replicating them would make any node a valid import target and give fleet-wide dedup. It is a candidate for a later release, not a defect — the current behaviour is safe, just node-scoped.
* **There is no automatic reconciliation between the manifest and storage.** Orphaned manifest entries are logged, not self-healed.
* **The scheduled sync agent is not in this release.** Passes are triggered manually; a cron job or a link-up hook on the spoke is the current answer.
# Configuration (/arc-enterprise/configuration)
Arc Enterprise is configured through `arc.toml`, with environment variables overriding any key. The Enterprise-only sections are `[license]`, `[cluster]`, `[tiered_storage]`, `[audit_log]`, `[governance]`, and `[query_management]`.
Read [Deployment patterns](/arc-enterprise/configuration/deployment-patterns/) before anything else. The choice between shared object storage and local disks with peer replication determines most of the other settings on this page, and it is expensive to reverse once a cluster holds data.
# Configuration Overview (/arc-enterprise/configuration/overview)
Arc uses a TOML configuration file (`arc.toml`) with environment variable overrides for flexibility.
## Enterprise license [#enterprise-license]
Configure your Arc Enterprise license key to enable enterprise features:
```toml
[license]
key = "ARC-XXXX-XXXX-XXXX-XXXX"
```
Environment variable:
```bash
ARC_LICENSE_KEY="ARC-XXXX-XXXX-XXXX-XXXX"
```
On startup, Arc validates your license and enables the features included in your plan. See [Arc Enterprise Overview](/arc-enterprise/) for the full feature list.
As of **26.09.1**, a transient license-server failure no longer crash-loops Enterprise pods: startup retries briefly and then falls back to the last **signature-verified** license cached on disk, honored until that license's own expiry. A definitive server rejection (revoked, expired, unknown key) still disables enterprise features immediately.
### Air-gapped: Offline license file (26.09.1+) [#air-gapped-offline-license-file-26091]
For environments with no route to `enterprise.basekick.net`, download an offline license file from the activation server admin (an explicit **site license**: unbound, valid on any machine until expiry, audit-logged at mint) and point Arc at it:
```toml
[license]
file_path = "/etc/arc/license.json" # or ARC_LICENSE_FILE_PATH
```
The file is verified from disk against Arc's pinned public key — no network calls of any kind. `file_path` wins over `key`; a rejected file means OSS mode (never a silent fallback to online licensing). Keep the file readable only by the Arc user (`0600`).
## Configuration files [#configuration-files]
### Primary: arc.toml [#primary-arctoml]
The main configuration file with production-ready defaults:
```toml
# Server Configuration
[server]
port = 8000
# Logging
[log]
level = "info" # debug, info, warn, error
format = "console" # json or console
# Database (query engine)
[database]
# Auto-detected if not set (recommended)
# max_connections = 28 # 2x CPU cores
# memory_limit = "8GB" # ~50% system RAM
# thread_count = 14 # CPU cores
enable_wal = false
# Storage Backend
[storage]
backend = "local" # local, s3, minio, azure, azblob
local_path = "./data/arc"
# Ingestion
[ingest]
max_buffer_size = 50000 # records before flush
max_buffer_age_ms = 5000 # ms before force flush
# flush_workers = 16 # async flush workers (auto-detected)
# flush_queue_size = 64 # pending flush queue (auto-detected)
# shard_count = 32 # buffer shards
# Compaction
[compaction]
enabled = true
hourly_enabled = true
hourly_min_age_hours = 0
hourly_min_files = 5
# Authentication
[auth]
enabled = true
# Delete Operations
[delete]
enabled = true
confirmation_threshold = 10000
max_rows_per_delete = 1000000
# Retention Policies
[retention]
enabled = true
# Continuous Queries
[continuous_query]
enabled = true
```
### Environment variables [#environment-variables]
Override any setting via environment variables with the `ARC_` prefix:
```bash
# Server
ARC_SERVER_PORT=8000
ARC_SERVER_TLS_ENABLED=false
ARC_SERVER_TLS_CERT_FILE=/path/to/cert.pem
ARC_SERVER_TLS_KEY_FILE=/path/to/key.pem
ARC_SERVER_MAX_PAYLOAD_SIZE=1GB
# Logging
ARC_LOG_LEVEL=info
ARC_LOG_FORMAT=json
# Database
ARC_DATABASE_MAX_CONNECTIONS=28
ARC_DATABASE_MEMORY_LIMIT=8GB
ARC_DATABASE_THREAD_COUNT=14
# Features
ARC_AUTH_ENABLED=true
ARC_COMPACTION_ENABLED=true
ARC_DELETE_ENABLED=true
ARC_RETENTION_ENABLED=true
ARC_CONTINUOUS_QUERY_ENABLED=true
# Ingestion Concurrency
ARC_INGEST_FLUSH_WORKERS=32
ARC_INGEST_FLUSH_QUEUE_SIZE=200
ARC_INGEST_SHARD_COUNT=64
```
## Configuration priority [#configuration-priority]
Settings are applied in this order (highest to lowest):
1. **Environment variables** (e.g., `ARC_SERVER_PORT=8000`)
2. **arc.toml file**
3. **Built-in defaults**
## Storage backends [#storage-backends]
**Local Filesystem** - Default, simplest option for single-node deployments.
```toml
[storage]
backend = "local"
local_path = "./data/arc"
```
Environment variables:
```bash
ARC_STORAGE_BACKEND=local
ARC_STORAGE_LOCAL_PATH=./data/arc
```
**AWS S3** - Recommended for production cloud deployments.
```toml
[storage]
backend = "s3"
s3_bucket = "arc-production"
s3_region = "us-east-1"
# Credentials via env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
# Or use IAM roles (recommended)
```
Environment variables:
```bash
ARC_STORAGE_BACKEND=s3
ARC_STORAGE_S3_BUCKET=arc-data
ARC_STORAGE_S3_REGION=us-east-1
ARC_STORAGE_S3_ENDPOINT=s3.amazonaws.com
ARC_STORAGE_S3_ACCESS_KEY=your_key
ARC_STORAGE_S3_SECRET_KEY=your_secret
ARC_STORAGE_S3_USE_SSL=true
ARC_STORAGE_S3_PATH_STYLE=false
```
For EC2/EKS deployments, use IAM roles instead of access keys. Arc automatically uses instance credentials.
**MinIO** - Self-hosted S3-compatible storage.
```toml
[storage]
backend = "minio"
s3_bucket = "arc"
s3_endpoint = "minio:9000"
s3_access_key = "minioadmin"
s3_secret_key = "minioadmin123"
s3_use_ssl = false
s3_path_style = true # Required for MinIO
```
Environment variables:
```bash
ARC_STORAGE_BACKEND=minio
ARC_STORAGE_S3_ENDPOINT=minio:9000
ARC_STORAGE_S3_BUCKET=arc
ARC_STORAGE_S3_ACCESS_KEY=minioadmin
ARC_STORAGE_S3_SECRET_KEY=minioadmin123
ARC_STORAGE_S3_USE_SSL=false
ARC_STORAGE_S3_PATH_STYLE=true
```
**Azure Blob Storage** - For Azure cloud deployments.
```toml
[storage]
backend = "azure" # or "azblob"
azure_container = "arc-data"
azure_account_name = "your_account"
azure_account_key = "your_key"
# Or use managed identity:
# azure_use_managed_identity = true
```
Environment variables:
```bash
ARC_STORAGE_BACKEND=azure
ARC_STORAGE_AZURE_CONTAINER=arc-data
ARC_STORAGE_AZURE_ACCOUNT_NAME=your_account
ARC_STORAGE_AZURE_ACCOUNT_KEY=your_key
```
For Azure VMs/AKS, use managed identity for keyless authentication:
```toml
azure_use_managed_identity = true
```
## Key configuration areas [#key-configuration-areas]
### Server [#server]
Basic HTTP server settings:
```toml
[server]
port = 8000 # HTTP/HTTPS port to listen on
```
### TLS/SSL (HTTPS) [#tlsssl-https]
Arc supports native HTTPS/TLS without requiring a reverse proxy:
```toml
[server]
port = 443
tls_enabled = true
tls_cert_file = "/etc/letsencrypt/live/example.com/fullchain.pem"
tls_key_file = "/etc/letsencrypt/live/example.com/privkey.pem"
```
Environment variables:
```bash
ARC_SERVER_TLS_ENABLED=true
ARC_SERVER_TLS_CERT_FILE=/path/to/cert.pem
ARC_SERVER_TLS_KEY_FILE=/path/to/key.pem
```
* **Native packages** (deb/rpm): Use native TLS for simple deployments
* **Docker/Kubernetes**: Use a reverse proxy (Traefik, nginx, Ingress) for TLS termination
* **Development**: Use self-signed certificates for local HTTPS testing
When TLS is enabled, Arc automatically:
* Adds the `Strict-Transport-Security` (HSTS) header
* Validates certificate and key files on startup
### Max payload size [#max-payload-size]
Configure the maximum request payload size for write endpoints (msgpack, line protocol):
```toml
[server]
# Maximum payload size (applies to both compressed and decompressed)
# Supports units: B, KB, MB, GB
# Default: 1GB
max_payload_size = "1GB"
```
Environment variable:
```bash
ARC_SERVER_MAX_PAYLOAD_SIZE=2GB
```
If you're importing large datasets and encounter 413 errors, you can:
1. Increase `max_payload_size` (e.g., `"2GB"`)
2. Batch your imports into smaller chunks (recommended for reliability)
### Database (query engine) [#database-query-engine]
Query engine connection pool and resource settings:
```toml
[database]
# AUTO-DETECTION: If not set, Arc automatically configures:
# - max_connections: 2x CPU cores (min 4, max 64)
# - memory_limit: ~50% of system memory
# - thread_count: Number of CPU cores
# Manual override examples:
max_connections = 28 # Connection pool size
memory_limit = "8GB" # Query engine memory limit
thread_count = 14 # Query execution threads
enable_wal = false # Query engine WAL (not Arc WAL)
```
### Ingestion [#ingestion]
Buffer and concurrency settings for write performance:
```toml
[ingest]
# Maximum records to buffer before flushing to Parquet
max_buffer_size = 50000
# Maximum age (ms) before forcing a flush
max_buffer_age_ms = 5000
# Concurrency settings (auto-detected if not set)
# flush_workers = 16 # async flush workers (2x CPU cores, min 8, max 64)
# flush_queue_size = 64 # pending flush queue (4x workers, min 100)
# shard_count = 32 # buffer shards for lock distribution
```
Data flushes when **either** condition is met:
1. Buffer reaches `max_buffer_size` records
2. Buffer age exceeds `max_buffer_age_ms`
For deployments with many concurrent clients (50+), increase `flush_workers` and `flush_queue_size`:
```toml
[ingest]
flush_workers = 32
flush_queue_size = 200
shard_count = 64
```
### Compaction [#compaction]
Automatic file optimization:
```toml
[compaction]
enabled = true
hourly_enabled = true
hourly_min_age_hours = 0 # Files must be this old
hourly_min_files = 5 # Minimum files to trigger
daily_enabled = false # Daily tier (optional)
daily_min_age_hours = 24
daily_min_files = 3
```
### Authentication [#authentication]
Token-based API authentication:
```toml
[auth]
enabled = true # Enable/disable auth
db_path = "./data/arc_auth.db" # Token database
cache_ttl = 30 # Token cache TTL (seconds)
max_cache_size = 1000 # Max cached tokens
bootstrap_token = "" # Pre-set admin token value (v26.04.1+)
force_bootstrap = false # Add a recovery token without removing existing ones (v26.04.1+)
```
**`bootstrap_token`** — Set a known admin token at deploy time via `ARC_AUTH_BOOTSTRAP_TOKEN` instead of catching a randomly generated one from startup logs. On first run, Arc uses this value as the initial admin token. On subsequent restarts, it is a no-op.
**`force_bootstrap`** — Recovery path when the admin token is lost. Set `ARC_AUTH_FORCE_BOOTSTRAP=true` alongside `ARC_AUTH_BOOTSTRAP_TOKEN` to add a new `arc-recovery` admin token **without removing existing tokens**. Remove this flag after recovery. See the [Authentication configuration guide](/arc/configuration/authentication/#bootstrap--recovery) for full details.
### Delete operations [#delete-operations]
Safe deletion with confirmation:
```toml
[delete]
enabled = true
confirmation_threshold = 10000 # Require confirmation above this
max_rows_per_delete = 1000000 # Hard limit per operation
```
### Retention policies [#retention-policies]
Automatic data expiration:
```toml
[retention]
enabled = true
db_path = "./data/arc_retention.db"
```
### Continuous queries [#continuous-queries]
Scheduled automated queries:
```toml
[continuous_query]
enabled = true
db_path = "./data/arc_cq.db"
```
### Write-Ahead Log (WAL) [#write-ahead-log-wal]
Optional durability guarantee:
```toml
[wal]
enabled = false # Enable for zero data loss
directory = "./data/wal"
sync_mode = "fdatasync" # none, fdatasync, fsync
max_size_mb = 500
max_age_seconds = 3600
```
### Metrics [#metrics]
Timeseries metrics collection:
```toml
[metrics]
timeseries_retention_minutes = 60
timeseries_interval_seconds = 10
```
## Quick configuration examples [#quick-configuration-examples]
```toml
[server]
port = 8000
[log]
level = "debug"
format = "console"
[storage]
backend = "local"
local_path = "./dev_data"
[auth]
enabled = false
[compaction]
enabled = false
```
```toml
[server]
port = 8000
[log]
level = "info"
format = "json"
[database]
max_connections = 32
memory_limit = "16GB"
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
[ingest]
max_buffer_size = 100000
max_buffer_age_ms = 10000
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
daily_enabled = true
[wal]
enabled = true
sync_mode = "fdatasync"
```
```toml
[server]
port = 8000
[log]
level = "info"
format = "json"
[storage]
backend = "s3"
s3_bucket = "arc-production"
s3_region = "us-east-1"
# Use IAM roles for credentials
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
```
```toml
[server]
port = 443
tls_enabled = true
tls_cert_file = "/etc/letsencrypt/live/example.com/fullchain.pem"
tls_key_file = "/etc/letsencrypt/live/example.com/privkey.pem"
[log]
level = "info"
format = "json"
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
```
```toml
[server]
port = 8000
[storage]
backend = "minio"
s3_bucket = "arc"
s3_endpoint = "minio:9000"
[wal]
enabled = true
sync_mode = "fdatasync"
directory = "/var/lib/arc/wal"
max_size_mb = 1000
max_age_seconds = 3600
[compaction]
enabled = true
```
```toml
# Optimized for 50+ concurrent clients (e.g., many Telegraf agents)
[server]
port = 8000
[log]
level = "info"
format = "json"
[database]
max_connections = 64
memory_limit = "16GB"
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
[ingest]
max_buffer_size = 100000
max_buffer_age_ms = 10000
# Scale concurrency for many clients
flush_workers = 32 # More workers for parallel I/O
flush_queue_size = 200 # Larger queue for burst handling
shard_count = 64 # More shards to reduce lock contention
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
```
## Best practices [#best-practices]
### 1. Use arc.toml for permanent settings [#1-use-arctoml-for-permanent-settings]
Store configuration in `arc.toml` and version control it (without secrets):
```toml
[storage]
backend = "s3"
s3_bucket = "arc"
s3_region = "us-east-1"
# Credentials via environment variables
```
### 2. Use environment variables for secrets [#2-use-environment-variables-for-secrets]
```bash
export ARC_STORAGE_S3_ACCESS_KEY="your_access_key"
export ARC_STORAGE_S3_SECRET_KEY="your_secret_key"
```
### 3. Let Arc auto-detect resources [#3-let-arc-auto-detect-resources]
Arc automatically detects optimal query engine settings based on your system. Only override if you have specific requirements:
```toml
[database]
# Leave commented for auto-detection
# max_connections = 28
# memory_limit = "8GB"
# thread_count = 14
```
### 4. Enable features progressively [#4-enable-features-progressively]
Start simple, add features as needed:
1. Basic configuration (storage + auth)
2. Compaction (for query optimization)
3. Retention policies (for data management)
4. WAL (for zero data loss guarantee)
### 5. Monitor configuration impact [#5-monitor-configuration-impact]
Check metrics after configuration changes:
```bash
# Memory usage
curl http://localhost:8000/api/v1/metrics/memory
# Query performance
curl http://localhost:8000/api/v1/metrics/query-pool
# Compaction status
curl http://localhost:8000/api/v1/compaction/status
```
## Troubleshooting [#troubleshooting]
### Configuration not loading [#configuration-not-loading]
```bash
# Verify TOML syntax (use any TOML validator)
# Check file exists in expected location
ls -la arc.toml
# Arc looks for arc.toml in:
# 1. Current directory
# 2. /etc/arc/arc.toml (native install)
```
### Environment variables not working [#environment-variables-not-working]
```bash
# Verify they're set
env | grep ARC_
# Use correct prefix and format
export ARC_SERVER_PORT=8000 # Correct
export SERVER_PORT=8000 # Wrong - missing ARC_ prefix
```
### Resource issues [#resource-issues]
```bash
# Check current settings via metrics
curl http://localhost:8000/api/v1/metrics/memory
# Adjust in arc.toml:
[database]
memory_limit = "4GB"
max_connections = 16
```
## Enterprise configuration [#enterprise-configuration]
The following configuration sections are available with an Arc Enterprise license. See each feature's dedicated page for detailed configuration.
### Clustering [#clustering]
See [Clustering & High Availability](/arc-enterprise/configuration/clustering/) for full configuration reference.
```toml
[cluster]
enabled = true
node_id = "writer-01"
role = "writer"
cluster_name = "production"
seeds = ["10.0.1.10:9000"]
```
### Tiered storage [#tiered-storage]
See [Tiered Storage](/arc-enterprise/data-lifecycle/tiered-storage/) for full configuration reference.
```toml
[tiered_storage]
enabled = true
default_hot_max_age_days = 30
[tiered_storage.cold]
enabled = true
backend = "s3"
s3_bucket = "arc-archive"
```
### Audit logging [#audit-logging]
See [Audit Logging](/arc-enterprise/security/audit-logging/) for full configuration reference.
```toml
[audit_log]
enabled = true
retention_days = 90
```
### Query governance [#query-governance]
See [Query Governance](/arc-enterprise/query/query-governance/) for full configuration reference.
```toml
[governance]
enabled = true
default_rate_limit_per_min = 60
```
### Query management [#query-management]
See [Query Management](/arc-enterprise/query/query-management/) for full configuration reference.
```toml
[query_management]
enabled = true
```
## Next steps [#next-steps]
* **[Clustering & High Availability](/arc-enterprise/configuration/clustering/)** - Multi-node cluster configuration
* **[Tiered Storage](/arc-enterprise/data-lifecycle/tiered-storage/)** - Hot/cold storage tiering
* **[Audit Logging](/arc-enterprise/security/audit-logging/)** - Compliance and security logging
* **[Advanced Features](/arc-enterprise/advanced/compaction/)** - Compaction and WAL
# Continuous Queries (/arc-enterprise/data-lifecycle/continuous-queries)
Continuous queries enable automatic downsampling and aggregation of data into materialized views, reducing storage requirements while maintaining queryable historical data.
Arc Enterprise includes automatic CQ execution. See [Automated Scheduling](/arc-enterprise/operations/automated-scheduling/) to configure scheduled execution.
## Overview [#overview]
Continuous queries in Arc help you:
* **Downsample Data**: Aggregate high-frequency data into lower-frequency summaries
* **Reduce Storage**: Store aggregated data instead of raw metrics
* **Maintain History**: Keep long-term trends without full granularity
* **Improve Query Performance**: Query pre-aggregated data for faster results
* **Create Materialized Views**: Automatically maintain aggregated datasets
## How it works [#how-it-works]
Continuous queries use standard analytical SQL to aggregate data from source measurements into destination measurements:
1. **Define Query**: Specify aggregation logic using SQL
2. **Set Schedule**: Configure time intervals for grouping (e.g., hourly, daily)
3. **Execute Manually**: Trigger execution via API with start/end times
4. **Store Results**: Write aggregated data to a new measurement
5. **Apply Retention**: Optionally set custom retention for aggregated data
### Architecture [#architecture]
```text
Source Measurement (cpu)
↓
Continuous Query (AVG, MAX, MIN, etc.)
↓
Destination Measurement (cpu_hourly)
↓
Optional: Retention Policy
```
### Execution model and semantics [#execution-model-and-semantics]
Continuous queries are **recomputed once per interval**, not maintained as an
incremental/streaming aggregate. There is no running state carried between runs.
On each scheduled run, Arc:
1. Computes the time window `[last_processed_time, now)` — the slice since the
previous successful run (on the very first run it defaults to the last hour).
2. Substitutes that range into your query's `{start_time}` / `{end_time}`
placeholders and runs the aggregation **fresh over the full set of source rows
in that window** (read from Parquet).
3. Writes the results to the destination measurement, stamped with the window
start time and tagged so duplicate windows can be deduped (see
[Idempotency and `tag_columns`](#idempotency-and-tag_columns)).
4. Advances `last_processed_time` to the window's end.
Because the watermark advances to the end of each processed window, **windows are
tumbling and non-overlapping** — the next run starts where the previous one
ended; earlier windows are not revisited.
- **Late / out-of-order data is not reprocessed.** A window is computed once and
the watermark then moves past it. Rows that arrive *after* their window has
already been processed (late events, corrections, backfills) are **not folded
back** into that window's aggregate. There is no lookback/grace window today.
- **Window boundaries are processing-time based** (`now` at run time), not
event-time. A window's correctness assumes its data has been ingested by the
time the interval fires.
- **Output is idempotent at compaction, not atomically exactly-once.** A retry,
an overlapping manual re-run, or a crash between the write and the watermark
update re-emits a window's rows, so duplicates can appear transiently. They are
collapsed to one row per `(dimensions, time)` the next time the destination
partition compacts — provided you declared the grouping dimensions in
[`tag_columns`](#idempotency-and-tag_columns) (a query with no grouping is
deduped automatically). Until compaction runs, a query over the destination may
see the duplicate rows.
This model fits **periodic roll-ups and downsampling of in-order, timely data**
(for example building 1m/5m bars from a clean feed). If your workload involves
late corrections, out-of-order events, or strict exactly-once aggregation, design
around these semantics (e.g. reprocess on a delay, or do the final roll-up as a
query-time aggregation).
## API endpoints [#api-endpoints]
### Create continuous query [#create-continuous-query]
Define a new continuous query:
```bash
POST /api/v1/continuous_queries
```
**Request Body**:
```json
{
"name": "cpu_hourly_avg",
"database": "telegraf",
"source_measurement": "cpu",
"destination_measurement": "cpu_hourly",
"query": "SELECT time_bucket('1 hour', time) AS time, host, AVG(usage_idle) AS avg_usage_idle, AVG(usage_user) AS avg_usage_user, COUNT(*) AS sample_count FROM telegraf.cpu GROUP BY time_bucket('1 hour', time), host",
"tag_columns": ["host"],
"interval": "1h",
"retention_policy": "90d",
"is_active": true
}
```
**Parameters**:
* `name` (string, required): Unique query identifier
* `database` (string, required): Target database name
* `source_measurement` (string, required): Source measurement to aggregate
* `destination_measurement` (string, required): Where to store results
* `query` (string, required): SQL aggregation query
* `tag_columns` (array of strings, optional): The **grouping dimension** columns in the query's output (e.g. `["host"]` for `GROUP BY host`). See [Idempotency and `tag_columns`](#idempotency-and-tag_columns) below — set this for any `GROUP BY` query so re-runs don't produce duplicate rows.
* `interval` (string, required): Time bucket interval (`1m`, `5m`, `1h`, `1d`, etc.)
* `retention_policy` (string, optional): Retention for aggregated data (e.g., `90d`, `365d`)
* `is_active` (boolean, required): Enable/disable the query
#### Idempotency and `tag_columns` [#idempotency-and-tag_columns]
Continuous-query output is made idempotent by Arc's compaction step: duplicate emissions of the same window (from a retry, an overlapping manual run, or a crash between the write and the watermark advance) are collapsed to one row per `(grouping dimensions, time)` when the destination partition compacts.
For this to work, Arc must know which output columns are the grouping dimensions:
* **A query with `GROUP BY `** (e.g. `GROUP BY host`) must list those dimensions in `tag_columns` (e.g. `"tag_columns": ["host"]`). Arc writes them as Parquet tag metadata and dedups on `(tags, time)`.
* **A query with no grouping** (one row per window, e.g. `SELECT AVG(x) …`) needs no `tag_columns` — Arc detects it produces one row per timestamp and dedups on time automatically.
* **If you group but forget to declare `tag_columns`**, Arc detects the multiple-rows-per-timestamp output and does **not** dedup it (to avoid deleting distinct series). The query still runs and its rows are correct, but duplicate windows will accumulate; a warning is logged asking you to add `tag_columns`.
`tag_columns` may not include `time` (time is always part of the dedup key). Names must be plain identifiers (letters, digits, `_`, `-`). Output-row timestamps are stamped with the window's start time; a query that does not select a `time` column now gets the correct window timestamp instead of the ingestion wall-clock.
This is not a breaking change. Existing continuous queries keep running with no action — the CQ database is migrated automatically and `tag_columns` is optional. Add `tag_columns` to a grouped CQ when you want its duplicate windows to collapse; until you do, it behaves exactly as before (append-only). Note that a CQ which does **not** select a `time` column now stamps output with the window start rather than the ingestion wall-clock, so its destination has a one-time timestamp discontinuity at the upgrade.
### List continuous queries [#list-continuous-queries]
Retrieve all continuous queries:
```bash
GET /api/v1/continuous_queries
```
**Response**:
```json
[
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"name": "cpu_hourly_avg",
"database": "telegraf",
"source_measurement": "cpu",
"destination_measurement": "cpu_hourly",
"interval": "1h",
"retention_policy": "90d",
"is_active": true,
"created_at": "2024-01-15T10:30:00Z",
"last_executed_at": "2024-01-20T02:00:00Z"
}
]
```
### Get single query [#get-single-query]
Retrieve a specific continuous query:
```bash
GET /api/v1/continuous_queries/{query_id}
```
### Update continuous query [#update-continuous-query]
Update an existing continuous query:
```bash
PUT /api/v1/continuous_queries/{query_id}
```
**Request Body**: Same as create query
### Delete continuous query [#delete-continuous-query]
Remove a continuous query:
```bash
DELETE /api/v1/continuous_queries/{query_id}
```
Deleting a continuous query does not delete the destination measurement or its data. The aggregated data remains queryable.
### Execute continuous query [#execute-continuous-query]
Manually trigger a continuous query:
```bash
POST /api/v1/continuous_queries/{query_id}/execute
```
**Request Body**:
```json
{
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-31T23:59:59Z",
"dry_run": false
}
```
**Parameters**:
* `start_time` (string, required): Start timestamp (ISO 8601 format)
* `end_time` (string, required): End timestamp (ISO 8601 format)
* `dry_run` (boolean, optional): Test without writing data (default: `false`)
**Response**:
```json
{
"query_id": "550e8400-e29b-41d4-a716-446655440000",
"rows_processed": 1000000,
"rows_written": 720,
"execution_time_ms": 2500,
"time_range": {
"start": "2024-01-01T00:00:00Z",
"end": "2024-01-31T23:59:59Z"
},
"dry_run": false
}
```
### View execution history [#view-execution-history]
View past executions of a continuous query:
```bash
GET /api/v1/continuous_queries/{query_id}/executions?limit=50
```
**Response**:
```json
[
{
"execution_id": "abc123",
"executed_at": "2024-01-20T02:00:00Z",
"start_time": "2024-01-19T00:00:00Z",
"end_time": "2024-01-20T00:00:00Z",
"rows_processed": 86400,
"rows_written": 24,
"execution_time_ms": 1200,
"status": "success"
}
]
```
## Query syntax [#query-syntax]
Continuous queries use standard analytical SQL with temporal optimizations.
### Recommended approach [#recommended-approach]
Use `epoch_us()` for timestamp conversion and `date_trunc()` for time bucketing:
```sql
SELECT
date_trunc('hour', epoch_us(time)) AS time,
host,
AVG(usage_idle) AS avg_usage_idle,
MAX(usage_user) AS max_usage_user,
MIN(usage_system) AS min_usage_system,
COUNT(*) AS sample_count
FROM telegraf.cpu
GROUP BY date_trunc('hour', epoch_us(time)), host
```
### Common aggregations [#common-aggregations]
These are common examples. Arc supports the **full analytical SQL aggregate set** — `MEDIAN`, `MODE`, `QUANTILE_CONT`, `APPROX_QUANTILE`, `ARG_MAX`, `HISTOGRAM`, `CORR`, `REGR_*` and the rest all work. See the [Querying guide](/arc/guides/querying/#useful-sql-functions) for more.
* `AVG()` - Average values
* `SUM()` - Sum of values
* `MIN()` - Minimum value
* `MAX()` - Maximum value
* `COUNT()` - Row count
* `STDDEV()` - Standard deviation
* `PERCENTILE_CONT()` - Percentile calculations
### Time bucketing [#time-bucketing]
**Using `date_trunc()`**:
```sql
-- Hourly buckets
date_trunc('hour', epoch_us(time))
-- Daily buckets
date_trunc('day', epoch_us(time))
-- 5-minute buckets (requires rounding)
date_trunc('hour', epoch_us(time)) + INTERVAL '5 minutes' * floor(extract(minute from epoch_us(time)) / 5)
```
### Including sample counts [#including-sample-counts]
Always include `COUNT(*)` to track how many raw samples each aggregate represents:
```sql
SELECT
date_trunc('hour', epoch_us(time)) AS time,
host,
AVG(usage_idle) AS avg_usage_idle,
COUNT(*) AS sample_count -- Important for data quality
FROM telegraf.cpu
GROUP BY date_trunc('hour', epoch_us(time)), host
```
## Usage examples [#usage-examples]
### Example 1: hourly CPU metrics [#example-1-hourly-cpu-metrics]
Aggregate per-second CPU metrics into hourly averages:
```python
import os
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create continuous query
response = requests.post(
"http://localhost:8000/api/v1/continuous_queries",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "cpu_hourly",
"database": "telegraf",
"source_measurement": "cpu",
"destination_measurement": "cpu_hourly",
"query": """
SELECT
date_trunc('hour', epoch_us(time)) AS time,
host,
AVG(usage_idle) AS avg_usage_idle,
AVG(usage_user) AS avg_usage_user,
AVG(usage_system) AS avg_usage_system,
MAX(usage_user) AS max_usage_user,
COUNT(*) AS sample_count
FROM telegraf.cpu
GROUP BY date_trunc('hour', epoch_us(time)), host
""",
"interval": "1h",
"retention_policy": "365d",
"is_active": True
}
)
query_id = response.json()["id"]
# Execute for the last 30 days
from datetime import datetime, timedelta
end_time = datetime.utcnow()
start_time = end_time - timedelta(days=30)
result = requests.post(
f"http://localhost:8000/api/v1/continuous_queries/{query_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"start_time": start_time.isoformat() + "Z",
"end_time": end_time.isoformat() + "Z"
}
)
print(f"Processed {result.json()['rows_processed']} rows")
print(f"Generated {result.json()['rows_written']} aggregated rows")
```
### Example 2: daily request summary [#example-2-daily-request-summary]
Aggregate API request logs into daily summaries:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create daily request summary
response = requests.post(
"http://localhost:8000/api/v1/continuous_queries",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "requests_daily",
"database": "logs",
"source_measurement": "api_requests",
"destination_measurement": "api_requests_daily",
"query": """
SELECT
date_trunc('day', epoch_us(time)) AS time,
endpoint,
status_code,
COUNT(*) AS total_requests,
AVG(response_time_ms) AS avg_response_time,
MAX(response_time_ms) AS max_response_time,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY response_time_ms) AS p95_response_time
FROM api_requests
GROUP BY date_trunc('day', epoch_us(time)), endpoint, status_code
""",
"interval": "1d",
"retention_policy": "730d", # 2 years
"is_active": True
}
)
```
### Example 3: 5-minute sensor readings [#example-3-5-minute-sensor-readings]
Downsample IoT sensor data to 5-minute intervals:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create 5-minute sensor aggregation
response = requests.post(
"http://localhost:8000/api/v1/continuous_queries",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "sensors_5min",
"database": "iot",
"source_measurement": "temperature",
"destination_measurement": "temperature_5min",
"query": """
SELECT
date_trunc('hour', epoch_us(time)) +
INTERVAL '5 minutes' * floor(extract(minute from epoch_us(time)) / 5) AS time,
sensor_id,
location,
AVG(temperature) AS avg_temperature,
MIN(temperature) AS min_temperature,
MAX(temperature) AS max_temperature,
COUNT(*) AS sample_count
FROM temperature
GROUP BY
date_trunc('hour', epoch_us(time)) +
INTERVAL '5 minutes' * floor(extract(minute from epoch_us(time)) / 5),
sensor_id,
location
""",
"interval": "5m",
"retention_policy": "90d",
"is_active": True
}
)
```
### Example 4: dry run testing [#example-4-dry-run-testing]
Test a continuous query before execution:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create the query
response = requests.post(
"http://localhost:8000/api/v1/continuous_queries",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={...}
)
query_id = response.json()["id"]
# Test with dry run
dry_run = requests.post(
f"http://localhost:8000/api/v1/continuous_queries/{query_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-02T00:00:00Z",
"dry_run": True
}
)
print(f"Would process {dry_run.json()['rows_processed']} rows")
print(f"Would generate {dry_run.json()['rows_written']} aggregated rows")
# If satisfied, execute for real
if dry_run.json()['rows_written'] > 0:
result = requests.post(
f"http://localhost:8000/api/v1/continuous_queries/{query_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-02T00:00:00Z",
"dry_run": False
}
)
```
## Storage benefits [#storage-benefits]
Continuous queries significantly reduce storage requirements:
### Before downsampling [#before-downsampling]
**Raw CPU metrics** (1-second intervals):
* 1 year = 31,536,000 rows per host
* 10 hosts = 315,360,000 rows
* Storage: \~20GB
### After downsampling to hourly [#after-downsampling-to-hourly]
**Hourly aggregates**:
* 1 year = 8,760 rows per host
* 10 hosts = 87,600 rows
* Storage: \~50MB
**Reduction**: \~400x smaller while maintaining hourly trend visibility.
### Multi-tier strategy [#multi-tier-strategy]
Combine different granularities for optimal storage:
```python
# Tier 1: Keep raw data for 7 days
# Tier 2: Hourly aggregates for 90 days
requests.post("/api/v1/continuous_queries", json={
"name": "cpu_hourly",
"interval": "1h",
"retention_policy": "90d"
})
# Tier 3: Daily aggregates for 2 years
requests.post("/api/v1/continuous_queries", json={
"name": "cpu_daily",
"source_measurement": "cpu_hourly", # Aggregate the hourly data
"destination_measurement": "cpu_daily",
"interval": "1d",
"retention_policy": "730d"
})
# Use retention policy to delete raw data after 7 days
requests.post("/api/v1/retention", json={
"database": "telegraf",
"measurement": "cpu",
"retention_days": 7
})
```
## Best practices [#best-practices]
### 1. Start conservative [#1-start-conservative]
Begin with longer intervals and adjust based on actual needs:
```python
# Start with hourly
{"interval": "1h"}
# If too coarse, reduce to 15 minutes
{"interval": "15m"}
```
### 2. Preserve source data initially [#2-preserve-source-data-initially]
Keep raw data while testing aggregations:
```python
# Create continuous query
create_query(...)
# Test aggregations thoroughly
execute_dry_run(...)
execute_for_real(...)
# Only after validation, apply retention to raw data
requests.post("/api/v1/retention", json={
"measurement": "cpu",
"retention_days": 30 # Keep raw for 30 days
})
```
### 3. Use dry run extensively [#3-use-dry-run-extensively]
Always test queries with dry run before full execution:
```python
# Test on small time range first
dry_run(start="2024-01-01", end="2024-01-02")
# Gradually expand
dry_run(start="2024-01-01", end="2024-01-07")
# Finally, full execution
execute(start="2024-01-01", end="2024-12-31")
```
### 4. Include sample counts [#4-include-sample-counts]
Track the number of raw samples in each aggregate:
```sql
SELECT
date_trunc('hour', epoch_us(time)) AS time,
COUNT(*) AS sample_count, -- Essential for data quality
AVG(value) AS avg_value
FROM measurement
GROUP BY date_trunc('hour', epoch_us(time))
```
This helps identify:
* Missing data (low sample counts)
* Data quality issues
* Unexpected patterns
### 5. Monitor execution performance [#5-monitor-execution-performance]
Track continuous query execution times:
```python
result = execute_query(...)
print(f"Execution time: {result['execution_time_ms']}ms")
print(f"Throughput: {result['rows_processed'] / (result['execution_time_ms'] / 1000):.0f} rows/sec")
# Alert if execution takes too long
if result['execution_time_ms'] > 60000: # 1 minute
print("Warning: Slow execution!")
```
### 6. Use appropriate intervals [#6-use-appropriate-intervals]
Match intervals to data characteristics:
**High-Frequency Data** (IoT sensors at 1-second intervals):
* 5-minute aggregates for recent analysis
* Hourly aggregates for medium-term
* Daily aggregates for long-term trends
**Medium-Frequency Data** (Application metrics at 1-minute intervals):
* Hourly aggregates for recent analysis
* Daily aggregates for long-term
**Low-Frequency Data** (Business metrics at hourly intervals):
* Daily aggregates
* Monthly aggregates for multi-year analysis
## Troubleshooting [#troubleshooting]
### No rows written [#no-rows-written]
**Problem**: Execution returns `rows_written: 0`.
**Solutions**:
* Verify source measurement contains data in the specified time range
* Check that the query syntax is correct
* Ensure `GROUP BY` clause matches aggregation columns
* Use dry run to inspect query results
### Query syntax errors [#query-syntax-errors]
**Problem**: Execution fails with SQL error.
**Solutions**:
* Test the query directly using the `/query` endpoint
* Verify column names exist in source measurement
* Check for dialect-specific syntax requirements
* Use `epoch_us()` for timestamp conversion
### Slow execution [#slow-execution]
**Problem**: Continuous query takes longer than expected.
**Solutions**:
* Reduce the time range per execution
* Ensure source measurement is properly compacted
* Consider creating indexes on frequently grouped columns
* Monitor query engine performance
### Duplicate data [#duplicate-data]
**Problem**: Re-running the query creates duplicate aggregates.
**Solutions**:
* Delete destination measurement data before re-execution:
```python
requests.post("/api/v1/delete", json={
"database": "telegraf",
"measurement": "cpu_hourly",
"where": f"time >= '{start_time}' AND time <= '{end_time}'"
})
```
* Or use `UPSERT` semantics if supported (future feature)
## Related topics [#related-topics]
* [Retention Policies](/arc-enterprise/data-lifecycle/retention-policies/) - Automatically delete old raw data after downsampling
* [Delete Operations](/arc-enterprise/data-lifecycle/delete-operations/) - Manually remove data ranges
* [Compaction](/arc-enterprise/advanced/compaction/) - Optimize file structure for better query performance
# Delete Operations (/arc-enterprise/data-lifecycle/delete-operations)
Arc supports deleting data using a rewrite-based approach that provides precise deletion with zero overhead on write and query operations.
Delete operations must be explicitly enabled in configuration for safety.
## Overview [#overview]
Arc's delete operations provide:
* **Precise Control**: Delete specific rows using WHERE clauses
* **Zero Runtime Overhead**: No performance impact on writes or queries
* **Physical Removal**: Data is permanently removed by rewriting Parquet files
* **Safety Mechanisms**: Multiple safeguards prevent accidental deletion
* **Dry Run Mode**: Test operations before execution
## How it works [#how-it-works]
Arc uses a rewrite-based deletion approach:
### 1. Find affected files [#1-find-affected-files]
Scan the measurement directory to identify Parquet files containing rows that match the WHERE clause.
### 2. Rewrite files [#2-rewrite-files]
For each affected file:
1. Load the file into an Arrow table
2. Filter out matching rows: `SELECT * FROM table WHERE NOT (delete_clause)`
3. Write filtered data to a temporary file
4. Atomically replace the original file using `os.replace()`
### 3. Cleanup [#3-cleanup]
* Files that become empty after filtering are deleted entirely
* Files with remaining data are replaced with their rewritten versions
* All operations use atomic file replacement to ensure data integrity
### Atomic safety [#atomic-safety]
System crashes during deletion result in either the old file or the new file being present, never corruption or partial writes.
## Configuration [#configuration]
Delete operations must be explicitly enabled and configured.
### Configuration file [#configuration-file]
Edit `arc.toml`:
```toml
[delete]
enabled = true
confirmation_threshold = 10000
max_rows_per_delete = 1000000
```
### Environment variables [#environment-variables]
```bash
export DELETE_ENABLED=true
export DELETE_CONFIRMATION_THRESHOLD=10000
export DELETE_MAX_ROWS=1000000
```
### Configuration parameters [#configuration-parameters]
* `enabled` (boolean): Enable/disable delete functionality (default: `false`)
* `confirmation_threshold` (integer): Row count requiring explicit confirmation (default: `10000`)
* `max_rows_per_delete` (integer): Maximum rows allowed per operation (default: `1000000`)
## API endpoints [#api-endpoints]
### Delete data [#delete-data]
Execute a delete operation:
```bash
POST /api/v1/delete
```
**Request Body**:
```json
{
"database": "telegraf",
"measurement": "cpu",
"where": "host = 'server01' AND time < '2024-01-01'",
"dry_run": false,
"confirm": false
}
```
**Parameters**:
* `database` (string, required): Target database name
* `measurement` (string, required): Target measurement name
* `where` (string, required): SQL WHERE clause for deletion
* `dry_run` (boolean, optional): Test without deleting (default: `false`)
* `confirm` (boolean, optional): Confirm large operations (default: `false`)
**Response**:
```json
{
"deleted_count": 15000,
"affected_files": 3,
"rewritten_files": 2,
"deleted_files": 1,
"execution_time_ms": 1250,
"files": [
{
"path": "/data/telegraf/cpu/2023-12-15.parquet",
"action": "rewritten",
"rows_before": 10000,
"rows_after": 5000
},
{
"path": "/data/telegraf/cpu/2023-12-20.parquet",
"action": "deleted",
"rows_before": 5000,
"rows_after": 0
}
]
}
```
### Get configuration [#get-configuration]
Retrieve current delete configuration:
```bash
GET /api/v1/delete/config
```
**Response**:
```json
{
"enabled": true,
"confirmation_threshold": 10000,
"max_rows_per_delete": 1000000
}
```
## Safety mechanisms [#safety-mechanisms]
### 1. WHERE clause required [#1-where-clause-required]
Delete operations **must** include a WHERE clause to prevent accidental full-table deletion.
**Intentional Full Delete**:
```json
{
"where": "1=1" // Explicitly delete all rows
}
```
### 2. Confirmation threshold [#2-confirmation-threshold]
Operations exceeding the configured threshold require explicit confirmation:
```json
{
"where": "time < '2024-01-01'",
"confirm": true // Required if deleted_count > threshold
}
```
**Without Confirmation**:
```json
{
"error": "Operation would delete 15000 rows, exceeding threshold of 10000. Set confirm=true to proceed."
}
```
### 3. Maximum rows limit [#3-maximum-rows-limit]
Hard cap prevents extremely large operations that could exhaust resources:
```json
{
"error": "Operation would delete 2000000 rows, exceeding maximum of 1000000"
}
```
### 4. Atomic file replacement [#4-atomic-file-replacement]
Files are replaced atomically using `os.replace()`, ensuring:
* No partial writes
* No data corruption
* Recovery from crashes (either old or new file exists)
## Usage examples [#usage-examples]
### Example 1: delete old data [#example-1-delete-old-data]
```python
import os
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Delete data older than a specific date
response = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "time < '2024-01-01'"
}
)
print(f"Deleted {response.json()['deleted_count']} rows")
print(f"Execution time: {response.json()['execution_time_ms']}ms")
```
### Example 2: delete specific host data [#example-2-delete-specific-host-data]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Delete data from a specific host
response = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "host = 'server01' OR host = 'server02'"
}
)
```
### Example 3: dry run first [#example-3-dry-run-first]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Always test with dry run before deleting
dry_run = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "host = 'server01'",
"dry_run": True
}
)
print(f"Would delete {dry_run.json()['deleted_count']} rows")
print(f"Affected files: {dry_run.json()['affected_files']}")
# Review the files that would be affected
for file in dry_run.json()['files']:
print(f" {file['path']}: {file['rows_before']} -> {file['rows_after']} rows")
# If satisfied, execute for real
if input("Proceed? (yes/no): ") == "yes":
result = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "host = 'server01'",
"dry_run": False
}
)
print(f"Deleted {result.json()['deleted_count']} rows")
```
### Example 4: delete with confirmation [#example-4-delete-with-confirmation]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Large delete requiring confirmation
response = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": "time < '2023-01-01'",
"confirm": True # Explicitly confirm large operation
}
)
```
### Example 5: complex WHERE clause [#example-5-complex-where-clause]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Delete based on multiple conditions
response = requests.post(
"http://localhost:8000/api/v1/delete",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"database": "telegraf",
"measurement": "cpu",
"where": """
host IN ('server01', 'server02', 'server03')
AND time BETWEEN '2023-01-01' AND '2023-06-30'
AND usage_idle < 10
"""
}
)
```
## Performance characteristics [#performance-characteristics]
Delete operations are computationally expensive but designed for infrequent use:
### Execution times [#execution-times]
**Small Files** (10MB):
* Read + Filter + Write: \~50-100ms per file
**Medium Files** (100MB):
* Read + Filter + Write: \~500ms-1s per file
**Large Files** (1GB):
* Read + Filter + Write: \~2-5s per file
### Performance factors [#performance-factors]
* **File Size**: Larger files take longer to rewrite
* **Selectivity**: Fewer deleted rows = faster (less data movement)
* **Storage I/O**: Disk speed affects read/write performance
* **Concurrent Load**: Other operations may slow deletion
## Best practices [#best-practices]
### 1. Keep disabled by default [#1-keep-disabled-by-default]
Only enable delete operations when needed:
```toml
[delete]
enabled = false # Enable only when necessary
```
### 2. Always use dry run [#2-always-use-dry-run]
Test operations before execution to verify scope:
```python
# Step 1: Dry run
result = requests.post(..., json={"dry_run": True})
print(f"Would delete {result['deleted_count']} rows")
# Step 2: Review affected files
for file in result['files']:
print(f"{file['path']}: {file['action']}")
# Step 3: Execute if satisfied
result = requests.post(..., json={"dry_run": False, "confirm": True})
```
### 3. Consider retention policies [#3-consider-retention-policies]
For time-based deletion, use [retention policies](/arc-enterprise/data-lifecycle/retention-policies/) instead:
```python
# Instead of manual deletes:
# requests.post("/api/v1/delete", json={"where": "time < '2024-01-01'"})
# Use retention policies:
requests.post("/api/v1/retention", json={
"retention_days": 90,
"buffer_days": 7
})
```
### 4. Monitor execution times [#4-monitor-execution-times]
Track deletion performance for capacity planning:
```python
import time
start = time.time()
result = requests.post("/api/v1/delete", json={...})
elapsed = time.time() - start
print(f"Deleted {result['deleted_count']} rows in {elapsed:.2f}s")
```
### 5. Batch large deletes [#5-batch-large-deletes]
Break large deletions into smaller batches by time range:
```python
from datetime import datetime, timedelta
# Instead of one large delete:
# WHERE time < '2023-01-01'
# Batch by month:
start = datetime(2022, 1, 1)
while start < datetime(2023, 1, 1):
end = start + timedelta(days=30)
requests.post("/api/v1/delete", json={
"where": f"time >= '{start.isoformat()}' AND time < '{end.isoformat()}'"
})
start = end
```
### 6. Understand storage impact [#6-understand-storage-impact]
Deletion rewrites files, which may temporarily increase storage usage:
```python
# Before deletion: 100MB original file
# During deletion: 100MB original + 60MB temp file = 160MB
# After deletion: 60MB rewritten file
```
Ensure sufficient disk space for temporary files during operations.
## Limitations [#limitations]
### Not for frequent operations [#not-for-frequent-operations]
Delete operations rewrite entire Parquet files, making them expensive. They are designed for **infrequent, manual operations** only.
**Use Cases**:
* Removing test data
* Deleting specific hosts/sensors
* One-time cleanup operations
**Not Suitable For**:
* Automated recurring deletions (use retention policies)
* High-frequency data cleanup
* Real-time data removal
### Explicit WHERE required [#explicit-where-required]
Full-table deletion requires explicit `WHERE 1=1`:
```python
# This will fail:
{"where": ""} # Error: WHERE clause required
# Explicit full delete:
{"where": "1=1", "confirm": True}
```
### Maximum row limits [#maximum-row-limits]
Large deletions are subject to `max_rows_per_delete` configuration:
```python
# Will fail if exceeds limit:
{"where": "time < '2020-01-01'"} # May exceed max_rows
# Solution: Batch by time range
{"where": "time >= '2023-01-01' AND time < '2023-02-01'"}
```
### File-level locking [#file-level-locking]
During deletion, affected files are locked. Concurrent writes may be delayed.
## Troubleshooting [#troubleshooting]
### Delete not enabled [#delete-not-enabled]
**Problem**: `DELETE_ENABLED=false` or not configured.
**Solution**:
```toml
[delete]
enabled = true
```
### Confirmation required [#confirmation-required]
**Problem**: Operation exceeds confirmation threshold.
**Solution**: Add `confirm: true`:
```json
{"confirm": true}
```
### Exceeds maximum rows [#exceeds-maximum-rows]
**Problem**: Deletion would affect more rows than `max_rows_per_delete`.
**Solutions**:
1. Batch the operation by time range
2. Increase `max_rows_per_delete` (carefully)
3. Use retention policies for large-scale cleanup
### No rows deleted [#no-rows-deleted]
**Problem**: `deleted_count: 0` but expected deletions.
**Solutions**:
* Verify WHERE clause syntax matches data
* Check that data exists in the specified database/measurement
* Use dry run to inspect affected files
### Slow execution [#slow-execution]
**Problem**: Delete operations take longer than expected.
**Solutions**:
* Check file sizes (large files take longer)
* Monitor disk I/O performance
* Batch operations during low-traffic periods
* Consider using retention policies for time-based cleanup
## Related topics [#related-topics]
* [Retention Policies](/arc-enterprise/data-lifecycle/retention-policies/) - Automated time-based deletion
* [Continuous Queries](/arc-enterprise/data-lifecycle/continuous-queries/) - Downsample before deletion
* [Compaction](/arc-enterprise/advanced/compaction/) - File optimization for better performance
# Data Lifecycle (/arc-enterprise/data-lifecycle)
Arc OSS provides retention policies and continuous queries but expects you to trigger them yourself through the API. Arc Enterprise adds a scheduler: define the policy once with a cron expression and the cluster runs it. See [Automated scheduling](/arc-enterprise/operations/automated-scheduling/).
Tiered storage is Enterprise-only. It moves aged partitions to archive storage classes while keeping them queryable, which is usually the largest single lever on storage cost.
# Retention Policies (/arc-enterprise/data-lifecycle/retention-policies)
Retention policies allow you to automatically manage data lifecycle by defining how long data should be kept.
Arc Enterprise includes automatic retention enforcement. See [Automated Scheduling](/arc-enterprise/operations/automated-scheduling/) to configure scheduled execution.
## Overview [#overview]
Retention policies in Arc help you:
* Define data retention periods at database or measurement level
* Automatically clean up old data through manual execution
* Reduce storage costs by removing unnecessary historical data
* Maintain compliance with data retention requirements
* Test deletion operations safely with dry-run mode
## How it works [#how-it-works]
Arc implements retention through physical file deletion:
1. **Scanning**: Examines Parquet files in measurement directories
2. **Metadata Analysis**: Reads file metadata to find maximum timestamps
3. **Identification**: Locates files where all rows are older than the cutoff date
4. **Deletion**: Physically removes entire files from disk
**Cutoff Calculation**: `cutoff_date = today - retention_days - buffer_days`
## API endpoints [#api-endpoints]
### Create policy [#create-policy]
Create a new retention policy:
```bash
POST /api/v1/retention
```
**Request Body**:
```json
{
"name": "delete_old_metrics",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": true
}
```
**Parameters**:
* `name` (string, required): Unique policy identifier
* `database` (string, required): Target database name
* `measurement` (string, optional): Target measurement (null for database-wide)
* `retention_days` (integer, required): Number of days to retain data
* `buffer_days` (integer, required): Safety margin in days
* `is_active` (boolean, required): Enable/disable the policy
### List policies [#list-policies]
Retrieve all retention policies:
```bash
GET /api/v1/retention
```
**Response**:
```json
[
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"name": "delete_old_metrics",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": true,
"created_at": "2024-01-15T10:30:00Z",
"last_executed_at": "2024-01-20T02:00:00Z",
"last_deleted_count": 1500
}
]
```
### Get single policy [#get-single-policy]
Retrieve a specific retention policy:
```bash
GET /api/v1/retention/{policy_id}
```
### Update policy [#update-policy]
Update an existing retention policy:
```bash
PUT /api/v1/retention/{policy_id}
```
**Request Body**: Same as create policy
### Delete policy [#delete-policy]
Remove a retention policy:
```bash
DELETE /api/v1/retention/{policy_id}
```
### Execute policy [#execute-policy]
Manually trigger a retention policy:
```bash
POST /api/v1/retention/{policy_id}/execute
```
**Request Body**:
```json
{
"dry_run": false,
"confirm": true
}
```
**Dry Run Example**:
```json
{
"dry_run": true,
"confirm": false
}
```
**Response**:
```json
{
"policy_id": "550e8400-e29b-41d4-a716-446655440000",
"cutoff_date": "2023-10-22T00:00:00Z",
"files_to_delete": [
"/data/telegraf/cpu/2023-10-15.parquet",
"/data/telegraf/cpu/2023-10-20.parquet"
],
"total_files": 2,
"dry_run": true
}
```
### View execution history [#view-execution-history]
View past executions of a retention policy:
```bash
GET /api/v1/retention/{policy_id}/executions?limit=50
```
**Response**:
```json
[
{
"execution_id": "abc123",
"executed_at": "2024-01-20T02:00:00Z",
"deleted_count": 1500,
"execution_time_ms": 2500,
"status": "success"
}
]
```
## Configuration parameters [#configuration-parameters]
### Retention days [#retention-days]
The number of days to keep data before it becomes eligible for deletion. Choose based on:
* Business requirements
* Compliance regulations
* Storage capacity
* Query patterns
**Example**: `retention_days: 90` keeps data for 90 days.
### Buffer days [#buffer-days]
A safety margin added to the retention period to prevent accidental deletion of recent data.
**Recommended Values**:
* Development: 7 days
* Production: 14-30 days
**Example**: With `retention_days: 90` and `buffer_days: 7`, data older than 97 days will be deleted.
### Database vs measurement level [#database-vs-measurement-level]
**Database-wide policy**:
```json
{
"database": "telegraf",
"measurement": null,
"retention_days": 365
}
```
**Measurement-specific policy**:
```json
{
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90
}
```
Use measurement-specific policies for granular control over different data types.
## Usage examples [#usage-examples]
### Example 1: clean old metrics [#example-1-clean-old-metrics]
```python
import os
import requests
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create a retention policy for old CPU metrics
response = requests.post(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "cpu_cleanup",
"database": "telegraf",
"measurement": "cpu",
"retention_days": 90,
"buffer_days": 7,
"is_active": True
}
)
policy_id = response.json()["id"]
# Test with dry run first
dry_run = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"dry_run": True, "confirm": False}
)
print(f"Would delete {dry_run.json()['total_files']} files")
# Execute for real
if input("Proceed? (yes/no): ") == "yes":
result = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={"dry_run": False, "confirm": True}
)
print(f"Deleted {result.json()['total_files']} files")
```
### Example 2: database-wide retention [#example-2-database-wide-retention]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Apply retention to all measurements in a database
response = requests.post(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"},
json={
"name": "database_cleanup",
"database": "telegraf",
"measurement": None, # Apply to all measurements
"retention_days": 180,
"buffer_days": 14,
"is_active": True
}
)
```
### Example 3: list and monitor policies [#example-3-list-and-monitor-policies]
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# List all policies
policies = requests.get(
"http://localhost:8000/api/v1/retention",
headers={"Authorization": f"Bearer {ARC_TOKEN}"}
)
for policy in policies.json():
print(f"Policy: {policy['name']}")
print(f" Last executed: {policy['last_executed_at']}")
print(f" Last deleted: {policy['last_deleted_count']} rows")
# Get execution history
history = requests.get(
f"http://localhost:8000/api/v1/retention/{policy['id']}/executions?limit=10",
headers={"Authorization": f"Bearer {ARC_TOKEN}"}
)
print(f" Recent executions: {len(history.json())}")
```
## Best practices [#best-practices]
### 1. Always test first [#1-always-test-first]
Use dry-run mode before executing retention policies:
```python
# Always start with dry run
result = requests.post(
f"http://localhost:8000/api/v1/retention/{policy_id}/execute",
json={"dry_run": True, "confirm": False}
)
# Review what will be deleted
print(f"Files to delete: {result.json()['files_to_delete']}")
```
### 2. Use buffer days [#2-use-buffer-days]
Implement a safety buffer to prevent accidental deletion:
```json
{
"retention_days": 90,
"buffer_days": 14 // 14-day safety margin
}
```
### 3. Start conservative [#3-start-conservative]
Begin with longer retention periods and gradually shorten:
```json
// Start here
{"retention_days": 365, "buffer_days": 30}
// After monitoring, reduce if needed
{"retention_days": 180, "buffer_days": 14}
```
### 4. Test in non-production [#4-test-in-non-production]
Create and test policies in a development environment first:
```bash
# Development environment
export ARC_ENV=dev
# Test policies thoroughly before production
```
### 5. Monitor execution history [#5-monitor-execution-history]
Regularly check the `last_deleted_count` field:
```python
# Check if deletion counts are as expected
policy = requests.get(f"/api/v1/retention/{policy_id}").json()
if policy['last_deleted_count'] > 10000:
print("Warning: Large deletion detected!")
```
### 6. Use measurement-specific policies [#6-use-measurement-specific-policies]
Create granular policies for different data types:
```python
# High-frequency metrics - shorter retention
{"measurement": "cpu", "retention_days": 30}
# Business metrics - longer retention
{"measurement": "revenue", "retention_days": 730}
```
## Important limitations [#important-limitations]
### Local storage only [#local-storage-only]
Currently, retention policies only work with local filesystem storage. Cloud storage backends (S3, MinIO, GCS) are not yet implemented.
### File-level granularity [#file-level-granularity]
Retention operates at the file level, not row level. A file is only deleted if **all** rows are older than the cutoff date.
For optimal retention policy effectiveness, ensure your data is properly compacted. Files with mixed timestamps may not be eligible for deletion.
### No rollback [#no-rollback]
Deleted data cannot be recovered. Always:
1. Use dry-run mode first
2. Maintain backups of critical data
3. Test in non-production environments
### Sequential processing [#sequential-processing]
Retention policies process measurements sequentially. Large databases may take time to process.
### Works best with compacted files [#works-best-with-compacted-files]
Retention policies are most effective when files contain data from similar time periods. Enable [automatic compaction](/arc-enterprise/advanced/compaction/) for better results.
## Troubleshooting [#troubleshooting]
### No files being deleted [#no-files-being-deleted]
**Problem**: Dry run shows 0 files to delete.
**Solutions**:
* Check that data actually exists older than `retention_days + buffer_days`
* Verify the policy targets the correct database and measurement
* Ensure files are fully older than the cutoff (file-level granularity)
### Policy not executing [#policy-not-executing]
**Problem**: Manual execution returns an error.
**Solutions**:
* Verify the policy `is_active` is set to `true`
* Check that `confirm: true` is set for actual execution
* Ensure you have write permissions on the data directory
### Unexpected file count [#unexpected-file-count]
**Problem**: More/fewer files than expected are being deleted.
**Solutions**:
* Remember: Only files where **all rows** are older than cutoff are deleted
* Check file timestamps using `ls -l` on the measurement directory
* Review recent compaction activity that may have merged files
## Related topics [#related-topics]
* [Delete Operations](/arc-enterprise/data-lifecycle/delete-operations/) - Manual delete operations for specific data
* [Continuous Queries](/arc-enterprise/data-lifecycle/continuous-queries/) - Downsample data before deletion
* [Compaction](/arc-enterprise/advanced/compaction/) - Optimize file structure for better retention
# Tiered Storage (/arc-enterprise/data-lifecycle/tiered-storage)
Reduce storage costs with automatic hot/cold data tiering. Recent data stays on fast local storage while older data moves to cost-efficient archive storage.
{/* TODO(diagram): what a query touching cold-tier data does under each retrieval_mode -
the request path, where it blocks on an archive restore, and what the client sees
meanwhile. The retrieval modes are the page's subtlest concept and are text-only. */}
## Overview [#overview]
Arc Enterprise implements a 2-tier storage model:
```text
┌─────────────────────────────────────────────────────────────┐
│ Arc Tiered Storage │
│ │
│ HOT TIER (Local / Primary Storage Backend) │
│ ├── Recent data (configurable, default: 30 days) │
│ ├── Optimized for low-latency queries │
│ └── Cost: $$$ │
│ │
│ │ Automatic migration (age-based) │
│ ▼ │
│ │
│ COLD TIER (S3 Glacier / Azure Archive) │
│ ├── Historical data (30+ days) │
│ ├── Optimized for cost efficiency │
│ └── Cost: $ │
│ │
└─────────────────────────────────────────────────────────────┘
```
**Key features:**
* **Age-based migration** — Data older than a configurable threshold automatically moves to the cold tier
* **Per-database policies** — Override global defaults for specific databases
* **Hot-only databases** — Exclude specific databases from tiering entirely
* **Scheduled migrations** — Cron-based scheduler runs migrations automatically
* **Manual migrations** — Trigger migrations on-demand via API
* **Zero recompression** — Files move as-is with no re-encoding overhead
* **Transparent queries** — Queries automatically span both tiers
## Configuration [#configuration]
### Global settings [#global-settings]
```toml
[tiered_storage]
enabled = true
migration_schedule = "0 2 * * *" # Cron: run at 2am daily
migration_max_concurrent = 4 # Parallel file transfers
migration_batch_size = 100 # Files per migration batch
default_hot_max_age_days = 30 # Data older than this moves to cold
```
### Cold tier backend [#cold-tier-backend]
```toml
[tiered_storage.cold]
enabled = true
backend = "s3"
s3_bucket = "arc-archive"
s3_region = "us-east-1"
# s3_access_key = "" # Use env: ARC_TIERED_STORAGE_COLD_S3_ACCESS_KEY
# s3_secret_key = "" # Use env: ARC_TIERED_STORAGE_COLD_S3_SECRET_KEY
s3_use_ssl = true
s3_path_style = false
s3_storage_class = "GLACIER" # GLACIER, DEEP_ARCHIVE, GLACIER_IR, STANDARD_IA
retrieval_mode = "standard" # standard, expedited, bulk
```
**Storage classes:**
| Class | Use Case | Retrieval Time |
| -------------- | -------------------------------------- | ---------------- |
| `STANDARD_IA` | Infrequent access, immediate retrieval | Milliseconds |
| `GLACIER_IR` | Archive with instant retrieval | Milliseconds |
| `GLACIER` | Long-term archive (default) | Minutes to hours |
| `DEEP_ARCHIVE` | Lowest cost, rare access | Up to 12 hours |
```toml
[tiered_storage.cold]
enabled = true
backend = "azure"
azure_container = "arc-archive"
azure_account_name = "your_account"
# azure_account_key = "" # Use env: ARC_TIERED_STORAGE_COLD_AZURE_ACCOUNT_KEY
# Or use managed identity:
# azure_use_managed_identity = true
```
### Environment variables [#environment-variables]
```bash
# Global tiering settings
ARC_TIERED_STORAGE_ENABLED=true
ARC_TIERED_STORAGE_MIGRATION_SCHEDULE="0 2 * * *"
ARC_TIERED_STORAGE_MIGRATION_MAX_CONCURRENT=4
ARC_TIERED_STORAGE_MIGRATION_BATCH_SIZE=100
ARC_TIERED_STORAGE_DEFAULT_HOT_MAX_AGE_DAYS=30
# Cold tier (S3)
ARC_TIERED_STORAGE_COLD_ENABLED=true
ARC_TIERED_STORAGE_COLD_BACKEND=s3
ARC_TIERED_STORAGE_COLD_S3_BUCKET=arc-archive
ARC_TIERED_STORAGE_COLD_S3_REGION=us-east-1
ARC_TIERED_STORAGE_COLD_S3_ACCESS_KEY=your_key
ARC_TIERED_STORAGE_COLD_S3_SECRET_KEY=your_secret
ARC_TIERED_STORAGE_COLD_S3_USE_SSL=true
ARC_TIERED_STORAGE_COLD_S3_STORAGE_CLASS=GLACIER
ARC_TIERED_STORAGE_COLD_RETRIEVAL_MODE=standard
# Cold tier (Azure)
ARC_TIERED_STORAGE_COLD_BACKEND=azure
ARC_TIERED_STORAGE_COLD_AZURE_CONTAINER=arc-archive
ARC_TIERED_STORAGE_COLD_AZURE_ACCOUNT_NAME=your_account
ARC_TIERED_STORAGE_COLD_AZURE_ACCOUNT_KEY=your_key
```
Store access keys and secret keys in environment variables rather than in `arc.toml`. This keeps secrets out of version control.
## Per-database policies [#per-database-policies]
Override the global `default_hot_max_age_days` for specific databases, or exclude databases from tiering entirely.
### Create policy [#create-policy]
```bash
curl -X POST http://localhost:8000/api/v1/tiering/policies \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"database": "telemetry",
"hot_max_age_days": 7
}'
```
### Create hot-only policy [#create-hot-only-policy]
Exclude a database from tiering:
```bash
curl -X POST http://localhost:8000/api/v1/tiering/policies \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"database": "realtime",
"hot_only": true
}'
```
### List policies [#list-policies]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/tiering/policies
```
### Get policy [#get-policy]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/tiering/policies/telemetry
```
### Update policy [#update-policy]
```bash
curl -X PUT http://localhost:8000/api/v1/tiering/policies/telemetry \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"hot_max_age_days": 14}'
```
### Delete policy [#delete-policy]
```bash
curl -X DELETE http://localhost:8000/api/v1/tiering/policies/telemetry \
-H "Authorization: Bearer $TOKEN"
```
## API reference [#api-reference]
All tiering endpoints require admin authentication.
### Get tiering status [#get-tiering-status]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/tiering/status
```
**Response:**
```json
{
"success": true,
"data": {
"enabled": true,
"cold_backend": "s3",
"default_hot_max_age_days": 30,
"migration_schedule": "0 2 * * *",
"last_migration": "2026-02-13T02:00:00Z"
}
}
```
### List files by tier [#list-files-by-tier]
```bash
# All files
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/tiering/files
# Filter by tier
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/tiering/files?tier=cold"
# Filter by database
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/tiering/files?database=telemetry&limit=50"
```
### Trigger manual migration [#trigger-manual-migration]
```bash
curl -X POST http://localhost:8000/api/v1/tiering/migrate \
-H "Authorization: Bearer $TOKEN"
```
**Response:**
```json
{
"success": true,
"data": {
"message": "Migration started",
"files_eligible": 42
}
}
```
### Get migration statistics [#get-migration-statistics]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/tiering/stats
```
**Response:**
```json
{
"success": true,
"data": {
"total_files_migrated": 1250,
"total_bytes_migrated": 53687091200,
"hot_files": 340,
"cold_files": 1250,
"last_migration_duration_ms": 45000,
"last_migration_files": 42
}
}
```
## Best practices [#best-practices]
1. **Start with 30-day hot retention** — This is a good default for most workloads. Monitor query patterns and adjust based on how often historical data is accessed.
2. **Use GLACIER for long-term archive** — It offers the best cost-to-retrieval tradeoff. Use `STANDARD_IA` if you need frequent access to cold data.
3. **Schedule migrations during off-peak hours** — The default `0 2 * * *` (2am daily) works well for most deployments.
4. **Use per-database policies for different retention needs** — Real-time dashboards may need 7-day hot data, while compliance databases may need 90 days.
5. **Mark real-time databases as hot-only** — Databases used exclusively for real-time dashboards should skip tiering to avoid any retrieval latency.
6. **Use IAM roles or managed identity** — For cloud deployments, use IAM roles (AWS) or managed identity (Azure) instead of access keys.
## Next steps [#next-steps]
* [Audit Logging](/arc-enterprise/security/audit-logging/) — Track tiering operations for compliance
* [Automated Scheduling](/arc-enterprise/operations/automated-scheduling/) — Combine tiering with scheduled retention policies
# AWS Marketplace (/arc-enterprise/installation/aws-marketplace)
Deploy Arc on AWS with a single click using our pre-configured AMI from AWS Marketplace.
## Overview [#overview]
Arc is available on AWS Marketplace as a ready-to-run AMI. No license keys, no sales calls—just subscribe and launch.
**What you get:**
* Pre-installed Arc with systemd service
* Ubuntu-based AMI
* All features enabled
* Free to use (AGPL-3.0)
* Optional enterprise support ($500/month)
## Prerequisites [#prerequisites]
* AWS Account
* EC2 key pair for SSH access
* VPC with appropriate subnets
* (Production) ACM certificate for HTTPS
## Quick start (dev/testing only) [#quick-start-devtesting-only]
This pattern exposes your database directly to the internet. Use only for testing and evaluation.
1. **Subscribe to Arc on AWS Marketplace**
[Open Arc on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-mkhhh2vk4fyss)
2. **Launch an EC2 instance** with a public IP
3. **Configure security group** to allow port 8000 (restrict to your IP)
4. **Get your admin token:**
```bash
# SSH into your instance (Ubuntu-based AMI)
ssh -i your-key.pem ubuntu@your-instance-ip
# View Arc logs to find the admin token
sudo journalctl -u arc | grep "Admin token"
# You'll see something like:
# Admin token: ark_abc123...xyz
```
5. **Test the connection:**
```bash
export ARC_URL="http://your-instance-ip:8000"
export ARC_TOKEN="your-token-here"
curl $ARC_URL/health
```
## Production deployment (recommended) [#production-deployment-recommended]
For production workloads, deploy Arc behind an Application Load Balancer in a private subnet.
### Architecture [#architecture]
```text
┌──────────────────────────────────────────────────────────┐
│ VPC (10.0.0.0/16) │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Public Subnet (10.0.0.0/24) │ │
│ │ │ │
│ │ ┌──────────────────────────────────┐ │ │
│ │ │ Application Load Balancer (ALB) │ │ │
│ │ │ - HTTPS (443) with SSL cert │ │ │
│ │ │ - Terminates TLS │ │ │
│ │ │ - Health checks │ │ │
│ │ └────────────┬─────────────────────┘ │ │
│ │ │ │ │
│ └───────────────┼────────────────────────────────────┘ │
│ │ HTTP (8000) │
│ ┌───────────────┼────────────────────────────────────┐ │
│ │ Private Subnet (10.0.1.0/24) │ │ │
│ │ ▼ │ │
│ │ ┌──────────────────────────────────────┐ │ │
│ │ │ Arc Instance │ │ │
│ │ │ - No public IP │ │ │
│ │ │ - Port 8000 from ALB only │ │ │
│ │ │ - EBS storage for Parquet files │ │ │
│ │ └──────────────────────────────────────┘ │ │
│ │ │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ Optional: │
│ • Telegraf in same VPC │
│ • Grafana in same VPC │
│ • NAT Gateway for outbound │
└──────────────────────────────────────────────────────────┘
```
**Benefits:**
* SSL termination at ALB (free certificate from ACM)
* Arc in private subnet (no internet exposure)
* Security groups restrict traffic
* Health checks and monitoring
### Step 1: subscribe to Arc [#step-1-subscribe-to-arc]
1. Go to [Arc on AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-mkhhh2vk4fyss)
2. Click **View purchase options**
3. Click **Subscribe**
4. Wait for subscription to activate, then click **Continue to Configuration**
### Step 2: Launch configuration [#step-2-launch-configuration]
1. Select your **Region**
2. Choose an **Instance type**:
* Testing: `t3.large` (2 vCPU, 8 GB RAM)
* Production: `m8a.xlarge` or larger
3. Click **Continue to Launch**
### Step 3: configure network settings [#step-3-configure-network-settings]
1. **VPC:** Select your existing VPC or create a new one
2. **Subnet:** Choose a private subnet (no internet gateway route)
3. **Security Group:** Create a new one that allows:
* Inbound: Port 8000 from your ALB security group only
* Outbound: As needed for your environment
4. Click **Launch**
### Step 4: create a target group [#step-4-create-a-target-group]
Before creating the load balancer, create a target group.
1. Go to **EC2 Console** → **Target Groups** → **Create target group**
2. Configure:
* **Target type:** Instances
* **Target group name:** `arc-target-group`
* **Protocol:** HTTP
* **Port:** 8000
* **VPC:** Same VPC as Arc instance
3. **Health checks:**
* **Path:** `/health`
* **Healthy threshold:** 2
* **Unhealthy threshold:** 2
* **Timeout:** 5 seconds
* **Interval:** 30 seconds
4. Click **Next**, select your Arc instance, click **Include as pending below**
5. Click **Create target group**
### Step 5: create the application load balancer [#step-5-create-the-application-load-balancer]
1. Go to **EC2 Console** → **Load Balancers** → **Create load balancer**
2. Select **Application Load Balancer**
3. Configure:
* **Name:** `arc-alb`
* **Scheme:** Internet-facing
* **IP address type:** IPv4
4. **Network mapping:**
* Select your VPC
* Select at least two public subnets (one per AZ)
5. **Security groups:**
* Allow inbound HTTPS (443) from your allowed IP ranges
6. **Listeners:**
* Protocol: HTTPS, Port: 443
* Default action: Forward to `arc-target-group`
7. **Secure listener settings:**
* Security policy: `ELBSecurityPolicy-TLS13-1-2-2021-06`
* Certificate: Select from ACM (free) or import your own
8. Click **Create load balancer**
### Step 6: point your domain to the load balancer [#step-6-point-your-domain-to-the-load-balancer]
1. Go to **Load Balancers** and copy the **DNS name**
**Using Route 53:**
1. Go to **Route 53** → **Hosted zones** → your domain
2. Click **Create record**
3. Configure:
* **Record name:** `arc` (for `arc.yourdomain.com`)
* **Record type:** A
* **Alias:** Yes
* **Route traffic to:** Application Load Balancer
* **Region:** Your ALB's region
* **Load balancer:** Select your ALB
4. Click **Create records**
**Using External DNS:**
Create a CNAME record pointing to the ALB DNS name:
* **Name:** `arc`
* **Type:** CNAME
* **Value:** `arc-alb-xxxxx.us-east-1.elb.amazonaws.com`
CNAME records don't work for apex domains. Use a subdomain like `arc.yourdomain.com`.
### Step 7: verify security groups [#step-7-verify-security-groups]
Ensure you have two properly configured security groups:
**ALB Security Group:**
* Inbound: HTTPS (443) from `0.0.0.0/0` (or your allowed IP ranges)
* Outbound: HTTP (8000) to Arc Security Group
**Arc Security Group:**
* Inbound: HTTP (8000) from ALB Security Group only
* Outbound: All traffic (or restrict as needed)
### Step 8: verify target health [#step-8-verify-target-health]
1. Go to **Target Groups** → `arc-target-group`
2. Click **Targets** tab
3. Wait for status to change from "initial" to "healthy"
If unhealthy, check:
* Security group allows port 8000 from ALB
* Arc is running: `sudo systemctl status arc`
* Health check path is correct: `/health`
### Step 9: get your admin token [#step-9-get-your-admin-token]
SSH into your Arc instance through a bastion host or Session Manager:
```bash
# Using a bastion host (Ubuntu-based AMI)
ssh -i your-key.pem -J ubuntu@bastion-ip ubuntu@arc-private-ip
# Or use AWS Systems Manager Session Manager
aws ssm start-session --target i-your-instance-id
# Get the admin token
sudo journalctl -u arc | grep "Admin token"
# You'll see:
# Admin token: ark_abc123...xyz
```
### Step 10: verify deployment [#step-10-verify-deployment]
Test the health endpoint through your ALB:
```bash
curl https://arc.yourdomain.com/health
# Expected: {"status":"healthy"}
```
Test ingestion using MessagePack columnar format:
```bash
export ARC_URL="https://arc.yourdomain.com"
export ARC_TOKEN="your-token-here"
# Write test data
echo '{"m":"cpu","columns":{"time":[1734372000000],"host":["server01"],"usage":[95.0]}}' | \
python3 -c "import sys,msgpack,json; sys.stdout.buffer.write(msgpack.packb(json.load(sys.stdin)))" | \
curl -X POST "$ARC_URL/api/v1/write/msgpack" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/msgpack" \
-H "x-arc-database: default" \
--data-binary @-
# Query it back
curl -X POST "$ARC_URL/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql":"SELECT * FROM default.cpu","format":"json"}'
```
## Instance types [#instance-types]
| Use Case | Instance Type | vCPU | Memory | Notes |
| ----------------- | ------------- | ---- | ------ | ------------------------- |
| Testing | t3.large | 2 | 8 GB | Burstable, cost-effective |
| Small Production | m8a.xlarge | 4 | 16 GB | General purpose |
| Medium Production | m8a.2xlarge | 8 | 32 GB | Recommended |
| High Throughput | c7i.4xlarge | 16 | 32 GB | Compute optimized |
## Storage [#storage]
Arc stores data in Parquet files on the EBS volume attached to the instance.
**Recommendations:**
* Use `gp3` volumes for best price/performance
* Size based on your data retention needs
* Enable EBS encryption for data at rest
```bash
# Check disk usage
df -h /app/data
```
## Pricing [#pricing]
| Component | Cost | Notes |
| ------------------ | ---------------- | ------------------------------------------------------ |
| Arc Software | Free | AGPL-3.0 license |
| EC2 Instance | Variable | See [EC2 Pricing](https://aws.amazon.com/ec2/pricing/) |
| EBS Storage | \~$0.08/GB/month | gp3 pricing |
| ALB | \~$20/month | Plus data transfer |
| SSL Certificate | Free | AWS Certificate Manager |
| Enterprise Support | $500/month | Optional |
## Service management [#service-management]
Arc runs as a systemd service on the AMI.
```bash
# Check status
sudo systemctl status arc
# View logs
sudo journalctl -u arc -f
# Restart service
sudo systemctl restart arc
# Stop service
sudo systemctl stop arc
```
## Troubleshooting [#troubleshooting]
### Target shows unhealthy [#target-shows-unhealthy]
1. Check Arc is running:
```bash
sudo systemctl status arc
```
2. Check Arc is listening on port 8000:
```bash
sudo ss -tlnp | grep 8000
```
3. Test health endpoint locally:
```bash
curl http://localhost:8000/health
```
4. Check security group allows traffic from ALB
### 504 Gateway Timeout [#504-gateway-timeout]
The ALB can't reach the Arc instance. Check:
* Security group allows port 8000 from ALB
* Arc instance is in the correct subnet
* Target group has the correct port (8000)
### Can't find admin token [#cant-find-admin-token]
```bash
# Check all Arc logs
sudo journalctl -u arc | head -200
# Or search specifically
sudo journalctl -u arc | grep -i "admin\|token"
```
## Next steps [#next-steps]
* [Configure Telegraf integration](/arc-enterprise/integrations/telegraf/)
* [Set up Grafana dashboards](/arc-enterprise/integrations/grafana/)
* [Configure retention policies](/arc-enterprise/data-lifecycle/retention-policies/)
# Docker Installation (/arc-enterprise/installation/docker)
Install and run Arc using Docker for quick setup and isolated environments.
## Prerequisites [#prerequisites]
* Docker 20.10 or higher
* 4GB RAM minimum, 8GB+ recommended
## Quick start [#quick-start]
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
Verify it's running:
```bash
curl http://localhost:8000/health
```
## Get your admin token [#get-your-admin-token]
When Arc starts for the first time, it generates an admin token.
Copy this token immediately - you won't see it again!
```bash
docker logs arc 2>&1 | grep -i "admin"
```
You should see:
```text
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Initial admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save it:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
## Storage backends [#storage-backends]
**Local Filesystem** - Default, data stored in Docker volume.
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-e ARC_STORAGE_BACKEND=local \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
**Data locations:**
| Path | Description |
| ----------------------- | ------------- |
| `/app/data/arc/` | Parquet files |
| `/app/data/arc_auth.db` | Auth tokens |
**AWS S3** - Production cloud storage.
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-e ARC_STORAGE_BACKEND=s3 \
-e ARC_STORAGE_S3_BUCKET=arc-data \
-e ARC_STORAGE_S3_REGION=us-east-1 \
-e AWS_ACCESS_KEY_ID=your_key \
-e AWS_SECRET_ACCESS_KEY=your_secret \
ghcr.io/basekick-labs/arc:latest
```
On EC2, use IAM roles instead of access keys for better security.
**MinIO** - Self-hosted S3-compatible storage.
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-e ARC_STORAGE_BACKEND=minio \
-e ARC_STORAGE_S3_ENDPOINT=minio:9000 \
-e ARC_STORAGE_S3_BUCKET=arc \
-e ARC_STORAGE_S3_ACCESS_KEY=minioadmin \
-e ARC_STORAGE_S3_SECRET_KEY=minioadmin123 \
-e ARC_STORAGE_S3_USE_SSL=false \
ghcr.io/basekick-labs/arc:latest
```
**Azure Blob Storage** - For Azure deployments.
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-e ARC_STORAGE_BACKEND=azure \
-e ARC_STORAGE_AZURE_CONTAINER=arc-data \
-e ARC_STORAGE_AZURE_ACCOUNT_NAME=your_account \
-e ARC_STORAGE_AZURE_ACCOUNT_KEY=your_key \
ghcr.io/basekick-labs/arc:latest
```
## Configuration [#configuration]
### Environment variables [#environment-variables]
Common configuration options:
| Variable | Default | Description |
| -------------------------- | --------- | --------------------------------------------------------------------- |
| `ARC_SERVER_PORT` | `8000` | HTTP port |
| `ARC_STORAGE_BACKEND` | `local` | Storage: `local`, `s3`, `minio`, `azure` |
| `ARC_LOG_LEVEL` | `info` | Logging: `debug`, `info`, `warn`, `error` |
| `ARC_AUTH_ENABLED` | `true` | Enable authentication |
| `ARC_AUTH_BOOTSTRAP_TOKEN` | *(unset)* | Pre-set admin token value on first run (v26.04.1+) |
| `ARC_AUTH_FORCE_BOOTSTRAP` | `false` | Add a recovery admin token without removing existing ones (v26.04.1+) |
| `ARC_COMPACTION_ENABLED` | `true` | Enable auto-compaction |
| `ARC_WAL_ENABLED` | `false` | Enable WAL for durability |
### Custom configuration file [#custom-configuration-file]
Mount a custom `arc.toml`:
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
-v /path/to/arc.toml:/app/arc.toml \
ghcr.io/basekick-labs/arc:latest
```
## Container management [#container-management]
### View logs [#view-logs]
```bash
docker logs -f arc # Follow logs
docker logs --tail=100 arc # Last 100 lines
```
### Start/stop/restart [#startstoprestart]
```bash
docker start arc
docker stop arc
docker restart arc
```
### Update Arc [#update-arc]
```bash
docker stop arc && docker rm arc
docker pull ghcr.io/basekick-labs/arc:latest
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
## Production deployment [#production-deployment]
### Pin version + resource limits [#pin-version--resource-limits]
```bash
docker run -d \
--name arc \
-p 8000:8000 \
-v arc-data:/app/data \
--memory="8g" \
--cpus="4" \
--restart unless-stopped \
ghcr.io/basekick-labs/arc:latest
```
### Health check [#health-check]
```bash
docker ps --filter "name=arc" --filter "health=healthy"
```
## Docker Compose [#docker-compose]
```yaml
version: '3.8'
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
container_name: arc
ports:
- "8000:8000"
environment:
- ARC_STORAGE_BACKEND=local
- ARC_AUTH_ENABLED=true
- ARC_COMPACTION_ENABLED=true
volumes:
- arc-data:/app/data
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
volumes:
arc-data:
```
```yaml
version: '3.8'
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
container_name: arc
ports:
- "8000:8000"
environment:
- ARC_STORAGE_BACKEND=minio
- ARC_STORAGE_S3_ENDPOINT=minio:9000
- ARC_STORAGE_S3_BUCKET=arc
- ARC_STORAGE_S3_ACCESS_KEY=minioadmin
- ARC_STORAGE_S3_SECRET_KEY=minioadmin123
- ARC_STORAGE_S3_USE_SSL=false
depends_on:
- minio
restart: unless-stopped
minio:
image: minio/minio:latest
container_name: minio
ports:
- "9000:9000"
- "9001:9001"
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin123
command: server /data --console-address ":9001"
volumes:
- minio-data:/data
volumes:
minio-data:
```
```yaml
version: '3.8'
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
container_name: arc
ports:
- "8000:8000"
environment:
- ARC_STORAGE_BACKEND=local
- ARC_AUTH_ENABLED=true
- ARC_COMPACTION_ENABLED=true
- ARC_WAL_ENABLED=true
- ARC_WAL_SYNC_MODE=fdatasync
- ARC_LOG_LEVEL=info
- ARC_LOG_FORMAT=json
volumes:
- arc-data:/app/data
- arc-wal:/app/data/wal
deploy:
resources:
limits:
memory: 8G
cpus: '4'
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
volumes:
arc-data:
arc-wal:
```
## Troubleshooting [#troubleshooting]
### Container won't start [#container-wont-start]
```bash
# Check logs
docker logs arc
# Check port availability
sudo lsof -i :8000
# Check container status
docker ps -a
```
### Permission errors [#permission-errors]
```bash
# Remove and recreate volume
docker stop arc && docker rm arc
docker volume rm arc-data
# Restart with docker run command
```
### Out of memory [#out-of-memory]
```bash
# Check memory usage
docker stats arc
# Restart with memory limit
docker run -d --name arc --memory="4g" ...
```
### Can't find admin token [#cant-find-admin-token]
```bash
docker logs arc 2>&1 | grep -i "admin"
docker logs arc | head -100
```
## Next steps [#next-steps]
* [Write your first data](/arc-enterprise/getting-started/#write-data)
* [Configure storage backends](/arc-enterprise/configuration/overview/)
* [Deploy on Kubernetes](/arc-enterprise/installation/kubernetes/)
# Installation (/arc-enterprise/installation)
Arc Enterprise ships as the same binary as Arc OSS. Every install target below is the OSS install plus a license key — set `ARC_LICENSE_KEY`, or the `[license]` section of `arc.toml`, and the licensed features activate on startup.
Which target you pick mostly determines how you will run a cluster later. Docker and native installs place one node per host and you assign roles yourself; the Helm chart models writer, reader, and compactor as separate workloads from the start.
After installation, continue with [Configuration](/arc-enterprise/configuration/) to set the storage backend and cluster role.
# Kubernetes Installation (/arc-enterprise/installation/kubernetes)
Deploy Arc on Kubernetes using Helm for production-grade analytical data management.
## Prerequisites [#prerequisites]
* Kubernetes 1.24+
* Helm 3.0+
* `kubectl` configured to access your cluster
* Persistent storage (for local storage backend)
## Quick start [#quick-start]
```bash
# Install Arc
helm install arc https://github.com/basekick-labs/arc/releases/latest/download/arc-26.09.1.tgz
# Port forward to access locally
kubectl port-forward svc/arc 8000:8000
# Verify installation
curl http://localhost:8000/health
```
## Get your admin token [#get-your-admin-token]
```bash
# Get the pod name
kubectl get pods -l app=arc
# View logs to find admin token
kubectl logs -l app=arc | grep -i "admin"
```
You should see:
```text
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Initial admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Copy this token immediately - you won't see it again!
## Installation methods [#installation-methods]
```bash
helm install arc https://github.com/basekick-labs/arc/releases/latest/download/arc-26.09.1.tgz
```
```bash
# Download chart
helm pull https://github.com/basekick-labs/arc/releases/latest/download/arc-26.09.1.tgz
tar -xzf arc-26.09.1.tgz
# Edit values
vim arc/values.yaml
# Install with custom values
helm install arc ./arc -f custom-values.yaml
```
```bash
# Create namespace
kubectl create namespace arc
# Install in namespace
helm install arc \
https://github.com/basekick-labs/arc/releases/latest/download/arc-26.09.1.tgz \
--namespace arc
```
## Storage backends [#storage-backends]
**Local storage** - each Arc node keeps its own PersistentVolume and the cluster
stays in sync via peer-to-peer replication (Pattern 1). No object storage is
required. Set `storage.mode: local` and size each role's PVC.
```yaml
# values.yaml
storage:
mode: local
local:
storageClass: "" # default storage class
minio:
enabled: false # no object storage in local mode
writer:
replicas: 1
persistence:
size: 50Gi # local Parquet + WAL
reader:
replicas: 2
persistence:
size: 50Gi # reader needs a full data replica
compactor:
enabled: true
replicas: 1
persistence:
size: 50Gi
```
```bash
helm install arc-ent helm/arc-enterprise -f values.yaml \
--set license.key=ARC-ENT-... \
--set cluster.sharedSecret.value=$(openssl rand -hex 32)
```
**AWS S3** - Recommended for EKS. Set `storage.mode: shared` and point the
shared block at your bucket. Authenticate with IRSA (preferred) or static keys.
**IRSA (recommended) — no static keys.** Set `credentials.useIRSA: true` so the
chart omits the access/secret-key env vars and Arc authenticates via the AWS
credential chain (the pod's IAM role), then attach the role to the
ServiceAccount:
```yaml
# values.yaml
storage:
mode: shared
shared:
external: true # use your own S3 (not bundled MinIO)
bucket: arc-production
region: us-east-1
endpoint: https://s3.us-east-1.amazonaws.com
useSSL: true
usePathStyle: false
credentials:
useIRSA: true # authenticate via the pod IAM role
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/arc-s3
minio:
enabled: false # don't deploy bundled MinIO
```
```bash
helm install arc ./arc -f values.yaml
```
Primary-S3 **query** reads via the credential chain require Arc **26.06.2** or
later. On 26.06.1 IRSA authenticates writes but not query reads — pin
`image.tag` to 26.06.2 or later (26.09.1 is current) for full IRSA support. The IAM role's
trust policy must permit the cluster OIDC provider + this ServiceAccount, and
the role needs `s3:GetObject`/`PutObject`/`ListBucket` on the bucket. When the
chart creates the ServiceAccount, the install fails if the
`eks.amazonaws.com/role-arn` annotation is missing.
**Static keys** (when IRSA is not available). Provide them inline or, better, via
an existing Secret with `access-key` / `secret-key` entries:
```yaml
storage:
mode: shared
shared:
external: true
bucket: arc-production
region: us-east-1
endpoint: https://s3.us-east-1.amazonaws.com
useSSL: true
credentials:
existingSecret: arc-s3-credentials # keys: access-key, secret-key
# or inline: accessKey / secretKey
minio:
enabled: false
```
**MinIO** - Bundled S3-compatible storage (default for `storage.mode: shared`).
```yaml
# values.yaml
storage:
mode: shared
shared:
external: false # bundled MinIO (default)
bucket: arc-data
usePathStyle: true
useSSL: false
minio:
enabled: true
credentials:
rootUser: arcminio
rootPassword: # or set credentials.existingSecret
```
```bash
helm install arc ./arc -f values.yaml
```
`useIRSA` is only valid with external S3 (`external: true`). The bundled MinIO
needs static credentials — the chart rejects `useIRSA: true` with bundled MinIO.
The chart emits S3 config only (`ARC_STORAGE_BACKEND=s3`); point `endpoint` at
any S3-compatible service. For local-disk + peer replication instead of shared
object storage, use `storage.mode: local` (see
[Deployment Patterns](/arc-enterprise/configuration/deployment-patterns/)).
## Configuration profiles [#configuration-profiles]
The Enterprise chart ships two ready-to-deploy presets in the chart root:
`values-shared-storage.yaml` (shared object storage via bundled MinIO) and
`values-local-storage.yaml` (per-node PVCs + peer replication). Both require a
license key and a cluster shared secret.
Shared object storage with the bundled MinIO — the recommended cloud-native
layout. All writer pods accept writes concurrently (Pattern 2 multi-writer) and
read/write the same bucket; the bucket is the durability layer, so the writer
and compactor PVCs hold only WAL/scratch.
```yaml
# values-shared-storage.yaml (excerpt)
storage:
mode: shared
shared:
external: false # bundled MinIO
bucket: arc-data
usePathStyle: true
useSSL: false
minio:
enabled: true
replicas: 1
persistence:
size: 100Gi
writer:
replicas: 1 # 1 = single writer; 3 = HA (2 is refused)
persistence:
size: 20Gi # WAL only; bucket holds Parquet
reader:
replicas: 2 # emptyDir in shared mode (no PVC)
compactor:
enabled: true
replicas: 1
persistence:
size: 20Gi # scratch only; bucket holds Parquet
```
```bash
helm install arc-ent helm/arc-enterprise \
-f helm/arc-enterprise/values-shared-storage.yaml \
--set license.key=ARC-ENT-... \
--set cluster.sharedSecret.value=$(openssl rand -hex 32) \
--set minio.credentials.rootUser=arcminio \
--set minio.credentials.rootPassword=$(openssl rand -hex 32)
```
`license.key`, `cluster.sharedSecret.value`, and (for bundled MinIO)
`minio.credentials.rootUser` / `minio.credentials.rootPassword` are mandatory —
the chart refuses to install if any of them is empty.
Per-node PersistentVolumes with peer-to-peer replication (Pattern 1). Each node
keeps its own copy of the Parquet files; the Raft-backed cluster manifest is the
source of truth. Use this for bare metal, VMs, edge, or anywhere shared object
storage is unavailable. No MinIO is deployed.
```yaml
# values-local-storage.yaml (excerpt)
storage:
mode: local
minio:
enabled: false
writer:
replicas: 1
persistence:
size: 50Gi # local Parquet + WAL
reader:
replicas: 2
persistence:
size: 50Gi # reader needs a full data replica
compactor:
enabled: true
replicas: 1
persistence:
size: 50Gi # local Parquet + scratch
```
```bash
helm install arc-ent helm/arc-enterprise \
-f helm/arc-enterprise/values-local-storage.yaml \
--set license.key=ARC-ENT-... \
--set cluster.sharedSecret.value=$(openssl rand -hex 32)
```
`cluster.replication.*` (pull workers, fetch/serve timeouts, startup catch-up)
applies **only** in local mode. In shared mode the bucket is the durability
layer and peer replication is disabled.
Point shared mode at your own S3 (or any S3-compatible service) instead of the
bundled MinIO. See the [Storage Backends](#storage-backends) tabs above for the
full IRSA vs static-key options.
```yaml
storage:
mode: shared
shared:
external: true # use your own S3 (not bundled MinIO)
bucket: arc-production
region: us-east-1
endpoint: https://s3.us-east-1.amazonaws.com
useSSL: true
usePathStyle: false
credentials:
useIRSA: true # authenticate via the pod IAM role
minio:
enabled: false # don't deploy bundled MinIO
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/arc-s3
```
```bash
helm install arc-ent helm/arc-enterprise -f values.yaml \
--set license.key=ARC-ENT-... \
--set cluster.sharedSecret.value=$(openssl rand -hex 32)
```
## Helm values reference [#helm-values-reference]
### Image & service account [#image--service-account]
```yaml
# Container image (tag defaults to the chart appVersion)
image:
repository: ghcr.io/basekick-labs/arc
tag: "" # set "26.09.1" (or >=26.06.2) for full IRSA query-read support
pullPolicy: IfNotPresent
imagePullSecrets: []
# ServiceAccount shared by all Arc pods (writer/reader/compactor).
# Attach an AWS IAM role via the role-arn annotation for IRSA.
serviceAccount:
create: false # true = chart creates the ServiceAccount
name: ""
annotations: {} # eks.amazonaws.com/role-arn: arn:aws:iam::...:role/arc-s3
```
### License & authentication [#license--authentication]
```yaml
license:
existingSecret: "" # Secret with key "license-key"
key: "" # your ARC-ENT-... license key (REQUIRED)
auth:
bootstrapToken:
existingSecret: "" # Secret with key "bootstrap-token"
value: "" # leave empty to let the Raft leader generate one
```
### Cluster [#cluster]
```yaml
cluster:
name: arc-prod
# HMAC peer authentication — REQUIRED (chart refuses to install if empty).
sharedSecret:
existingSecret: "" # Secret with key "shared-secret"
value: "" # REQUIRED — e.g. $(openssl rand -hex 32)
# TLS between cluster nodes (recommended for multi-writer / production).
tls:
enabled: false
existingSecret: "" # tls.crt, tls.key (and optionally ca.crt)
# Single switch governing writer + compactor failover.
failover:
enabled: true
# Peer replication tuning — consulted ONLY when storage.mode=local.
replication:
pullWorkers: 4
fetchTimeoutMs: 60000
serveTimeoutMs: 120000
catchup:
enabled: true
barrierTimeoutMs: 10000
```
### Storage [#storage]
The chart supports two modes via `storage.mode`. It emits S3 config only
(`ARC_STORAGE_BACKEND=s3`); there is no Azure path.
```yaml
storage:
mode: shared # "shared" or "local"
# Shared mode — S3-compatible object storage (bundled MinIO or external S3).
shared:
external: false # false = bundled MinIO; true = your own S3
bucket: arc-data
region: us-east-1
endpoint: "" # auto-set for bundled MinIO; set for external S3
prefix: "" # optional key prefix (multi-tenant bucket sharing)
usePathStyle: true # true for MinIO and many S3-compatible services
useSSL: false # true for production S3
credentials:
useIRSA: false # true = AWS credential chain (external S3 only)
existingSecret: "" # keys: access-key, secret-key (ignored if useIRSA)
accessKey: ""
secretKey: ""
# Local mode — per-node PVCs + peer replication.
local:
storageClass: "" # fallback for roles that don't set their own
```
### Bundled MinIO [#bundled-minio]
Rendered only when `storage.mode=shared` and `storage.shared.external=false`.
```yaml
minio:
enabled: true
replicas: 1
persistence:
size: 100Gi
storageClass: ""
credentials:
existingSecret: "" # keys: root-user, root-password
rootUser: "" # REQUIRED (no weak defaults)
rootPassword: "" # REQUIRED
```
### Roles (writer / reader / compactor) [#roles-writer--reader--compactor]
Each role is a StatefulSet with its own replica count, resources, persistence,
and scheduling.
```yaml
writer:
replicas: 1 # 3 = HA; 2 is REFUSED (no failure tolerance)
resources:
requests: { cpu: 500m, memory: 1Gi }
limits: { cpu: 4000m, memory: 8Gi }
persistence:
size: 20Gi # shared mode: WAL only; local mode: WAL + Parquet
storageClass: ""
wal:
enabled: true
syncMode: fdatasync # fdatasync | fsync | async
nodeSelector: {}
tolerations: []
affinity: {}
extraEnv: [] # extra env vars passed through to Arc
reader:
replicas: 2 # scale horizontally for query throughput
resources:
requests: { cpu: 500m, memory: 1Gi }
limits: { cpu: 4000m, memory: 8Gi }
persistence:
size: 50Gi # shared mode: emptyDir (no PVC); local mode: PVC
storageClass: ""
nodeSelector: {}
tolerations: []
affinity: {}
extraEnv: []
compactor:
enabled: true
replicas: 1 # exactly one active compactor — failover replaces it
resources:
requests: { cpu: 1000m, memory: 4Gi }
limits: { cpu: 4000m, memory: 16Gi }
persistence:
size: 50Gi # scratch space for compaction jobs
storageClass: ""
nodeSelector: {}
tolerations: []
affinity: {}
extraEnv: []
```
### Services & telemetry [#services--telemetry]
```yaml
service:
writer:
type: ClusterIP # backend for the L7 layer below — not a client-facing endpoint
port: 8000
annotations: {}
reader:
type: ClusterIP # expose via Ingress / annotated LoadBalancer
port: 8000
annotations: {}
# Disable for air-gapped / defense deployments.
telemetry:
enabled: true
```
A ClusterIP Service balances per TCP connection, and keep-alive clients pin to
whichever writer pod they first dialed — one writer takes nearly all traffic
while the rest idle. Front the writer Service with an ingress controller,
Envoy/HAProxy, or a cloud ALB so each request is balanced independently. See
[Clustering — multi-writer](/arc-enterprise/configuration/clustering/#pattern-2--shared-object-storage-multi-writer).
## Operations [#operations]
### View logs [#view-logs]
```bash
# Follow logs
kubectl logs -l app=arc -f
# Last 100 lines
kubectl logs -l app=arc --tail=100
# Logs from last hour
kubectl logs -l app=arc --since=1h
```
### Check status [#check-status]
```bash
# Pod status
kubectl get pods -l app=arc
# Describe pod
kubectl describe pod -l app=arc
# Check events
kubectl get events --field-selector involvedObject.name=arc-0
```
### Scale (restart) [#scale-restart]
The Enterprise chart deploys each role as a StatefulSet (`writer`, `reader`,
`compactor`).
```bash
# Restart a role
kubectl rollout restart statefulset arc-ent-writer
# Or delete a pod (will be recreated)
kubectl delete pod -l app.kubernetes.io/component=writer
```
### Port forward [#port-forward]
```bash
kubectl port-forward svc/arc 8000:8000
```
### Access shell [#access-shell]
```bash
kubectl exec -it $(kubectl get pod -l app=arc -o jsonpath='{.items[0].metadata.name}') -- /bin/sh
```
## Upgrade [#upgrade]
```bash
# Upgrade to new version
helm upgrade arc https://github.com/basekick-labs/arc/releases/latest/download/arc-26.09.1.tgz
# With custom values
helm upgrade arc ./arc -f values-prod.yaml
```
## Uninstall [#uninstall]
```bash
# Uninstall Arc
helm uninstall arc
# Delete PVCs (optional - removes all data!)
kubectl delete pvc -l app=arc
# Delete namespace (if dedicated)
kubectl delete namespace arc
```
## Monitoring [#monitoring]
### Prometheus metrics [#prometheus-metrics]
Arc exposes Prometheus metrics at `/metrics`:
```yaml
# ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: arc
spec:
selector:
matchLabels:
app: arc
endpoints:
- port: http
path: /metrics
interval: 30s
```
### Readiness/liveness probes [#readinessliveness-probes]
```yaml
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 30
readinessProbe:
httpGet:
path: /ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
```
## Troubleshooting [#troubleshooting]
### Pod won't start [#pod-wont-start]
```bash
# Check pod status
kubectl describe pod -l app=arc
# Check events
kubectl get events --sort-by='.lastTimestamp'
# Common issues:
# - ImagePullBackOff: Check image name/tag
# - Pending: Check PVC status, node resources
# - CrashLoopBackOff: Check logs
```
### Storage issues [#storage-issues]
```bash
# Check PVC status
kubectl get pvc -l app=arc
# Check PV
kubectl get pv
# Describe PVC for errors
kubectl describe pvc -l app=arc
```
### Connection issues [#connection-issues]
```bash
# Check service
kubectl get svc arc
# Test from within cluster
kubectl run curl --image=curlimages/curl -it --rm -- curl http://arc:8000/health
```
### Memory issues [#memory-issues]
```bash
# Check resource usage
kubectl top pod -l app=arc
# Increase limits in values.yaml
resources:
limits:
memory: "16Gi"
```
## High availability (EKS) [#high-availability-eks]
In **shared mode** the Enterprise chart runs Arc as a Pattern 2 multi-writer
cluster: every writer pod accepts writes concurrently behind a Kubernetes
Service, and each writer PUTs to the same S3 bucket independently. Singleton
background tasks (retention, continuous queries, deletes) run on whichever pod
is the cluster Raft leader, so they execute exactly once.
Failover is Service-based: clients always talk to the writer Service, which
load-balances across healthy pods. If a writer pod dies, the Service stops
routing to it and Raft re-elects a leader for the singleton tasks — no client
URL changes.
Set `writer.replicas` to control the topology:
| `writer.replicas` | Behaviour |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `1` | Single writer (lowest cost). The Service still fronts it, so client URLs are identical to the multi-writer case. No failure tolerance. |
| `3` | HA + horizontal scale. Raft quorum tolerates one pod failure; writes round-robin across all healthy pods. |
| `2` | **Refused by chart validation** — a quorum of 2 stalls Raft writes on any single-pod loss, so it offers no failure tolerance over `1`. |
In shared mode the reader uses `emptyDir` (no PVC — the bucket holds Parquet),
and writer/compactor PVCs are WAL/scratch only (\~20Gi). In **local mode**
(Pattern 1), HA instead relies on per-node PVCs and peer replication, and the
reader needs a full data replica (\~50Gi); `cluster.replication.*` tuning applies
only in this mode.
For the full topology comparison see
[Deployment Patterns](/arc-enterprise/configuration/deployment-patterns/), and for the cluster
shared secret and inter-node TLS see
[Cluster Security](/arc-enterprise/security/cluster-security/).
## Next steps [#next-steps]
* [Write your first data](/arc-enterprise/getting-started/#write-data)
* [Configure storage backends](/arc-enterprise/configuration/overview/)
* [Set up monitoring](/arc-enterprise/operations/telemetry/)
* [Enable WAL for durability](/arc-enterprise/advanced/wal/)
# Native Installation (/arc-enterprise/installation/native)
Install Arc directly on Linux using native packages (.deb, .rpm) or build from source.
Current release: ****
## Prerequisites [#prerequisites]
* Linux (x86\_64 or ARM64)
* 4GB RAM minimum, 8GB+ recommended
* systemd (for service management)
## Quick install [#quick-install]
The following commands automatically fetch and install the latest Arc release.
**x86\_64 (AMD/Intel):**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc_${LATEST_VERSION}_amd64.deb
sudo dpkg -i arc_${LATEST_VERSION}_amd64.deb
sudo systemctl enable arc && sudo systemctl start arc
curl http://localhost:8000/health
```
**ARM64:**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc_${LATEST_VERSION}_arm64.deb
sudo dpkg -i arc_${LATEST_VERSION}_arm64.deb
sudo systemctl enable arc && sudo systemctl start arc
```
**x86\_64 (AMD/Intel):**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1.x86_64.rpm
sudo rpm -i arc-${LATEST_VERSION}-1.x86_64.rpm
sudo systemctl enable arc && sudo systemctl start arc
curl http://localhost:8000/health
```
**ARM64:**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1.aarch64.rpm
sudo rpm -i arc-${LATEST_VERSION}-1.aarch64.rpm
sudo systemctl enable arc && sudo systemctl start arc
```
Works on Arch Linux and Arch-based distros such as Omarchy.
**x86\_64 (AMD/Intel):**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1-x86_64.pkg.tar.zst
sudo pacman -U arc-${LATEST_VERSION}-1-x86_64.pkg.tar.zst
sudo systemctl enable arc && sudo systemctl start arc
curl http://localhost:8000/health
```
**ARM64:**
```bash
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1-aarch64.pkg.tar.zst
sudo pacman -U arc-${LATEST_VERSION}-1-aarch64.pkg.tar.zst
sudo systemctl enable arc && sudo systemctl start arc
```
Prerequisites: Go 1.25+, Git, Make
```bash
# Clone and build
git clone https://github.com/basekick-labs/arc.git
cd arc
make build
# Run
./arc
```
Development commands:
```bash
make deps # Install dependencies
make build # Build binary
make run # Run without building
make test # Run tests
make test-coverage # Run tests with coverage
make bench # Run benchmarks
make lint # Run linter
make clean # Clean build artifacts
```
## Get your admin token [#get-your-admin-token]
When Arc starts for the first time, it generates an admin token.
Copy this token immediately - you won't see it again!
```bash
sudo journalctl -u arc | grep -i "admin"
```
You should see:
```text
======================================================================
FIRST RUN - INITIAL ADMIN TOKEN GENERATED
======================================================================
Initial admin API token: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
======================================================================
```
Save it:
```bash
export ARC_TOKEN="arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
```
## Storage backends [#storage-backends]
**Local Filesystem** - Default, data stored on disk.
Edit `/etc/arc/arc.toml`:
```toml
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
```
Or via environment:
```bash
ARC_STORAGE_BACKEND=local
ARC_STORAGE_LOCAL_PATH=/var/lib/arc/data
```
**AWS S3** - Production cloud storage.
Edit `/etc/arc/arc.toml`:
```toml
[storage]
backend = "s3"
s3_bucket = "arc-production"
s3_region = "us-east-1"
# Use IAM roles or environment variables for credentials
```
Environment variables:
```bash
ARC_STORAGE_BACKEND=s3
ARC_STORAGE_S3_BUCKET=arc-data
ARC_STORAGE_S3_REGION=us-east-1
AWS_ACCESS_KEY_ID=your_key
AWS_SECRET_ACCESS_KEY=your_secret
```
On EC2, use IAM instance profiles for automatic credential management.
**MinIO** - Self-hosted S3-compatible storage.
Edit `/etc/arc/arc.toml`:
```toml
[storage]
backend = "minio"
s3_bucket = "arc"
s3_endpoint = "minio.local:9000"
s3_access_key = "minioadmin"
s3_secret_key = "minioadmin123"
s3_use_ssl = false
s3_path_style = true
```
**Azure Blob Storage** - For Azure deployments.
Edit `/etc/arc/arc.toml`:
```toml
[storage]
backend = "azure"
azure_container = "arc-data"
azure_account_name = "your_account"
azure_account_key = "your_key"
```
## Service management [#service-management]
### Start/stop/restart [#startstoprestart]
```bash
sudo systemctl start arc # Start
sudo systemctl stop arc # Stop
sudo systemctl restart arc # Restart
sudo systemctl status arc # Status
```
### View logs [#view-logs]
```bash
sudo journalctl -u arc -f # Follow logs
sudo journalctl -u arc -n 100 # Last 100 lines
sudo journalctl -u arc --since "1 hour ago"
```
### Enable/disable auto-start [#enabledisable-auto-start]
```bash
sudo systemctl enable arc # Enable on boot
sudo systemctl disable arc # Disable on boot
```
## Configuration [#configuration]
Configuration file: `/etc/arc/arc.toml`
```bash
sudo nano /etc/arc/arc.toml
sudo systemctl restart arc
```
### Common options [#common-options]
```toml
[server]
port = 8000
[storage]
backend = "local"
local_path = "/var/lib/arc/data"
[auth]
enabled = true
[compaction]
enabled = true
hourly_enabled = true
daily_enabled = true
[wal]
enabled = false # Enable for zero data loss
sync_mode = "fdatasync"
[log]
level = "info"
format = "json"
```
See [Configuration Overview](/arc-enterprise/configuration/overview/) for all options.
## Data directory [#data-directory]
| Installation Type | Default Data Directory |
| ----------------- | ---------------------- |
| Package install | `/var/lib/arc/data` |
| Source build | `./data/arc` |
## Updating Arc [#updating-arc]
```bash
# Automatic update to latest version
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc_${LATEST_VERSION}_amd64.deb
sudo dpkg -i arc_${LATEST_VERSION}_amd64.deb
sudo systemctl restart arc
```
```bash
# Automatic update to latest version
LATEST_VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/arc/releases/download/v${LATEST_VERSION}/arc-${LATEST_VERSION}-1.x86_64.rpm
sudo rpm -U arc-${LATEST_VERSION}-1.x86_64.rpm
sudo systemctl restart arc
```
```bash
cd arc
git pull
make build
# Restart Arc manually
```
## Uninstalling [#uninstalling]
```bash
sudo systemctl stop arc
sudo dpkg -r arc
# Optional: Remove data
sudo rm -rf /var/lib/arc /etc/arc
```
```bash
sudo systemctl stop arc
sudo rpm -e arc
# Optional: Remove data
sudo rm -rf /var/lib/arc /etc/arc
```
## Troubleshooting [#troubleshooting]
### Arc won't start [#arc-wont-start]
```bash
# Check logs
sudo journalctl -u arc -n 50
# Check port availability
sudo lsof -i :8000
```
### Permission errors [#permission-errors]
```bash
sudo mkdir -p /var/lib/arc/data
sudo chown -R arc:arc /var/lib/arc
```
### Memory issues [#memory-issues]
Override in `/etc/arc/arc.toml`:
```toml
[database]
memory_limit = "4GB"
max_connections = 16
thread_count = 8
```
## Next steps [#next-steps]
* [Write your first data](/arc-enterprise/getting-started/#write-data)
* [Configure storage backends](/arc-enterprise/configuration/overview/)
* [Deploy on Kubernetes](/arc-enterprise/installation/kubernetes/)
* [Set up compaction](/arc-enterprise/advanced/compaction/)
# Grafana Integration (/arc-enterprise/integrations/grafana)
Connect Arc to Grafana for real-time monitoring, alerting, and beautiful visualizations using the Arc datasource plugin.
## Overview [#overview]
The Arc datasource plugin for Grafana provides:
* **Apache Arrow Protocol**: High-performance columnar data transfer
* **Native SQL Support**: Full analytical SQL with syntax highlighting
* **Template Variables**: Dynamic dashboards with filters
* **Alerting**: Built-in alert rule support
* **Multi-database**: Query across different Arc databases
* **Real-time Dashboards**: Sub-second query performance
## Installation [#installation]
### From Grafana plugin catalog [#from-grafana-plugin-catalog]
1. In Grafana, go to **Configuration** → **Plugins**
2. Search for **Arc**
3. Click **Install**
4. Restart Grafana if prompted
### From release [#from-release]
```bash
# Grab the latest release tag, then download that release's plugin zip
VERSION=$(curl -s https://api.github.com/repos/basekick-labs/grafana-arc-datasource/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget https://github.com/basekick-labs/grafana-arc-datasource/releases/download/v${VERSION}/basekick-arc-datasource-${VERSION}.zip
# Extract to Grafana plugins directory
unzip basekick-arc-datasource-${VERSION}.zip -d /var/lib/grafana/plugins/
# Restart Grafana
systemctl restart grafana-server
```
### From source [#from-source]
```bash
# Clone repository
git clone https://github.com/basekick-labs/grafana-arc-datasource
cd grafana-arc-datasource
# Install dependencies
npm install
# Build plugin
npm run build
# Build backend
mage -v
# Install to Grafana
cp -r dist /var/lib/grafana/plugins/basekick-arc-datasource
systemctl restart grafana-server
```
## Configuration [#configuration]
### 1. Add data source [#1-add-data-source]
1. In Grafana, go to **Configuration** → **Data sources**
2. Click **Add data source**
3. Search for and select **Arc**
4. Configure connection settings
### 2. Connection settings [#2-connection-settings]
| Setting | Description | Required | Default |
| ------------- | ---------------------------- | -------- | ----------------------- |
| **URL** | Arc API endpoint | Yes | `http://localhost:8000` |
| **API Key** | Authentication token | Yes | - |
| **Database** | Default database name | No | `default` |
| **Timeout** | Query timeout in seconds | No | `30` |
| **Use Arrow** | Enable Apache Arrow protocol | No | `true` |
### 3. Example configuration [#3-example-configuration]
```yaml
URL: http://localhost:8000
API Key: arc_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Database: prod
Timeout: 30
```
Click **Save & Test** to verify the connection.
### 4. Get your API token [#4-get-your-api-token]
```bash
# Docker - check logs for admin token
docker logs 2>&1 | grep "Admin token"
# Or create a new token specifically for Grafana
curl -X POST http://localhost:8000/api/v1/auth/tokens \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "grafana-datasource",
"description": "Grafana datasource access"
}'
```
## Creating queries [#creating-queries]
### Query editor [#query-editor]
The Arc datasource provides a SQL query editor with:
* Syntax highlighting
* Auto-completion
* Time range macros
* Multi-database support
### Basic query example [#basic-query-example]
**CPU Usage:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(usage_idle) * -1 + 100 AS cpu_usage,
host
FROM prod.cpu
WHERE cpu = 'cpu-total'
AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
### Time macros [#time-macros]
Grafana provides powerful time macros for dynamic queries:
| Macro | Description | Example |
| --------------------------- | -------------------------- | ------------------------------------------- |
| `$__timeFilter(columnName)` | Complete time range filter | `WHERE $__timeFilter(time)` |
| `$__timeFrom()` | Start of time range | `time >= $__timeFrom()` |
| `$__timeTo()` | End of time range | `time < $__timeTo()` |
| `$__interval` | Auto-calculated interval | `time_bucket(INTERVAL '$__interval', time)` |
**How macros expand:**
```sql
-- Your query
WHERE $__timeFilter(time)
-- Expands to
WHERE time >= '2025-01-17 10:00:00' AND time < '2025-01-17 11:00:00'
```
### Example queries [#example-queries]
**Memory Usage:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(used_percent) AS memory_used,
host
FROM prod.mem
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
**Network Traffic (bytes to bits):**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(bytes_recv) * 8 AS bits_in,
AVG(bytes_sent) * 8 AS bits_out,
host,
interface
FROM prod.net
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host, interface
ORDER BY time ASC
```
**Disk I/O:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(read_bytes) AS disk_read,
AVG(write_bytes) AS disk_write,
host
FROM prod.diskio
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
## Template variables [#template-variables]
Create dynamic dashboards with variables that filter your data.
### Creating variables [#creating-variables]
1. Go to **Dashboard settings** → **Variables**
2. Click **Add variable**
3. Configure variable settings
### Variable examples [#variable-examples]
**Host Variable:**
```sql
SELECT DISTINCT host FROM prod.cpu ORDER BY host
```
**Interface Variable:**
```sql
SELECT DISTINCT interface FROM prod.net ORDER BY interface
```
**Database Variable:**
```sql
SELECT DISTINCT schema_name FROM information_schema.schemata
WHERE schema_name NOT IN ('information_schema', 'pg_catalog')
ORDER BY schema_name
```
### Using variables in queries [#using-variables-in-queries]
Reference variables with `$variable` syntax:
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(usage_idle) * -1 + 100 AS cpu_usage
FROM $database.cpu
WHERE host = '$server'
AND cpu = 'cpu-total'
AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time)
ORDER BY time ASC
```
### Multi-select variables [#multi-select-variables]
Enable **Multi-value** in variable settings, then use `IN`:
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(usage_idle) * -1 + 100 AS cpu_usage,
host
FROM prod.cpu
WHERE host IN ($hosts) -- Multi-select variable
AND cpu = 'cpu-total'
AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
## Alerting [#alerting]
The Arc datasource fully supports Grafana alerting.
### Creating alert rules [#creating-alert-rules]
1. Open a panel with an Arc query
2. Go to **Alert** tab
3. Click **Create alert rule from this panel**
4. Configure alert conditions
### Example alert query [#example-alert-query]
**High CPU Usage (> 80%):**
```sql
SELECT
time,
100 - usage_idle AS cpu_usage,
host
FROM prod.cpu
WHERE cpu = 'cpu-total'
AND time >= NOW() - INTERVAL '5 minutes'
ORDER BY time ASC
```
**Alert Condition:**
* `WHEN avg() OF query(A, 5m, now) IS ABOVE 80`
### Example alert: Memory usage [#example-alert-memory-usage]
**Query:**
```sql
SELECT
time,
used_percent AS memory_used,
host
FROM prod.mem
WHERE time >= NOW() - INTERVAL '5 minutes'
ORDER BY time ASC
```
**Alert Condition:**
* `WHEN avg() OF query(A, 5m, now) IS ABOVE 90`
### Alert notifications [#alert-notifications]
Configure notification channels:
1. Go to **Alerting** → **Contact points**
2. Add notification channel (Email, Slack, PagerDuty, etc.)
3. Link alert rules to notification channels
## Dashboard examples [#dashboard-examples]
### System monitoring dashboard [#system-monitoring-dashboard]
Create a comprehensive system monitoring dashboard:
**Panels:**
1. **CPU Usage by Host** (Time series)
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(100 - usage_idle) AS cpu_usage,
host
FROM prod.cpu
WHERE cpu = 'cpu-total' AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
2. **Memory Usage** (Time series)
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(used_percent) AS memory_used,
host
FROM prod.mem
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
3. **Disk Usage** (Gauge)
```sql
SELECT
host,
AVG(used_percent) AS disk_used
FROM prod.disk
WHERE $__timeFilter(time)
GROUP BY host
```
4. **Network Traffic** (Graph)
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
SUM(bytes_recv) * 8 / 1000000 AS mbps_in,
SUM(bytes_sent) * 8 / 1000000 AS mbps_out,
host
FROM prod.net
WHERE $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
5. **Top Hosts by CPU** (Bar gauge)
```sql
SELECT
host,
AVG(100 - usage_idle) AS avg_cpu
FROM prod.cpu
WHERE cpu = 'cpu-total'
AND time >= NOW() - INTERVAL '1 hour'
GROUP BY host
ORDER BY avg_cpu DESC
LIMIT 10
```
### Dashboard layout [#dashboard-layout]
```text
┌─────────────────────────────────────────────────┐
│ System Overview - Last 24 Hours │
│ [Host: All ▼] [Refresh: 30s ▼] │
├───────────────────────┬─────────────────────────┤
│ │ │
│ CPU Usage │ Memory Usage │
│ (Time Series) │ (Time Series) │
│ │ │
├───────────────────────┼─────────────────────────┤
│ │ │
│ Network Traffic │ Disk I/O │
│ (Graph) │ (Graph) │
│ │ │
├───────────────────────┴─────────────────────────┤
│ Top 10 Hosts by CPU Usage (Bar Gauge) │
└─────────────────────────────────────────────────┘
```
## Advanced queries [#advanced-queries]
### Window functions [#window-functions]
**Moving Average:**
```sql
SELECT
time,
usage_idle,
host,
AVG(usage_idle) OVER (
PARTITION BY host
ORDER BY time
ROWS BETWEEN 5 PRECEDING AND CURRENT ROW
) as moving_avg
FROM prod.cpu
WHERE cpu = 'cpu-total' AND $__timeFilter(time)
ORDER BY time ASC
```
### Percentiles [#percentiles]
**CPU Usage Percentiles:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
host,
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY usage_idle) as p50,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
FROM prod.cpu
WHERE cpu = 'cpu-total' AND $__timeFilter(time)
GROUP BY time_bucket(INTERVAL '$__interval', time), host
ORDER BY time ASC
```
### Cross-database queries [#cross-database-queries]
**Production vs Staging Comparison:**
```sql
SELECT
time_bucket(INTERVAL '$__interval', time) as time,
AVG(p.usage_idle) as prod_cpu_idle,
AVG(s.usage_idle) as staging_cpu_idle
FROM prod.cpu p
JOIN staging.cpu s ON p.time = s.time AND p.host = s.host
WHERE p.cpu = 'cpu-total'
AND s.cpu = 'cpu-total'
AND $__timeFilter(p.time)
GROUP BY time_bucket(INTERVAL '$__interval', time)
ORDER BY time ASC
```
## Performance optimization [#performance-optimization]
### 1. Use Apache Arrow [#1-use-apache-arrow]
Arrow protocol is enabled by default and provides significantly faster data transfer:
* Substantially faster than JSON for large result sets for large result sets
* Zero-copy deserialization
* Columnar format perfect for time-series
### 2. Optimize time ranges [#2-optimize-time-ranges]
* Use Grafana's time picker to limit data scanned
* Add time filters with `$__timeFilter()`
* Avoid querying months of data for real-time dashboards
### 3. Leverage time\_bucket() [#3-leverage-time_bucket]
Grafana automatically adjusts `$__interval` based on dashboard width:
```sql
-- Good: Automatic interval adjustment
time_bucket(INTERVAL '$__interval', time)
-- Bad: Fixed interval (too many points)
time_bucket(INTERVAL '1 second', time)
```
### 4. Use LIMIT for exploration [#4-use-limit-for-exploration]
```sql
SELECT * FROM prod.cpu
WHERE $__timeFilter(time)
LIMIT 1000 -- Limit result size
```
### 5. Enable query caching [#5-enable-query-caching]
In Grafana's data source settings:
* Enable **Cache timeout**: 60 seconds
* Repeated queries return instantly from cache
## Troubleshooting [#troubleshooting]
### Plugin not appearing [#plugin-not-appearing]
```bash
# Check plugin directory permissions
ls -la /var/lib/grafana/plugins/basekick-arc-datasource
# Verify plugin.json exists
cat /var/lib/grafana/plugins/basekick-arc-datasource/plugin.json
# Check Grafana logs
tail -f /var/log/grafana/grafana.log
# Restart Grafana
systemctl restart grafana-server
```
### Connection failed [#connection-failed]
```bash
# Verify Arc is running
curl http://localhost:8000/health
# Test API token
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/auth/verify
# Check network connectivity
ping localhost
```
### Query errors [#query-errors]
**"Table not found":**
```sql
-- List available tables
SHOW TABLES FROM prod;
-- Verify database exists
SHOW DATABASES;
```
**"Column not found":**
```sql
-- Describe table schema
DESCRIBE prod.cpu;
```
### Slow queries [#slow-queries]
```bash
# Check Arc query performance
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"sql": "EXPLAIN SELECT * FROM prod.cpu WHERE time > NOW() - INTERVAL '\''1 hour'\''",
"format": "json"
}'
# Trigger compaction
curl -X POST http://localhost:8000/api/v1/compaction/trigger \
-H "Authorization: Bearer $ARC_TOKEN"
```
### Backend plugin issues [#backend-plugin-issues]
```bash
# Ensure backend binary is compiled
cd /path/to/grafana-arc-datasource
mage -v
# Check binary permissions
chmod +x dist/gpx_arc-datasource_*
# Verify Go version
go version # Should be 1.21+
```
## Performance tips [#performance-tips]
1. **Use Arrow Protocol**: Enabled by default, provides considerably faster data transfer
2. **Optimize Time Ranges**: Smaller ranges = faster queries
3. **Leverage time\_bucket()**: Use `$__interval` for automatic aggregation
4. **Add Indexes**: Arc automatically indexes time columns
5. **Enable Caching**: Configure query caching in datasource settings
6. **Limit Result Size**: Use `LIMIT` for exploratory queries
7. **Use Variables**: Filter data with template variables instead of loading everything
## Resources [#resources]
* **[Grafana Arc Datasource GitHub](https://github.com/basekick-labs/grafana-arc-datasource)**
* **[Grafana Documentation](https://grafana.com/docs/grafana/latest/)**
* **[Arc Query API](/arc-enterprise/api-reference/overview/#querying)**
* **[DuckDB SQL Reference](https://duckdb.org/docs/sql/introduction)**
## Next steps [#next-steps]
* **[Query API Reference](/arc-enterprise/api-reference/overview/)**
* **[Telegraf Integration](/arc-enterprise/integrations/telegraf/)** - Collect system metrics
* **[Apache Superset Integration](/arc-enterprise/integrations/superset/)** - BI dashboards
# Integrations (/arc-enterprise/integrations)
Every integration below works the same way against Arc Enterprise as against Arc OSS — the wire protocols are identical. Two things change in a clustered deployment.
Writers and readers are different nodes, so point ingestion tools at the writer endpoint (behind an L7 load balancer if you run more than one writer) and dashboards at a reader endpoint. And each tool gets its own token, scoped by [RBAC](/arc-enterprise/security/rbac/) to the databases it needs, so a compromised dashboard credential cannot write.
# InfluxDB Client Compatibility (/arc-enterprise/integrations/influxdb-clients)
Arc's Line Protocol endpoints use the same paths as InfluxDB, enabling drop-in compatibility with all official InfluxDB client libraries. Point your existing InfluxDB client at Arc - it just works.
## Supported clients [#supported-clients]
All official InfluxDB client libraries work with Arc without code changes:
| Language | Library | Version |
| ------------------ | ------------------------------------------ | ------- |
| Go | `github.com/influxdata/influxdb-client-go` | v2.x |
| Python | `influxdb-client` | v1.x |
| JavaScript/Node.js | `@influxdata/influxdb-client` | v1.x |
| Java | `influxdb-client-java` | v6.x |
| C# | `InfluxDB.Client` | v4.x |
| PHP | `influxdb-client-php` | v3.x |
| Ruby | `influxdb-client-ruby` | v2.x |
**Also supported:**
* Telegraf (InfluxDB output plugin)
* Node-RED (`node-red-contrib-influxdb`)
* Grafana InfluxDB datasource
* Any tool using InfluxDB Line Protocol
## Endpoint mapping [#endpoint-mapping]
| InfluxDB Endpoint | Arc Endpoint | Use Case |
| ----------------- | --------------- | -------------------- |
| `/write` | `/write` | InfluxDB 1.x clients |
| `/api/v2/write` | `/api/v2/write` | InfluxDB 2.x clients |
## Authentication methods [#authentication-methods]
Arc supports all InfluxDB authentication styles:
| Method | Header/Parameter | Example |
| --------------- | ------------------------------- | ------------------ |
| Bearer Token | `Authorization: Bearer ` | Standard OAuth2 |
| Token Header | `Authorization: Token ` | InfluxDB 2.x style |
| Query Parameter | `?p=` | InfluxDB 1.x style |
## Quick start examples [#quick-start-examples]
### Python (influxdb-client) [#python-influxdb-client]
```python
from influxdb_client import InfluxDBClient, Point
from influxdb_client.client.write_api import SYNCHRONOUS
# Point to Arc instead of InfluxDB
client = InfluxDBClient(
url="http://localhost:8000",
token="your-arc-token",
org="myorg" # Required but ignored by Arc
)
write_api = client.write_api(write_options=SYNCHRONOUS)
# Write data - works exactly like InfluxDB
point = Point("cpu") \
.tag("host", "server01") \
.field("usage", 45.2)
write_api.write(bucket="mydb", record=point)
client.close()
```
### Go (influxdb-client-go) [#go-influxdb-client-go]
```go
package main
import (
"context"
"time"
influxdb2 "github.com/influxdata/influxdb-client-go/v2"
)
func main() {
// Point to Arc instead of InfluxDB
client := influxdb2.NewClient("http://localhost:8000", "your-arc-token")
defer client.Close()
writeAPI := client.WriteAPIBlocking("myorg", "mydb")
// Write data - works exactly like InfluxDB
p := influxdb2.NewPoint(
"cpu",
map[string]string{"host": "server01"},
map[string]interface{}{"usage": 45.2},
time.Now(),
)
writeAPI.WritePoint(context.Background(), p)
}
```
### JavaScript/Node.js (@influxdata/influxdb-client) [#javascriptnodejs-influxdatainfluxdb-client]
```javascript
const { InfluxDB, Point } = require('@influxdata/influxdb-client');
// Point to Arc instead of InfluxDB
const client = new InfluxDB({
url: 'http://localhost:8000',
token: 'your-arc-token'
});
const writeApi = client.getWriteApi('myorg', 'mydb');
// Write data - works exactly like InfluxDB
const point = new Point('cpu')
.tag('host', 'server01')
.floatField('usage', 45.2);
writeApi.writePoint(point);
writeApi.close();
```
### Node-RED (node-red-contrib-influxdb) [#node-red-node-red-contrib-influxdb]
Configure the InfluxDB node with:
* **Version**: 2.0
* **URL**: `http://your-arc-host:8000`
* **Token**: Your Arc API token
* **Organization**: Any value (ignored by Arc)
* **Bucket**: Your Arc database name
The node will automatically use `/api/v2/write` which Arc supports natively.
### Telegraf [#telegraf]
```toml
[[outputs.influxdb_v2]]
urls = ["http://localhost:8000"]
token = "your-arc-token"
organization = "myorg"
bucket = "telegraf"
```
Or use the native Arc output plugin for better performance:
```toml
[[outputs.arc]]
url = "http://localhost:8000/api/v1/write/msgpack"
api_key = "your-arc-token"
content_encoding = "gzip"
database = "telegraf"
```
## Migration from InfluxDB [#migration-from-influxdb]
### Step 1: update connection URL [#step-1-update-connection-url]
Change your InfluxDB URL to point to Arc:
```python
# Before (InfluxDB)
client = InfluxDBClient(url="http://influxdb:8086", ...)
# After (Arc)
client = InfluxDBClient(url="http://arc:8000", ...)
```
### Step 2: use Arc token [#step-2-use-arc-token]
Replace your InfluxDB token with an Arc API token:
```bash
# Get token from Arc logs on first startup
docker logs arc 2>&1 | grep -i "admin"
```
### Step 3: map buckets to databases [#step-3-map-buckets-to-databases]
InfluxDB "buckets" map to Arc "databases":
| InfluxDB | Arc |
| ------------ | ----------- |
| Organization | Ignored |
| Bucket | Database |
| Measurement | Measurement |
### Step 4: verify connection [#step-4-verify-connection]
```bash
# Test write
curl -X POST "http://localhost:8000/api/v2/write?bucket=mydb&org=myorg" \
-H "Authorization: Token your-arc-token" \
-d 'test,host=server01 value=1'
# Query data
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer your-arc-token" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM mydb.test LIMIT 10", "format": "json"}'
```
## Connection pooling [#connection-pooling]
For high-throughput applications, enable HTTP connection pooling in your client. This reuses TCP connections instead of opening new ones for each request.
### Python [#python]
```python
from influxdb_client import InfluxDBClient
import urllib3
# Enable connection pooling
http = urllib3.PoolManager(
num_pools=10,
maxsize=50,
retries=urllib3.Retry(3)
)
client = InfluxDBClient(
url="http://localhost:8000",
token="your-token",
org="myorg"
)
```
### Node.js [#nodejs]
```javascript
const { InfluxDB } = require('@influxdata/influxdb-client');
const http = require('http');
// Create agent with connection pooling
const agent = new http.Agent({
keepAlive: true,
maxSockets: 50,
maxFreeSockets: 10
});
const client = new InfluxDB({
url: 'http://localhost:8000',
token: 'your-token',
transportOptions: {
agent: agent
}
});
```
## Differences from InfluxDB [#differences-from-influxdb]
While Arc is compatible with InfluxDB clients, there are some differences:
| Feature | InfluxDB | Arc |
| ------------------ | -------------- | ----------------------- |
| Query Language | Flux, InfluxQL | Standard analytical SQL |
| Organizations | Supported | Ignored |
| Retention Policies | Per-bucket | Via retention API |
| Tasks | Built-in | Via continuous queries |
| Flux Functions | Full support | Not supported |
## Querying data [#querying-data]
Arc uses SQL instead of Flux or InfluxQL. Use the Arc query API:
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer your-token" \
-H "Content-Type: application/json" \
-d '{
"sql": "SELECT time, host, usage FROM mydb.cpu WHERE time > NOW() - INTERVAL '\''1 hour'\'' ORDER BY time DESC LIMIT 100",
"format": "json"
}'
```
Or use the [Arc Python SDK](/arc-enterprise/sdks/python/) for DataFrame support:
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
df = client.query.query_pandas(
"SELECT * FROM mydb.cpu WHERE time > NOW() - INTERVAL '1 hour'"
)
print(df.head())
```
## Troubleshooting [#troubleshooting]
### "404 Not Found" on /write [#404-not-found-on-write]
Ensure you're using Arc version 26.02.1 or later which includes the InfluxDB-compatible endpoints.
### Authentication errors [#authentication-errors]
Arc accepts tokens via:
* `Authorization: Bearer `
* `Authorization: Token `
* `?p=` query parameter
### "Organization not found" [#organization-not-found]
Arc ignores the organization parameter. Any value works.
### Data not appearing [#data-not-appearing]
1. Check the database exists or will be auto-created
2. Force a flush: `POST /api/v1/write/line-protocol/flush`
3. Verify with: `SELECT * FROM mydb.measurement LIMIT 1`
## Next steps [#next-steps]
* **[Python SDK](/arc-enterprise/sdks/python/)** - Native Arc client with DataFrame support
* **[Telegraf Integration](/arc-enterprise/integrations/telegraf/)** - Native Arc output plugin
* **[API Reference](/arc-enterprise/api-reference/overview/)** - Full endpoint documentation
# MQTT Integration (/arc-enterprise/integrations/mqtt)
Ingest data directly from MQTT brokers into Arc. Connect to IoT devices, industrial sensors, and message brokers without middleware.
## Overview [#overview]
Arc provides native MQTT subscription with dynamic, API-driven configuration. Manage multiple MQTT brokers and subscriptions at runtime without server restarts.
**Key features:**
* **API-driven subscription management** - Create, update, delete, start/stop subscriptions via REST API
* **Multiple simultaneous brokers** - Connect to different MQTT brokers for different data sources
* **Topic wildcards** - Subscribe using `+` (single level) and `#` (multi-level) wildcards
* **Auto-detection** - Automatically detects JSON and MessagePack message formats
* **High performance** - MessagePack columnar format for high-throughput ingestion
* **Topic mapping** - Extract tags from topic path segments
* **TLS/SSL support** - Client certificates and CA verification
* **Encrypted credentials** - Passwords encrypted at rest using AES-256-GCM
* **Auto-reconnect** - Exponential backoff on connection loss
* **QoS support** - QoS 0, 1, and 2
## Prerequisites [#prerequisites]
* Arc server running (v26.02.1 or higher)
* Arc API token (if authentication is enabled)
* MQTT broker accessible from Arc server
## Quick start [#quick-start]
### 1. Enable MQTT in Arc [#1-enable-mqtt-in-arc]
```toml
[mqtt]
enabled = true
```
Or via environment variable:
```bash
ARC_MQTT_ENABLED=true
```
Restart Arc to apply the configuration.
### 2. Create a subscription [#2-create-a-subscription]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"auto_start": true
}'
```
**Response:**
```json
{
"id": "sub_abc123",
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"status": "running",
"created_at": "2026-02-13T10:00:00Z"
}
```
### 3. Send test data [#3-send-test-data]
Publish a message to your MQTT broker:
```bash
mosquitto_pub -h localhost -t "sensors/temperature" \
-m '{"time": 1706745600000000, "value": 23.5, "device_id": "sensor-001"}'
```
### 4. Query the data [#4-query-the-data]
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"sql": "SELECT * FROM iot.temperature ORDER BY time DESC LIMIT 10",
"format": "json"
}'
```
## API reference [#api-reference]
### Subscription management [#subscription-management]
| Method | Endpoint | Description |
| -------- | --------------------------------- | ------------------------- |
| `POST` | `/api/v1/mqtt/subscriptions` | Create a new subscription |
| `GET` | `/api/v1/mqtt/subscriptions` | List all subscriptions |
| `GET` | `/api/v1/mqtt/subscriptions/{id}` | Get subscription details |
| `PUT` | `/api/v1/mqtt/subscriptions/{id}` | Update subscription |
| `DELETE` | `/api/v1/mqtt/subscriptions/{id}` | Delete subscription |
### Lifecycle control [#lifecycle-control]
| Method | Endpoint | Description |
| ------ | ----------------------------------------- | -------------------- |
| `POST` | `/api/v1/mqtt/subscriptions/{id}/start` | Start subscription |
| `POST` | `/api/v1/mqtt/subscriptions/{id}/stop` | Stop subscription |
| `POST` | `/api/v1/mqtt/subscriptions/{id}/restart` | Restart subscription |
### Monitoring [#monitoring]
| Method | Endpoint | Description |
| ------ | --------------------------------------- | ----------------------------------- |
| `GET` | `/api/v1/mqtt/subscriptions/{id}/stats` | Get subscription stats |
| `GET` | `/api/v1/mqtt/stats` | Aggregate stats (all subscriptions) |
| `GET` | `/api/v1/mqtt/health` | Health check |
## Subscription options [#subscription-options]
### Create subscription request [#create-subscription-request]
```json
{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#", "factory/+/metrics"],
"database": "iot",
"qos": 1,
"client_id": "arc-factory",
"username": "mqtt_user",
"password": "mqtt_pass",
"tls_enabled": false,
"tls_cert_path": "/path/to/client.crt",
"tls_key_path": "/path/to/client.key",
"tls_ca_path": "/path/to/ca.crt",
"topic_mapping": {},
"keep_alive_seconds": 60,
"connect_timeout_seconds": 30,
"reconnect_max_seconds": 60,
"auto_start": true
}
```
### Field reference [#field-reference]
| Field | Type | Required | Default | Description |
| ------------------------- | --------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | string | Yes | - | Unique subscription name |
| `broker` | string | Yes | - | MQTT broker URL (tcp\://, ssl://, ws\://, wss\://) |
| `topics` | array | Yes | - | List of topics to subscribe |
| `database` | string | Yes | - | Target Arc database |
| `qos` | int | No | 1 | QoS level: 0, 1, or 2 |
| `client_id` | string | No | auto | MQTT client ID |
| `username` | string | No | - | MQTT username |
| `password` | string | No | - | MQTT password (encrypted at rest) |
| `tls_enabled` | bool | No | false | Enable TLS/SSL |
| `tls_cert_path` | string | No | - | Client certificate path |
| `tls_key_path` | string | No | - | Client key path |
| `tls_ca_path` | string | No | - | CA certificate path |
| `topic_mapping` | object (`{string: string}`) | No | \{} | Per-topic target-database override: maps an exact MQTT topic to a database name, overriding `database` for messages on that topic. See [Topic Mapping](#topic-mapping-per-topic-database-override). |
| `keep_alive_seconds` | int | No | 60 | MQTT keep-alive interval |
| `connect_timeout_seconds` | int | No | 30 | Connection timeout |
| `reconnect_max_seconds` | int | No | 60 | Maximum reconnect backoff delay. The reconnect delay starts at 1 second and doubles up to this cap (the 1-second minimum is fixed by the MQTT client library and is not configurable). |
| `auto_start` | bool | No | true | Start on creation and server restart |
## Message formats [#message-formats]
Arc automatically detects the message format based on content.
### JSON single record [#json-single-record]
```json
{
"time": 1706745600000000,
"temperature": 23.5,
"humidity": 65.2,
"device_id": "sensor-001"
}
```
### JSON batch [#json-batch]
```json
[
{"time": 1706745600000000, "temperature": 23.5},
{"time": 1706745601000000, "temperature": 23.6},
{"time": 1706745602000000, "temperature": 23.4}
]
```
### MessagePack row-based [#messagepack-row-based]
Same structure as JSON, but MessagePack encoded. Detected via magic bytes.
### MessagePack columnar (fastest) [#messagepack-columnar-fastest]
```json
{
"m": "temperature",
"columns": {
"time": [1706745600000000, 1706745601000000],
"value": [23.5, 23.6],
"device_id": ["sensor-001", "sensor-001"]
}
}
```
**Performance:** MessagePack columnar format sustains high ingest throughput.
### Timestamp handling [#timestamp-handling]
* If `time` field is present: used as-is (auto-detects milliseconds/microseconds/nanoseconds)
* If `time` field is missing: current UTC time is used
## Measurement, tags, and fields [#measurement-tags-and-fields]
Arc derives the measurement, tags, and fields **from the message payload**, not from the topic structure. The topic itself is not parsed for the measurement name or for tag values.
For each decoded message:
* **Measurement** — taken from the payload's `m` field, or `measurement` field. If neither is present, it defaults to `mqtt`.
* **Tags** — taken from a `tags` object in the payload (string values).
* **Fields** — taken from a `fields` object if present; otherwise every remaining top-level key (anything other than `m`/`measurement`, `t`/`time`/`timestamp`, `tags`, `fields`) is treated as a field.
* **Timestamp** — from `t`, `time`, or `timestamp` (auto-detects ms/µs/ns); current UTC time if absent.
So to land in measurement `machine_metrics` with tags `line` and `machine_id`, publish a payload like:
```json
{
"m": "machine_metrics",
"time": 1706745600000000,
"tags": { "line": "A", "machine_id": "42" },
"fields": { "temperature": 71.5, "rpm": 1480 }
}
```
A flat payload with no `m`/`tags`/`fields` (e.g. `{"time": ..., "temperature": 23.5}`) is also accepted: it lands in the default `mqtt` measurement with the remaining keys as fields and no tags.
Deriving the measurement or tags from topic path segments (e.g. `tags_from_topic` / positional extraction) is **not** currently supported. Set the measurement and tags in the published payload as shown above.
## Topic mapping (per-topic database override) [#topic-mapping-per-topic-database-override]
`topic_mapping` maps an **exact MQTT topic string to a target database name**, overriding the subscription's `database` for messages received on that topic. It is a flat `{ "": "" }` object — it does not configure measurements or tags.
```json
{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["factory/line1/metrics", "factory/line2/metrics"],
"database": "iot",
"topic_mapping": {
"factory/line2/metrics": "iot_line2"
}
}
```
In this example, messages on `factory/line1/metrics` are written to the default `iot` database, while messages on `factory/line2/metrics` are routed to `iot_line2`.
The mapping key is matched against the message's actual topic by exact string equality — wildcard topic patterns (`+`, `#`) are not expanded for matching. A subscription may use wildcards in `topics`, but `topic_mapping` keys must be the concrete topics you want to route to a different database.
## Authentication [#authentication]
### Basic authentication [#basic-authentication]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "authenticated-broker",
"broker": "tcp://broker.example.com:1883",
"topics": ["data/#"],
"database": "production",
"username": "mqtt_user",
"password": "mqtt_password"
}'
```
### Password encryption [#password-encryption]
Passwords are encrypted at rest using AES-256-GCM. Set the encryption key:
```bash
# Generate a 32-byte key
openssl rand -base64 32
# Set environment variable before starting Arc
export ARC_ENCRYPTION_KEY="your-base64-encoded-32-byte-key"
```
The encryption key is only required when subscriptions have passwords. Subscriptions without credentials work without the key.
## TLS/SSL configuration [#tlsssl-configuration]
### Server certificate verification [#server-certificate-verification]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "secure-broker",
"broker": "ssl://broker.example.com:8883",
"topics": ["secure/#"],
"database": "production",
"tls_enabled": true,
"tls_ca_path": "/etc/arc/certs/ca.crt"
}'
```
### Client certificate authentication (mTLS) [#client-certificate-authentication-mtls]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "mtls-broker",
"broker": "ssl://broker.example.com:8883",
"topics": ["secure/#"],
"database": "production",
"tls_enabled": true,
"tls_cert_path": "/etc/arc/certs/client.crt",
"tls_key_path": "/etc/arc/certs/client.key",
"tls_ca_path": "/etc/arc/certs/ca.crt"
}'
```
## Configuration examples [#configuration-examples]
### IoT sensor network [#iot-sensor-network]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "iot-sensors",
"broker": "tcp://mosquitto:1883",
"topics": [
"sensors/+/temperature",
"sensors/+/humidity",
"sensors/+/pressure"
],
"database": "iot",
"qos": 1
}'
```
Devices set the measurement and tags in the payload — e.g. a temperature sensor publishes:
```json
{ "m": "temperature", "tags": { "sensor_id": "temp-001" }, "fields": { "value": 23.5 } }
```
### Industrial factory [#industrial-factory]
```bash
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "factory-floor",
"broker": "tcp://factory-mqtt:1883",
"topics": ["factory/+/+/metrics"],
"database": "manufacturing",
"qos": 2
}'
```
Machines publish the measurement and tags in the payload:
```json
{
"m": "machine_metrics",
"tags": { "line": "A", "machine_id": "42" },
"fields": { "temperature": 71.5, "rpm": 1480 }
}
```
## Monitoring [#monitoring-1]
### Subscription stats [#subscription-stats]
```bash
# Stats for a specific subscription
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/mqtt/subscriptions/{id}/stats
# Aggregate stats for all subscriptions
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/mqtt/stats
```
**Response:**
```json
{
"status": "success",
"running_count": 2,
"subscriptions_stats": {
"sub_abc123": {
"messages_received": 15420,
"bytes_received": 2458320,
"decode_errors": 0,
"last_message_at": "2026-02-13T10:30:15Z",
"topics": {
"sensors/temperature": 8500,
"sensors/humidity": 6920
}
}
}
}
```
### Health check [#health-check]
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/mqtt/health
```
**Response:**
```json
{
"status": "healthy",
"healthy": true,
"running_count": 2,
"connected_count": 2,
"disconnected_count": 0,
"service": "mqtt_subscriptions"
}
```
### Prometheus metrics [#prometheus-metrics]
Arc exposes MQTT metrics for Prometheus:
| Metric | Type | Description |
| ---------------------------------- | ------- | ------------------------------- |
| `arc_mqtt_messages_received_total` | Counter | Total messages received |
| `arc_mqtt_bytes_received_total` | Counter | Total bytes received |
| `arc_mqtt_decode_errors_total` | Counter | Message decode errors |
| `arc_mqtt_connection_status` | Gauge | Connection status (1=connected) |
## Querying MQTT data [#querying-mqtt-data]
### Basic query [#basic-query]
```sql
SELECT * FROM iot.temperature
ORDER BY time DESC
LIMIT 10;
```
### Time-based aggregation [#time-based-aggregation]
```sql
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(value) as avg_temp,
MIN(value) as min_temp,
MAX(value) as max_temp
FROM iot.temperature
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;
```
### Filter by tag [#filter-by-tag]
```sql
SELECT * FROM iot.sensor_data
WHERE sensor_id = 'temp-001'
AND time > NOW() - INTERVAL '24 hours'
ORDER BY time DESC;
```
## Troubleshooting [#troubleshooting]
### Connection failed [#connection-failed]
Check subscription status:
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/mqtt/subscriptions/{id}
```
If status is `error`, verify:
* Broker URL is correct (tcp\://, ssl://, ws\://)
* Broker is reachable from Arc server
* Credentials are correct
* TLS certificates are valid
### No data appearing [#no-data-appearing]
1. Verify subscription is running (status should be `"running"`)
2. Check stats for received messages
3. Verify topic pattern matches published topics
4. Check Arc logs for decode errors
### Messages not decoding [#messages-not-decoding]
Ensure messages are valid JSON or MessagePack:
```bash
# Test with simple JSON
mosquitto_pub -h localhost -t "test/data" \
-m '{"time": 1706745600000000, "value": 42}'
```
Check for decode errors in stats:
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/mqtt/subscriptions/{id}/stats
```
## Best practices [#best-practices]
1. **Use descriptive subscription names** — Names like `prod-factory-floor-sensors` are easier to manage than `sub1`.
2. **Separate databases by environment** — Use different target databases for production, staging, and development data.
3. **Use QoS appropriately** — QoS 0 for highest throughput, QoS 1 for reliable delivery (recommended), QoS 2 for exactly-once (highest overhead).
4. **Set reasonable reconnect intervals** — Default min 1s / max 60s works well. Avoid setting min too low to prevent broker overload.
5. **Use topic wildcards efficiently** — Subscribe to specific patterns (`sensors/+/temperature`) rather than overly broad ones (`#`).
6. **Monitor subscription health** — Set up alerts on `arc_mqtt_connection_status == 0` and `rate(arc_mqtt_decode_errors_total[5m]) > 0`.
## Docker Compose example [#docker-compose-example]
```yaml
version: '3.8'
services:
arc:
image: basekick/arc:latest
ports:
- "8000:8000"
volumes:
- arc-data:/data
environment:
ARC_MQTT_ENABLED: "true"
ARC_AUTH_ENABLED: "true"
ARC_ENCRYPTION_KEY: "${ARC_ENCRYPTION_KEY}"
depends_on:
- mosquitto
mosquitto:
image: eclipse-mosquitto:2
ports:
- "1883:1883"
volumes:
- ./mosquitto.conf:/mosquitto/config/mosquitto.conf
volumes:
arc-data:
```
## Next steps [#next-steps]
* [Tiered Storage](/arc-enterprise/data-lifecycle/tiered-storage/) — Manage MQTT data lifecycle with hot/cold tiering
* [Automated Scheduling](/arc-enterprise/operations/automated-scheduling/) — Downsample MQTT data automatically
* [Audit Logging](/arc-enterprise/security/audit-logging/) — Track MQTT subscription changes
# OpenTelemetry Integration (/arc-enterprise/integrations/opentelemetry)
Send traces, metrics, and logs from OpenTelemetry Collector to Arc for unified observability.
## Overview [#overview]
The Arc OpenTelemetry Exporter enables you to send all your telemetry data from the OpenTelemetry Collector to Arc:
* **Traces**: Distributed traces with full span hierarchy
* **Metrics**: All metric types (gauge, counter, histogram, summary)
* **Logs**: Structured logs with attributes
* **High Performance**: Uses Arc's columnar MessagePack format
* **Compression**: Automatic gzip compression
* **Retry Logic**: Configurable retry with exponential backoff
* **Authentication**: Bearer token support
**Performance:**
* Traces: 500K-1M spans/sec
* Metrics: 3M-6M data points/sec
* Logs: 1M-2M logs/sec
## Why OpenTelemetry + Arc? [#why-opentelemetry--arc]
Traditional observability requires 3+ separate systems:
* Jaeger for traces
* Prometheus for metrics
* Loki/Elasticsearch for logs
* **Manual correlation** between systems
**With Arc + OpenTelemetry:**
* All signals in one database
* Join traces, metrics, and logs in SQL
* No manual correlation needed
* Single query for complete context
* One storage backend to manage
This is **unified observability**.
## Installation [#installation]
### Option 1: OpenTelemetry Collector Builder (OCB) [#option-1-opentelemetry-collector-builder-ocb]
Add to your `builder-config.yaml`:
```yaml
exporters:
- gomod: github.com/basekick-labs/arc-opentelemetry-exporter
```
Build the collector:
```bash
ocb --config builder-config.yaml
```
### Option 2: pre-built binary [#option-2-pre-built-binary]
Download from the [releases page](https://github.com/basekick-labs/arc-opentelemetry-exporter/releases):
```bash
# Grab the latest release tag, then download that release's binary
VERSION=$(curl -s https://api.github.com/repos/basekick-labs/arc-opentelemetry-exporter/releases/latest | grep tag_name | cut -d '"' -f 4 | sed 's/v//')
wget -O arc-exporter https://github.com/basekick-labs/arc-opentelemetry-exporter/releases/download/v${VERSION}/arc-exporter-${VERSION}-linux-amd64
# Make executable
chmod +x arc-exporter
# Run
./arc-exporter --config=config.yaml
```
## Quick start [#quick-start]
### 1. Start Arc [#1-start-arc]
```bash
docker run -d -p 8000:8000 \
-e STORAGE_BACKEND=local \
-v arc-data:/app/data \
ghcr.io/basekick-labs/arc:latest
```
### 2. Get your API token [#2-get-your-api-token]
```bash
# Check logs for admin token
docker logs 2>&1 | grep "Admin token"
# Or create a new token
curl -X POST http://localhost:8000/api/v1/auth/tokens \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "otel-collector",
"description": "OpenTelemetry Collector access"
}'
```
### 3. Create collector configuration [#3-create-collector-configuration]
Create `otel-config.yaml`:
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1000
exporters:
arc:
endpoint: http://localhost:8000
auth_token: your-arc-token-here
# Recommended: Separate databases per signal type
traces_database: traces
metrics_database: metrics
logs_database: logs
# Optional: Custom measurement names
traces_measurement: distributed_traces
logs_measurement: logs
# Optional: HTTP settings
timeout: 30s
compression: gzip
# Optional: Retry configuration
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [arc]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [arc]
logs:
receivers: [otlp]
processors: [batch]
exporters: [arc]
```
### 4. Run OpenTelemetry Collector [#4-run-opentelemetry-collector]
```bash
./arc-exporter --config=otel-config.yaml
```
### 5. Send telemetry data [#5-send-telemetry-data]
Your applications instrumented with OpenTelemetry SDKs will now send data to Arc!
**Example: Python Application**
```python
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# Configure tracer
trace.set_tracer_provider(TracerProvider())
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
# Create spans
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-operation"):
# Your code here
print("Trace sent to Arc via OTel Collector!")
```
## Configuration [#configuration]
### Database organization strategies [#database-organization-strategies]
#### Strategy 1: single database (simple) [#strategy-1-single-database-simple]
All signals in one database:
```yaml
exporters:
arc:
endpoint: http://localhost:8000
database: default
```
**Structure:**
```text
default/
├── distributed_traces
├── logs
├── system_cpu_usage
└── http_requests_total
```
**Pros:** Simple, easy cross-signal correlation
**Cons:** All data in one namespace
#### Strategy 2: database per signal (recommended) [#strategy-2-database-per-signal-recommended]
Separate databases for each signal type:
```yaml
exporters:
arc:
endpoint: http://localhost:8000
traces_database: traces
metrics_database: metrics
logs_database: logs
```
**Structure:**
```text
traces/
└── distributed_traces
metrics/
├── system_cpu_usage
├── system_memory_usage
└── http_requests_total
logs/
└── logs
```
**Pros:**
* Clean separation of concerns
* Independent retention policies per signal
* Independent scaling and storage backends
* Easier permission management
* Matches traditional observability architecture
**Cons:** Slightly more complex configuration
**Recommended for production deployments.**
### Configuration options [#configuration-options]
| Option | Description | Default |
| ----------------------------------- | ------------------------------- | -------------------- |
| `endpoint` | Arc API endpoint | Required |
| `auth_token` | Authentication token | Optional |
| `database` | Single database for all signals | `default` |
| `traces_database` | Database for traces | - |
| `metrics_database` | Database for metrics | - |
| `logs_database` | Database for logs | - |
| `traces_measurement` | Table name for traces | `distributed_traces` |
| `logs_measurement` | Table name for logs | `logs` |
| `timeout` | HTTP request timeout | `30s` |
| `compression` | Compression type | `gzip` |
| `retry_on_failure.enabled` | Enable retry logic | `true` |
| `retry_on_failure.initial_interval` | Initial retry interval | `5s` |
| `retry_on_failure.max_interval` | Maximum retry interval | `30s` |
| `retry_on_failure.max_elapsed_time` | Maximum total retry time | `300s` |
## Data format [#data-format]
The exporter uses Arc's high-performance **columnar MessagePack format** with **dynamic columns**. All OpenTelemetry attributes automatically become individual columns for optimal query performance.
### Traces [#traces]
All span attributes and resource attributes become columns:
```json
{
"m": "distributed_traces",
"columns": {
"time": [1699900000000],
"trace_id": ["5b8efff798038103d269b633813fc60c"],
"span_id": ["def456..."],
"parent_span_id": ["ghi789..."],
"service_name": ["api-gateway"],
"operation_name": ["HTTP GET /users"],
"span_kind": ["server"],
"duration_ns": [1234567],
"status_code": [0],
"http.method": ["GET"],
"http.status_code": [200],
"http.url": ["/api/users"],
"host.name": ["server-1"]
}
}
```
**Dynamic schema**: Columns created automatically from span and resource attributes.
### Metrics [#metrics]
Each metric name becomes its own table. All attributes become columns:
```json
{
"m": "http_requests_total",
"columns": {
"time": [1699900000000],
"value": [42.0],
"service": ["api"],
"method": ["GET"],
"status": ["200"],
"host.name": ["server-1"]
}
}
```
**Metric name sanitization:**
* `system.cpu.usage` → `system_cpu_usage`
* `http.server.duration` → `http_server_duration`
* `process-memory-bytes` → `process_memory_bytes`
### Logs [#logs]
All log attributes and resource attributes become columns:
```json
{
"m": "logs",
"columns": {
"time": [1699900000000],
"severity": ["ERROR"],
"severity_number": [17],
"body": ["Database connection failed"],
"trace_id": ["abc123..."],
"span_id": ["def456..."],
"service_name": ["api-gateway"],
"http.method": ["POST"],
"user_id": ["12345"],
"host.name": ["server-1"]
}
}
```
## Querying data [#querying-data]
### Traces [#traces-1]
```sql
-- Recent traces
SELECT * FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '1 hour'
LIMIT 100;
-- Traces by service
SELECT
service_name,
operation_name,
duration_ns / 1000000 AS duration_ms,
"http.method",
"http.status_code"
FROM traces.distributed_traces
WHERE service_name = 'api-gateway'
AND time > NOW() - INTERVAL '1 hour'
ORDER BY time DESC;
-- Debug specific trace
SELECT * FROM traces.distributed_traces
WHERE trace_id = '5b8efff798038103d269b633813fc60c'
ORDER BY time;
-- Slow requests (p99 latency)
SELECT
service_name,
operation_name,
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY duration_ns) / 1000000 AS p99_ms
FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY service_name, operation_name
ORDER BY p99_ms DESC;
```
### Metrics [#metrics-1]
Each metric is its own table with all attributes as columns:
```sql
-- CPU usage
SELECT
time,
value,
"host.name",
cpu,
state
FROM metrics.system_cpu_usage
WHERE time > NOW() - INTERVAL '1 hour'
AND "host.name" = 'server-1'
ORDER BY time DESC;
-- HTTP requests by method and status
SELECT
time_bucket(INTERVAL '1 minute', time) AS minute,
method,
status,
SUM(value) AS total_requests
FROM metrics.http_requests_total
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY minute, method, status
ORDER BY minute DESC;
-- Memory usage aggregated
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
"host.name",
AVG(value) AS avg_memory_bytes
FROM metrics.system_memory_usage
WHERE time > NOW() - INTERVAL '6 hours'
GROUP BY bucket, "host.name"
ORDER BY bucket DESC;
```
### Logs [#logs-1]
All attributes are individual columns for fast filtering:
```sql
-- Recent error logs
SELECT
time,
severity,
body,
service_name,
"host.name",
trace_id
FROM logs.logs
WHERE severity IN ('ERROR', 'FATAL')
AND time > NOW() - INTERVAL '1 hour'
ORDER BY time DESC;
-- Logs for specific trace (correlation)
SELECT
time,
severity,
body,
service_name
FROM logs.logs
WHERE trace_id = '5b8efff798038103d269b633813fc60c'
ORDER BY time;
-- Count errors by service
SELECT
service_name,
"host.name",
COUNT(*) AS error_count
FROM logs.logs
WHERE severity IN ('ERROR', 'FATAL')
AND time > NOW() - INTERVAL '1 hour'
GROUP BY service_name, "host.name"
ORDER BY error_count DESC;
```
## Unified observability: Join across signals [#unified-observability-join-across-signals]
Arc's most powerful feature: **correlate traces, metrics, and logs in a single SQL query**.
### Example 1: failed requests with full context [#example-1-failed-requests-with-full-context]
Get traces, error logs, and CPU metrics for failed requests:
```sql
SELECT
t.time,
t.trace_id,
t.service_name,
t.operation_name,
t.duration_ns / 1000000 AS duration_ms,
t."http.status_code",
l.severity,
l.body AS error_message,
cpu.value AS cpu_usage
FROM traces.distributed_traces t
LEFT JOIN logs.logs l
ON t.trace_id = l.trace_id
LEFT JOIN metrics.system_cpu_usage cpu
ON t.service_name = cpu.service_name
AND time_bucket(INTERVAL '1 minute', t.time) = time_bucket(INTERVAL '1 minute', cpu.time)
WHERE t.status_code >= 2 -- OTel status: 2 = Error
AND t.time > NOW() - INTERVAL '1 hour'
ORDER BY t.time DESC
LIMIT 100;
```
**Result:** Traces + error logs + CPU usage at time of failure — all in one query!
### Example 2: service health dashboard [#example-2-service-health-dashboard]
Complete service health metrics:
```sql
WITH trace_stats AS (
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
service_name,
COUNT(*) AS request_count,
AVG(duration_ns / 1000000) AS avg_latency_ms,
SUM(CASE WHEN status_code >= 2 THEN 1 ELSE 0 END) AS error_count
FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, service_name
),
error_logs AS (
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
service_name,
COUNT(*) AS log_error_count
FROM logs.logs
WHERE severity IN ('ERROR', 'FATAL')
AND time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, service_name
),
cpu_stats AS (
SELECT
time_bucket(INTERVAL '5 minutes', time) AS bucket,
service_name,
AVG(value) AS avg_cpu
FROM metrics.system_cpu_usage
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, service_name
)
SELECT
ts.bucket AS time,
ts.service_name,
ts.request_count,
ROUND(ts.avg_latency_ms, 2) AS avg_latency_ms,
ts.error_count,
ROUND((ts.error_count::float / NULLIF(ts.request_count, 0) * 100), 2) AS error_rate_pct,
el.log_error_count,
ROUND(cs.avg_cpu, 2) AS avg_cpu_usage
FROM trace_stats ts
LEFT JOIN error_logs el ON ts.bucket = el.bucket AND ts.service_name = el.service_name
LEFT JOIN cpu_stats cs ON ts.bucket = cs.bucket AND ts.service_name = cs.service_name
ORDER BY ts.bucket DESC, ts.service_name;
```
**Result:**
* Request volume and latency (traces)
* Error rate (traces)
* Error log count (logs)
* CPU usage (metrics)
All from one database, in one query!
### Example 3: debug incident timeline [#example-3-debug-incident-timeline]
Unified timeline of traces and logs for a single request:
```sql
SELECT
t.time,
'trace' AS signal_type,
t.operation_name AS event,
t.duration_ns / 1000000 AS duration_ms,
t.status_code,
NULL AS severity,
NULL AS body
FROM traces.distributed_traces t
WHERE t.trace_id = '5b8efff798038103d269b633813fc60c'
UNION ALL
SELECT
l.time,
'log' AS signal_type,
l.service_name AS event,
NULL AS duration_ms,
NULL AS status_code,
l.severity,
l.body
FROM logs.logs l
WHERE l.trace_id = '5b8efff798038103d269b633813fc60c'
ORDER BY time;
```
**Result:** Complete chronological view of all events for a single request!
## Use cases [#use-cases]
### Application performance monitoring [#application-performance-monitoring]
Monitor service latency, error rates, and throughput:
```sql
SELECT
time_bucket(INTERVAL '1 minute', time) AS minute,
service_name,
COUNT(*) AS requests,
AVG(duration_ns / 1000000) AS avg_latency_ms,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ns) / 1000000 AS p95_ms,
SUM(CASE WHEN status_code >= 2 THEN 1 ELSE 0 END) AS errors
FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY minute, service_name
ORDER BY minute DESC;
```
### Distributed tracing [#distributed-tracing]
Analyze request flows across microservices:
```sql
-- Trace all spans in a distributed transaction
SELECT
span_id,
parent_span_id,
service_name,
operation_name,
duration_ns / 1000000 AS duration_ms,
"http.method",
"http.url"
FROM traces.distributed_traces
WHERE trace_id = 'your-trace-id'
ORDER BY time;
```
### Log analysis [#log-analysis]
Search and analyze structured logs:
```sql
-- Find all errors from a specific user session
SELECT
time,
severity,
body,
service_name,
user_id,
"http.method",
"http.url"
FROM logs.logs
WHERE user_id = '12345'
AND severity IN ('ERROR', 'WARN')
AND time > NOW() - INTERVAL '24 hours'
ORDER BY time DESC;
```
## Performance optimization [#performance-optimization]
### 1. Use batch processor [#1-use-batch-processor]
Always use the `batch` processor for high throughput:
```yaml
processors:
batch:
timeout: 1s
send_batch_size: 1000 # Adjust based on your load
```
### 2. Configure retry logic [#2-configure-retry-logic]
Handle transient failures:
```yaml
exporters:
arc:
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
```
### 3. Enable compression [#3-enable-compression]
Reduce network bandwidth:
```yaml
exporters:
arc:
compression: gzip # Default and recommended
```
### 4. Tune collector resources [#4-tune-collector-resources]
For high-volume deployments:
```yaml
service:
telemetry:
metrics:
level: detailed
extensions: [health_check]
```
### 5. Use separate databases [#5-use-separate-databases]
For production, use database-per-signal strategy:
```yaml
exporters:
arc:
traces_database: traces
metrics_database: metrics
logs_database: logs
```
## Troubleshooting [#troubleshooting]
### Collector not sending data [#collector-not-sending-data]
```bash
# Check collector logs
./arc-exporter --config=config.yaml
# Verify Arc is accessible
curl http://localhost:8000/health
# Test authentication
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/auth/verify
```
### High memory usage [#high-memory-usage]
Reduce batch size:
```yaml
processors:
batch:
timeout: 1s
send_batch_size: 500 # Reduce from 1000
```
### Data not appearing in Arc [#data-not-appearing-in-arc]
```sql
-- Check if data is being written
SHOW TABLES FROM traces;
SHOW TABLES FROM metrics;
SHOW TABLES FROM logs;
-- Verify recent data
SELECT COUNT(*) FROM traces.distributed_traces
WHERE time > NOW() - INTERVAL '5 minutes';
```
### Connection timeouts [#connection-timeouts]
Increase timeout:
```yaml
exporters:
arc:
timeout: 60s # Increase from 30s
```
## Resources [#resources]
* **[Arc OpenTelemetry Exporter GitHub](https://github.com/basekick-labs/arc-opentelemetry-exporter)**
* **[OpenTelemetry Collector Docs](https://opentelemetry.io/docs/collector/)**
* **[Arc Query API](/arc-enterprise/api-reference/overview/)**
* **[OpenTelemetry SDK Documentation](https://opentelemetry.io/docs/instrumentation/)**
## Next steps [#next-steps]
* **[Getting Started with Arc](/arc-enterprise/getting-started/)** - Install Arc
* **[Grafana Integration](/arc-enterprise/integrations/grafana/)** - Visualize OpenTelemetry data
* **[Query API Reference](/arc-enterprise/api-reference/overview/)** - Learn Arc SQL
* **[Data Lifecycle](/arc-enterprise/data-lifecycle/retention-policies/)** - Manage retention policies
***
**Ready for unified observability!**
# Redpanda Connect Integration (/arc-enterprise/integrations/redpanda-connect)
Stream data from any of Redpanda Connect's 200+ sources directly into Arc using the native Arc output plugin.
## Overview [#overview]
[Redpanda Connect](https://github.com/redpanda-data/connect) (formerly Benthos) is a stream processor that connects sources to sinks with a single YAML config file. It handles transformations, filtering, batching, retries, and backpressure out of the box. Arc has a native output plugin that speaks Arc's MessagePack ingestion protocol directly, so data flows from your source into Arc's columnar storage with no translation layer.
**Benefits:**
* Native MessagePack columnar format with zstd compression
* 200+ input connectors (Kafka, HTTP, MQTT, S3, GCS, Postgres CDC, etc.)
* Bloblang transformations for reshaping, filtering, and enriching data in-flight
* Interpolated measurement names for per-message routing to different Arc tables
* Single binary, no JVM, no cluster required
## Why this matters [#why-this-matters]
Arc already has native ingestion paths for metrics ([Telegraf](/arc-enterprise/integrations/telegraf/)) and IoT data ([MQTT](/arc-enterprise/integrations/mqtt/)). Redpanda Connect covers a different gap: event-driven data that needs reshaping, filtering, or enrichment before it lands in Arc.
| Tool | Best For |
| ---------------- | ------------------------------------------------------------------------ |
| Telegraf | Pulling metrics from systems at fixed intervals |
| Native MQTT | Subscribing to IoT brokers directly |
| Redpanda Connect | Event streams, CDC, webhooks, complex transformations, fan-out pipelines |
Some concrete examples where Redpanda Connect fits:
* **Kafka to Arc** — consume events, filter out bot traffic, normalize timestamps, write to Arc
* **Webhooks to Arc** — receive HTTP webhooks from third-party APIs, reshape the payload, store for analytics
* **CDC to Arc** — capture Postgres/MySQL change events and stream them into Arc for historical tracking
* **Multi-destination** — send the same data to Arc and Kafka (or S3, or Elasticsearch) with different transformations per sink
## Prerequisites [#prerequisites]
* **Redpanda Connect 4.88 or higher** (required for the `arc` output)
* Arc server running and accessible
* Arc API token (if auth is enabled)
## Quick start [#quick-start]
### 1. Install Redpanda Connect [#1-install-redpanda-connect]
```bash
# Homebrew (macOS/Linux)
brew install redpanda-data/tap/redpanda-connect
# Docker
docker run --rm -v $(pwd)/config.yaml:/config.yaml \
docker.redpanda.com/redpandadata/connect:latest run /config.yaml
# Direct binary download
# https://github.com/redpanda-data/connect/releases
```
Verify you have 4.88+:
```bash
redpanda-connect --version
```
### 2. Create a pipeline config [#2-create-a-pipeline-config]
Create `arc-pipeline.yaml`:
```yaml
input:
generate:
count: 10
interval: 1s
mapping: |
root.vehicle_id = "truck-" + random_int(min: 1, max: 5).string()
root.lat = 40.7128 + (random_int(min: -1000, max: 1000).number() / 10000)
root.lon = -74.0060 + (random_int(min: -1000, max: 1000).number() / 10000)
root.speed_kmh = random_int(min: 0, max: 120)
output:
arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: logistics
measurement: fleet_tracking
format: columnar
compression: zstd
batching:
count: 100
period: 1s
```
### 3. Run the pipeline [#3-run-the-pipeline]
```bash
export ARC_TOKEN="your-arc-token"
redpanda-connect run arc-pipeline.yaml
```
Expected output:
```text
INFO Running main config from specified file path=arc-pipeline.yaml
INFO Input type generate is now active
INFO Output type arc is now active
INFO Pipeline has terminated. Shutting down the service
```
### 4. Verify data in Arc [#4-verify-data-in-arc]
```bash
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT vehicle_id, speed_kmh FROM logistics.fleet_tracking ORDER BY time DESC LIMIT 10"}'
```
## Configuration reference [#configuration-reference]
| Option | Description | Default |
| ----------------- | --------------------------------------------------------- | ------------------------- |
| `base_url` | Base URL of the Arc instance | Required |
| `token` | Bearer token for authentication | Optional |
| `database` | Target database in Arc | `default` |
| `measurement` | Measurement (table) name, supports interpolation | Required |
| `format` | Payload format: `columnar` or `row` | `columnar` |
| `compression` | Compression: `zstd`, `gzip`, or `none` | `zstd` |
| `timestamp_field` | Field name in the message containing the timestamp | empty (uses current time) |
| `timestamp_unit` | Unit of numeric timestamps: `us`, `ms`, `s`, `ns`, `auto` | `auto` |
| `tags_mapping` | Bloblang mapping to extract tags (row format only) | Optional |
| `tls` | TLS configuration | Optional |
| `batching` | Batch policy (`count`, `period`, `byte_size`) | None |
| `max_in_flight` | Maximum parallel batches | `64` |
| `timeout` | HTTP request timeout | `5s` |
## Payload formats [#payload-formats]
### Columnar (default, recommended) [#columnar-default-recommended]
Transposes batched messages into column arrays. This is Arc's fastest ingestion path because it maps directly to Arc's Arrow buffers and avoids per-row overhead.
```yaml
output:
arc:
base_url: http://localhost:8000
database: logistics
measurement: fleet_tracking
format: columnar
compression: zstd
```
Requirement: all messages within a single batch must have the same set of fields. Arc validates this server-side and rejects batches with mismatched columns. Schema evolution across separate batches is fully supported.
### Row [#row]
Sends each message as an individual record with fields and optional tags. Useful when messages within a batch have varying schemas, or when you need per-message tags.
```yaml
output:
arc:
base_url: http://localhost:8000
database: logistics
measurement: fleet_tracking
format: row
tags_mapping: |
root = {"vehicle_id": this.vehicle_id, "fleet": this.fleet, "region": this.region}
```
## Real-world examples [#real-world-examples]
### Kafka events to Arc [#kafka-events-to-arc]
Consume JSON events from a Kafka topic, drop bot traffic, reshape fields, and normalize the timestamp:
```yaml
input:
kafka:
addresses: ["kafka:9092"]
topics: ["app-events"]
consumer_group: "arc-analytics"
pipeline:
processors:
- mapping: |
# Drop bot traffic
root = if this.user_id.has_prefix("bot-") { deleted() }
# Reshape the fields we care about
root.user_id = this.user_id
root.page = this.page
root.duration_ms = this.duration_ms
root.event_type = this.event
output:
arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: analytics
measurement: page_views
format: columnar
timestamp_field: timestamp
timestamp_unit: ms
compression: zstd
batching:
count: 5000
period: 5s
```
### HTTP webhooks to Arc [#http-webhooks-to-arc]
Expose an HTTP endpoint that receives webhooks and writes them to Arc:
```yaml
input:
http_server:
address: "0.0.0.0:8080"
path: /webhook
pipeline:
processors:
- mapping: |
root.source = meta("Http_Header_X_Webhook_Source")
root.received_at = now()
root.payload = this
output:
arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: webhooks
measurement: "${!metadata(\"Http_Header_X_Webhook_Source\")}"
format: row
compression: zstd
batching:
count: 100
period: 2s
```
### MQTT to Arc with transformations [#mqtt-to-arc-with-transformations]
When you want Redpanda Connect's transformation power on top of MQTT (instead of the native MQTT ingestion):
```yaml
input:
mqtt:
urls: ["tcp://broker.example.com:1883"]
topics: ["sensors/#"]
client_id: "arc-connect"
pipeline:
processors:
- mapping: |
root.device_id = meta("mqtt_topic").split("/").index(1)
root.reading = this.value
root.temperature_c = (this.value - 32) * 5 / 9
output:
arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: sensors
measurement: readings
format: columnar
compression: zstd
batching:
count: 1000
period: 1s
```
### Multi-destination fan-out [#multi-destination-fan-out]
Send the same events to Arc and Kafka simultaneously:
```yaml
output:
broker:
pattern: fan_out
outputs:
- arc:
base_url: http://localhost:8000
token: "${ARC_TOKEN}"
database: events
measurement: user_actions
format: columnar
- kafka:
addresses: ["kafka:9092"]
topic: processed-events
```
## Dynamic measurement routing [#dynamic-measurement-routing]
The `measurement` field supports Redpanda Connect's Bloblang interpolation. Messages with different types can be routed to different Arc tables in a single pipeline:
```yaml
output:
arc:
base_url: http://localhost:8000
database: telemetry
# Messages with {"asset_type": "truck", ...} go to the "truck" table
# Messages with {"asset_type": "drone", ...} go to the "drone" table
measurement: ${!json("asset_type")}
```
Or route from message metadata (e.g., from Kafka headers, HTTP headers, or MQTT topics):
```yaml
output:
arc:
base_url: http://localhost:8000
database: telemetry
measurement: ${!metadata("measurement")}
```
## Bloblang transformations [#bloblang-transformations]
[Bloblang](https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/) is Redpanda Connect's built-in mapping language. A few patterns that come up when writing to Arc:
### Drop messages conditionally [#drop-messages-conditionally]
```yaml
processors:
- mapping: |
root = if this.value == null { deleted() }
```
### Flatten nested structures [#flatten-nested-structures]
```yaml
processors:
- mapping: |
root.device_id = this.device.id
root.device_model = this.device.model
root.reading = this.payload.reading
```
### Parse timestamps from strings [#parse-timestamps-from-strings]
```yaml
processors:
- mapping: |
root.event_time = this.timestamp.ts_parse("2006-01-02T15:04:05Z")
root.event_name = this.event
```
### Enrich with static or derived fields [#enrich-with-static-or-derived-fields]
```yaml
processors:
- mapping: |
root = this
root.region = env("DEPLOY_REGION")
root.ingested_at = now()
```
## Querying the data [#querying-the-data]
Once data is in Arc, query it with standard SQL:
```sql
-- Latest records per vehicle
SELECT vehicle_id, lat, lon, speed_kmh, time
FROM logistics.fleet_tracking
WHERE time > NOW() - INTERVAL '1 hour'
ORDER BY time DESC
LIMIT 100;
-- Average speed by vehicle over the last 24h
SELECT
vehicle_id,
AVG(speed_kmh) as avg_speed,
MAX(speed_kmh) as max_speed,
COUNT(*) as reading_count
FROM logistics.fleet_tracking
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY vehicle_id
ORDER BY avg_speed DESC;
-- Hourly throughput of ingested events
SELECT
time_bucket(INTERVAL '1 hour', time) as hour,
COUNT(*) as events
FROM analytics.page_views
WHERE time > NOW() - INTERVAL '7 days'
GROUP BY hour
ORDER BY hour DESC;
```
## Performance tuning [#performance-tuning]
### Batch size [#batch-size]
Arc's columnar format is significantly more efficient with larger batches. Tune `batching.count` and `batching.period` based on your volume.
| Volume | Recommended `batching.count` |
| --------------------- | ---------------------------- |
| Low (\<1K msg/sec) | 100 – 500 |
| Medium (1K – 10K/sec) | 1000 – 5000 |
| High (>10K/sec) | 5000 – 10000 |
### Max in flight [#max-in-flight]
`max_in_flight` controls how many batches can be sent concurrently. Default is `64`. For very high throughput, increase it along with the Arc server's resources:
```yaml
output:
arc:
max_in_flight: 128
batching:
count: 5000
period: 1s
```
### Compression choice [#compression-choice]
* **`zstd`** (default) — Best decompression performance on the Arc server. Recommended for most workloads.
* **`gzip`** — Slightly smaller payloads but higher CPU. Use if the Arc server is I/O bound and CPU is plentiful.
* **`none`** — Only useful for debugging or when running on localhost with very small payloads.
### Format choice [#format-choice]
Prefer `columnar` whenever batches share a consistent schema. It is significantly faster end-to-end. Use `row` only when you need per-message tags or flexible per-message fields.
## Troubleshooting [#troubleshooting]
### 401 Unauthorized [#401-unauthorized]
The Arc token is missing, invalid, or not being expanded by the shell.
```yaml
output:
arc:
token: "${ARC_TOKEN}" # Make sure ARC_TOKEN is exported in your env
```
Test the token directly:
```bash
curl -H "Authorization: Bearer $ARC_TOKEN" http://localhost:8000/api/v1/query \
-d '{"sql": "SHOW DATABASES"}'
```
### 400 Bad Request with "column length mismatch" [#400-bad-request-with-column-length-mismatch]
Columnar format requires all messages in a batch to share the same set of fields. If some messages have extra or missing fields, Arc rejects the batch.
Options:
* Switch to `format: row` if messages have varying schemas
* Add a Bloblang step that normalizes fields before the output
* Reduce batch size so each batch is more homogeneous
### Messages written but nothing queryable [#messages-written-but-nothing-queryable]
Arc buffers data in memory before flushing to Parquet (default 5 seconds). If you're checking immediately after writing, wait a few seconds and try again. For very small batches in local dev, set:
```yaml
batching:
count: 10
period: 1s
```
### Measurement name rejected [#measurement-name-rejected]
Arc validates measurement names (alphanumeric, underscores, hyphens, max 64 chars, must start with a letter). If you're using interpolation, make sure the value is clean:
```yaml
measurement: ${!json("type").string()}
```
### Timestamps in the wrong unit [#timestamps-in-the-wrong-unit]
If your source produces timestamps in milliseconds but Arc is interpreting them as something else, set `timestamp_unit` explicitly:
```yaml
timestamp_field: ts
timestamp_unit: ms # us | ms | s | ns | auto
```
The `auto` default detects the unit from magnitude, which is usually correct but fails for edge cases (e.g. very small timestamps from the 1970s).
## Resources [#resources]
* [Arc output plugin source](https://github.com/redpanda-data/connect/tree/main/internal/impl/arc)
* [Arc output reference docs](https://docs.redpanda.com/redpanda-connect/components/outputs/arc/)
* [Redpanda Connect documentation](https://docs.redpanda.com/redpanda-connect/about/)
* [Bloblang language reference](https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/)
* [Basekick blog post on the integration](https://basekick.net/blog/arc-redpanda-connect-output-plugin?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)
## Next steps [#next-steps]
* Pair with [Grafana](/arc-enterprise/integrations/grafana/) to visualize the data Redpanda Connect ingests
* Use [Arc's native MQTT](/arc-enterprise/integrations/mqtt/) when you don't need transformations
* Use [Telegraf](/arc-enterprise/integrations/telegraf/) for system/infrastructure metrics
# Apache Superset Integration (/arc-enterprise/integrations/superset)
Connect Arc to Apache Superset for interactive dashboards and visualizations.
## Overview [#overview]
Arc provides a native SQLAlchemy dialect for Apache Superset, enabling:
* Full analytical SQL query support
* Multi-database schema support
* Cross-database joins
* Time-series visualizations
* Interactive dashboards
## Installation [#installation]
### Option 1: install in existing Superset [#option-1-install-in-existing-superset]
```bash
# Activate Superset environment
source venv/bin/activate
# Install Arc dialect
pip install arc-superset-dialect
```
### Option 2: Docker with Arc pre-configured [#option-2-docker-with-arc-pre-configured]
```bash
# Clone Arc Superset dialect repo
git clone https://github.com/basekick-labs/arc-superset-dialect.git
cd arc-superset-dialect
# Build and run
docker build -t superset-arc .
docker run -d \
-p 8088:8088 \
--name superset-arc \
superset-arc
```
Access Superset at `http://localhost:8088` (admin/admin)
## Connecting to Arc [#connecting-to-arc]
### 1. Add database connection [#1-add-database-connection]
In Superset UI:
1. Click **Settings** → **Database Connections**
2. Click **+ Database**
3. Select **Other** from database list
4. Enter connection string
### 2. Connection string format [#2-connection-string-format]
```text
arc://{api_token}@{host}:{port}/{database}
```
**Example:**
```text
arc://YourAPITokenHere@localhost:8000/default
```
### 3. Test connection [#3-test-connection]
Click **Test Connection** to verify Arc is reachable.
## Multi-database support [#multi-database-support]
Arc databases appear as **schemas** in Superset:
```yaml
Connection: arc://token@localhost:8000/default
Schemas available:
├── default
│ ├── cpu
│ ├── mem
│ └── disk
├── production
│ ├── cpu
│ └── mem
└── staging
├── cpu
└── mem
```
### Querying different databases [#querying-different-databases]
```sql
-- Query default database
SELECT * FROM default.cpu LIMIT 10;
-- Query production database
SELECT * FROM production.cpu LIMIT 10;
-- Cross-database query
SELECT
p.timestamp,
p.host,
p.usage_idle as prod_cpu,
s.usage_idle as staging_cpu
FROM production.cpu p
JOIN staging.cpu s ON p.timestamp = s.timestamp AND p.host = s.host
WHERE p.timestamp > NOW() - INTERVAL 1 HOUR;
```
## Creating charts [#creating-charts]
### Time-series line chart [#time-series-line-chart]
**SQL Query:**
```sql
SELECT
time_bucket(INTERVAL '5 minutes', timestamp) as time,
host,
AVG(usage_idle) as avg_idle
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 6 HOUR
GROUP BY time, host
ORDER BY time DESC;
```
**Chart Configuration:**
* **Chart Type**: Line Chart
* **Time Column**: time
* **Metrics**: avg\_idle
* **Group By**: host
### CPU vs memory correlation [#cpu-vs-memory-correlation]
**SQL Query:**
```sql
SELECT
c.timestamp,
c.host,
c.usage_idle as cpu_idle,
m.used_percent as mem_used
FROM default.cpu c
JOIN default.mem m ON c.timestamp = m.timestamp AND c.host = m.host
WHERE c.timestamp > NOW() - INTERVAL 1 HOUR
ORDER BY c.timestamp DESC;
```
**Chart Configuration:**
* **Chart Type**: Mixed Chart (Line + Bar)
* **X-axis**: timestamp
* **Y-axis 1**: cpu\_idle
* **Y-axis 2**: mem\_used
### Top hosts by CPU usage [#top-hosts-by-cpu-usage]
**SQL Query:**
```sql
SELECT
host,
AVG(usage_user + usage_system) as avg_usage,
MAX(usage_user + usage_system) as max_usage
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY host
ORDER BY avg_usage DESC
LIMIT 10;
```
**Chart Configuration:**
* **Chart Type**: Bar Chart
* **X-axis**: host
* **Y-axis**: avg\_usage
* **Sort**: Descending
### Heatmap - host activity [#heatmap---host-activity]
**SQL Query:**
```sql
SELECT
DATE_TRUNC('hour', timestamp) as hour,
host,
AVG(100 - usage_idle) as cpu_activity
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 7 DAY
GROUP BY hour, host;
```
**Chart Configuration:**
* **Chart Type**: Heatmap
* **X-axis**: hour
* **Y-axis**: host
* **Color**: cpu\_activity
## Creating dashboards [#creating-dashboards]
### 1. Create dashboard [#1-create-dashboard]
1. Click **Dashboards** → **+ Dashboard**
2. Name it: "System Monitoring"
3. Click **Edit Dashboard**
### 2. Add charts [#2-add-charts]
Drag and drop charts from the chart list or create new ones.
### 3. Add filters [#3-add-filters]
```sql
-- Host filter
SELECT DISTINCT host FROM default.cpu ORDER BY host;
-- Time range filter
-- Use Superset's built-in time range filter
```
### 4. Dashboard layout [#4-dashboard-layout]
Example monitoring dashboard layout:
```text
┌─────────────────────────────────────────┐
│ System Overview - Last 24 Hours │
├─────────────────┬───────────────────────┤
│ │ │
│ CPU Usage │ Memory Usage │
│ (Line Chart) │ (Line Chart) │
│ │ │
├─────────────────┼───────────────────────┤
│ │ │
│ Top 10 Hosts │ Disk I/O │
│ (Bar Chart) │ (Area Chart) │
│ │ │
├─────────────────┴───────────────────────┤
│ │
│ Host Activity Heatmap (7 days) │
│ (Heatmap) │
│ │
└─────────────────────────────────────────┘
```
## Advanced features [#advanced-features]
### Custom SQL [#custom-sql]
Superset supports the full analytical SQL dialect:
```sql
-- Window functions
SELECT
timestamp,
host,
usage_idle,
AVG(usage_idle) OVER (
PARTITION BY host
ORDER BY timestamp
ROWS BETWEEN 5 PRECEDING AND CURRENT ROW
) as moving_avg
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 1 HOUR;
-- CTEs (Common Table Expressions)
WITH hourly_avg AS (
SELECT
DATE_TRUNC('hour', timestamp) as hour,
host,
AVG(usage_idle) as avg_idle
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY hour, host
)
SELECT * FROM hourly_avg
WHERE avg_idle < 50
ORDER BY hour DESC;
-- Percentiles
SELECT
host,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY usage_idle) as p50,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY host;
```
### Alerts [#alerts]
Configure alerts in Superset:
1. Go to **Settings** → **Alerts & Reports**
2. Click **+ Alert**
3. Configure:
* **Chart**: Select your chart
* **Condition**: Greater than, Less than, etc.
* **Threshold**: Value
* **Recipients**: Email addresses
* **Schedule**: Cron expression
**Example Alert - High CPU Usage:**
```sql
SELECT
host,
AVG(100 - usage_idle) as cpu_usage
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 5 MINUTE
GROUP BY host
HAVING AVG(100 - usage_idle) > 80;
```
Alert when query returns rows (CPU > 80%)
### Scheduled reports [#scheduled-reports]
Email dashboards on a schedule:
1. Go to **Dashboards** → Your Dashboard
2. Click **...** → **Set up email report**
3. Configure:
* **Recipients**: Email list
* **Schedule**: Daily at 8 AM
* **Format**: PNG or PDF
## Performance tips [#performance-tips]
### 1. Use time filters [#1-use-time-filters]
Always filter by time to reduce data scanned:
```sql
-- Good: Time filter
WHERE timestamp > NOW() - INTERVAL 24 HOUR
-- Bad: No filter (scans all data)
SELECT * FROM default.cpu
```
### 2. Limit result size [#2-limit-result-size]
```sql
-- Add LIMIT to exploratory queries
SELECT * FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 1 HOUR
LIMIT 1000;
```
### 3. Enable query caching [#3-enable-query-caching]
When Arc reads from S3-compatible storage, caching fetched blocks speeds up the
repeated queries a dashboard produces. In Arc's `arc.toml`:
```toml
[query]
enable_s3_cache = true
s3_cache_size = "128MB"
s3_cache_ttl_seconds = 3600
```
See [query caching](/arc-enterprise/advanced/caching/) for the caches Arc keeps on the
query path.
### 4. Use materialized queries [#4-use-materialized-queries]
For slow dashboards, create materialized views:
```sql
-- Pre-aggregate data
CREATE TABLE default.cpu_hourly AS
SELECT
DATE_TRUNC('hour', timestamp) as hour,
host,
AVG(usage_idle) as avg_idle,
MAX(usage_idle) as max_idle,
MIN(usage_idle) as min_idle
FROM default.cpu
GROUP BY hour, host;
-- Query materialized data
SELECT * FROM default.cpu_hourly
WHERE hour > NOW() - INTERVAL 7 DAY;
```
### 5. Optimize chart SQL [#5-optimize-chart-sql]
```sql
-- Good: Aggregate first
SELECT
DATE_TRUNC('hour', timestamp) as hour,
AVG(usage_idle) as avg_idle
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY hour;
-- Bad: Return all rows
SELECT timestamp, usage_idle
FROM default.cpu
WHERE timestamp > NOW() - INTERVAL 24 HOUR;
-- Then aggregate in Superset (slow)
```
## Troubleshooting [#troubleshooting]
### Connection refused [#connection-refused]
```bash
# Check Arc is running
curl http://localhost:8000/health
# Verify token
curl -H "Authorization: Bearer $ARC_TOKEN" http://localhost:8000/auth/verify
```
### No schemas showing [#no-schemas-showing]
```sql
-- Verify databases exist
SHOW DATABASES;
-- Check tables in database
SHOW TABLES;
```
### Slow queries [#slow-queries]
```bash
# Check compaction status
curl http://localhost:8000/api/compaction/status
# Manually trigger compaction
curl -X POST http://localhost:8000/api/compaction/trigger \
-H "Authorization: Bearer $ARC_TOKEN"
```
### Token expired [#token-expired]
Create a new token:
```bash
# Docker
docker exec -it arc-api python3 -c "
from api.auth import AuthManager
auth = AuthManager(db_path='/data/arc.db')
token = auth.create_token('superset', description='Superset connection')
print(token)
"
# Native
python3 -c "
from api.auth import AuthManager
auth = AuthManager(db_path='./data/arc.db')
token = auth.create_token('superset', description='Superset connection')
print(token)
"
```
Update connection string in Superset with new token.
## Suggested dashboards [#suggested-dashboards]
Two dashboard layouts that work well against Arc Enterprise measurements. Build them from the queries below.
### System monitoring dashboard [#system-monitoring-dashboard]
**Panels:**
* CPU Usage by Host (last 24h)
* Memory Usage Trends
* Disk I/O Operations
* Network Traffic
* Top 10 Busiest Hosts
* System Health Heatmap
### IoT sensor dashboard [#iot-sensor-dashboard]
**Panels:**
* Temperature Trends
* Sensor Online/Offline Status
* Alert History
* Anomaly Detection
* Geographic Distribution
## Resources [#resources]
* **[Arc Superset Dialect GitHub](https://github.com/basekick-labs/arc-superset-dialect)**
* **[PyPI Package](https://pypi.org/project/arc-superset-dialect/)**
* **[Superset Documentation](https://superset.apache.org/docs/intro)**
* **[DuckDB SQL Reference](https://duckdb.org/docs/sql/introduction)**
## Next steps [#next-steps]
* **[Query API Reference](/arc-enterprise/api-reference/overview/)**
* **[SQL Query Guide](/arc/guides/querying/)**
# Telegraf Integration (/arc-enterprise/integrations/telegraf)
Use Telegraf to collect system metrics and send them directly to Arc using the native Arc output plugin.
## Overview [#overview]
Arc provides a native Telegraf output plugin that sends metrics in MessagePack columnar format for maximum performance. The plugin supports gzip compression and integrates seamlessly with Arc's multi-database architecture.
**Benefits:**
* Native MessagePack columnar format for high-throughput ingestion
* Built-in gzip compression
* Direct database targeting
* All 300+ Telegraf input plugins supported
* Full analytical SQL support
## Prerequisites [#prerequisites]
* **Telegraf 1.37 or higher** (required for Arc output plugin)
* Arc server running and accessible
* Arc API token
## Quick start [#quick-start]
### 1. Install Telegraf [#1-install-telegraf]
```bash
# Ubuntu/Debian
wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
echo "deb https://repos.influxdata.com/ubuntu focal stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt update && sudo apt install telegraf
# macOS
brew install telegraf
# Or download from https://portal.influxdata.com/downloads/
```
Verify you have Telegraf 1.37+:
```bash
telegraf --version
```
### 2. Configure Telegraf for Arc [#2-configure-telegraf-for-arc]
Edit `/etc/telegraf/telegraf.conf`:
```toml
# Arc Output Plugin
[[outputs.arc]]
# Arc MessagePack endpoint
url = "http://localhost:8000/api/v1/write/msgpack"
# Arc API token
api_key = "$ARC_TOKEN"
# Enable gzip compression (recommended)
content_encoding = "gzip"
# Target database in Arc
database = "telegraf"
```
### 3. Enable input plugins [#3-enable-input-plugins]
```toml
# System metrics
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.mem]]
[[inputs.net]]
interfaces = ["eth*", "en*"]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
```
### 4. Start Telegraf [#4-start-telegraf]
```bash
# Start service
sudo systemctl start telegraf
# Enable on boot
sudo systemctl enable telegraf
# Check status
sudo systemctl status telegraf
# View logs
sudo journalctl -u telegraf -f
```
### 5. Verify data in Arc [#5-verify-data-in-arc]
```bash
# Check measurements
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SHOW TABLES FROM telegraf", "format": "json"}'
# Query CPU data
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM telegraf.cpu ORDER BY time DESC LIMIT 10", "format": "json"}'
```
## Configuration examples [#configuration-examples]
### Minimal configuration [#minimal-configuration]
```toml
[agent]
interval = "10s"
flush_interval = "10s"
[[outputs.arc]]
url = "http://localhost:8000/api/v1/write/msgpack"
api_key = "$ARC_TOKEN"
content_encoding = "gzip"
database = "telegraf"
[[inputs.cpu]]
[[inputs.mem]]
[[inputs.disk]]
```
### High-performance configuration [#high-performance-configuration]
```toml
[agent]
interval = "10s"
flush_interval = "10s"
metric_batch_size = 5000 # Larger batches for higher throughput
metric_buffer_limit = 50000 # Buffer more metrics
[[outputs.arc]]
url = "http://localhost:8000/api/v1/write/msgpack"
api_key = "$ARC_TOKEN"
content_encoding = "gzip"
database = "metrics"
# Enable all system metrics
[[inputs.cpu]]
percpu = true
totalcpu = true
[[inputs.disk]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.kernel]]
[[inputs.diskio]]
```
### Multi-environment configuration [#multi-environment-configuration]
```toml
# Production metrics → production database
[[outputs.arc]]
url = "https://arc-prod.example.com/api/v1/write/msgpack"
api_key = "$ARC_PROD_TOKEN"
content_encoding = "gzip"
database = "production"
# Staging metrics → staging database
[[outputs.arc]]
url = "https://arc-staging.example.com/api/v1/write/msgpack"
api_key = "$ARC_STAGING_TOKEN"
content_encoding = "gzip"
database = "staging"
```
## Available input plugins [#available-input-plugins]
### System metrics [#system-metrics]
```toml
# CPU usage by core
[[inputs.cpu]]
percpu = true
totalcpu = true
# Memory usage
[[inputs.mem]]
# Disk usage and I/O
[[inputs.disk]]
[[inputs.diskio]]
# Network statistics
[[inputs.net]]
# Process information
[[inputs.processes]]
# System load
[[inputs.system]]
# Kernel statistics
[[inputs.kernel]]
# Swap usage
[[inputs.swap]]
```
### Docker monitoring [#docker-monitoring]
```toml
[[inputs.docker]]
endpoint = "unix:///var/run/docker.sock"
gather_services = false
container_names = []
timeout = "5s"
perdevice = true
total = true
```
### PostgreSQL monitoring [#postgresql-monitoring]
```toml
[[inputs.postgresql]]
address = "postgres://user:pass@localhost/dbname?sslmode=disable"
databases = ["mydb"]
```
### Redis monitoring [#redis-monitoring]
```toml
[[inputs.redis]]
servers = ["tcp://localhost:6379"]
```
### NGINX monitoring [#nginx-monitoring]
```toml
[[inputs.nginx]]
urls = ["http://localhost/nginx_status"]
```
### HTTP response time [#http-response-time]
```toml
[[inputs.http_response]]
urls = [
"https://example.com",
"https://api.example.com/health"
]
method = "GET"
response_timeout = "5s"
follow_redirects = true
```
### Custom exec plugin [#custom-exec-plugin]
```toml
[[inputs.exec]]
commands = ["/usr/local/bin/custom_metrics.sh"]
timeout = "5s"
data_format = "influx"
```
## Querying Telegraf data in Arc [#querying-telegraf-data-in-arc]
### View available measurements [#view-available-measurements]
```sql
SHOW TABLES FROM telegraf;
```
**Common measurements from Telegraf:**
* `cpu` - CPU usage per core
* `mem` - Memory statistics
* `disk` - Disk usage
* `diskio` - Disk I/O stats
* `net` - Network statistics
* `processes` - Process counts
* `system` - System load
* `docker` - Container metrics
### CPU usage analysis [#cpu-usage-analysis]
```sql
-- Average CPU usage by host (last hour)
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
host,
AVG(usage_user + usage_system) as avg_usage
FROM telegraf.cpu
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, host
ORDER BY bucket DESC;
-- Highest CPU usage instances
SELECT
host,
cpu,
MAX(usage_user + usage_system) as max_usage
FROM telegraf.cpu
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY host, cpu
ORDER BY max_usage DESC
LIMIT 10;
```
### Memory analysis [#memory-analysis]
```sql
-- Memory usage trend
SELECT
time_bucket(INTERVAL '1 hour', time) as hour,
host,
AVG(used_percent) as avg_mem_usage
FROM telegraf.mem
WHERE time > NOW() - INTERVAL '7 days'
GROUP BY hour, host
ORDER BY hour DESC;
-- Hosts with high memory usage
SELECT
host,
AVG(used_percent) as avg_usage,
MAX(used_percent) as max_usage
FROM telegraf.mem
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY host
HAVING AVG(used_percent) > 80
ORDER BY avg_usage DESC;
```
### Disk analysis [#disk-analysis]
```sql
-- Disk usage by mount point
SELECT
host,
path,
AVG(used_percent) as avg_usage
FROM telegraf.disk
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY host, path
ORDER BY avg_usage DESC;
-- Disk I/O operations
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
name,
SUM(reads) as total_reads,
SUM(writes) as total_writes
FROM telegraf.diskio
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, name
ORDER BY bucket DESC;
```
### Network analysis [#network-analysis]
```sql
-- Network throughput
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
interface,
SUM(bytes_sent) / (5 * 60) as bytes_sent_per_sec,
SUM(bytes_recv) / (5 * 60) as bytes_recv_per_sec
FROM telegraf.net
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, interface
ORDER BY bucket DESC;
```
### Docker container monitoring [#docker-container-monitoring]
```sql
-- Container CPU usage
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
container_name,
AVG(usage_percent) as avg_cpu
FROM telegraf.docker_container_cpu
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, container_name
ORDER BY bucket DESC;
-- Container memory usage
SELECT
container_name,
AVG(usage) as avg_memory_bytes,
MAX(usage) as max_memory_bytes
FROM telegraf.docker_container_mem
WHERE time > NOW() - INTERVAL '24 hours'
GROUP BY container_name
ORDER BY avg_memory_bytes DESC;
```
## Performance tuning [#performance-tuning]
### Optimize batch size [#optimize-batch-size]
```toml
[agent]
metric_batch_size = 5000 # Send 5000 metrics per request
metric_buffer_limit = 50000 # Buffer 50k metrics before dropping
```
**Guidelines:**
* **Low volume** (\<1000 metrics/sec): batch\_size = 1000
* **Medium volume** (1000-10000/sec): batch\_size = 5000
* **High volume** (>10000/sec): batch\_size = 10000
### Collection intervals [#collection-intervals]
```toml
[agent]
interval = "10s" # Collect every 10 seconds
flush_interval = "10s" # Send every 10 seconds
```
For real-time monitoring, use smaller intervals (5s). For cost optimization, use larger intervals (60s).
### Enable compression [#enable-compression]
Always use gzip compression for better network efficiency:
```toml
[[outputs.arc]]
content_encoding = "gzip" # Compress payloads
```
## Troubleshooting [#troubleshooting]
### Telegraf can't connect to Arc [#telegraf-cant-connect-to-arc]
```bash
# Test Arc connectivity
curl http://localhost:8000/health
# Test with token
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT 1", "format": "json"}'
# Check Telegraf logs
sudo journalctl -u telegraf -f | grep -i error
```
### No data appearing [#no-data-appearing]
```bash
# Verify Telegraf is running
sudo systemctl status telegraf
# Check Telegraf config syntax
telegraf --config /etc/telegraf/telegraf.conf --test
# Check Arc received data
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT COUNT(*) FROM telegraf.cpu", "format": "json"}'
```
### Authentication errors [#authentication-errors]
Ensure your API key is correct in the configuration:
```toml
[[outputs.arc]]
api_key = "$ARC_TOKEN" # Must be a valid Arc API token
```
### Metrics being dropped [#metrics-being-dropped]
```bash
# Increase buffer
[agent]
metric_buffer_limit = 100000 # Increase from default
# Check Arc health
curl http://localhost:8000/health
```
### Version check [#version-check]
The Arc output plugin requires Telegraf 1.37+:
```bash
telegraf --version
# Telegraf 1.37.0 (or higher required)
```
## Dashboard integration [#dashboard-integration]
### Grafana with Arc [#grafana-with-arc]
Use the [Arc Grafana datasource plugin](/arc-enterprise/integrations/grafana/) for native integration:
```text
1. Install the Arc datasource from Grafana marketplace
2. Configure connection to your Arc instance
3. Use analytical SQL in your dashboard panels
```
See [Grafana Integration](/arc-enterprise/integrations/grafana/) for detailed setup instructions.
## Best practices [#best-practices]
### 1. Use tags efficiently [#1-use-tags-efficiently]
```toml
[global_tags]
environment = "production"
datacenter = "us-east-1"
region = "us-east"
```
Tags enable powerful GROUP BY queries but increase cardinality.
### 2. Filter unnecessary metrics [#2-filter-unnecessary-metrics]
```toml
[[inputs.cpu]]
percpu = false # Aggregate across CPUs
totalcpu = true
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs"] # Skip temporary filesystems
```
### 3. Use measurement filters [#3-use-measurement-filters]
```toml
[[outputs.arc]]
namepass = ["cpu*", "mem*", "disk*"] # Only send specific metrics
# OR
namedrop = ["docker_*"] # Exclude Docker metrics
```
### 4. Set reasonable collection intervals [#4-set-reasonable-collection-intervals]
```toml
[[inputs.cpu]]
interval = "10s" # Fast-changing metrics
[[inputs.disk]]
interval = "60s" # Slow-changing metrics
```
## Resources [#resources]
* **[Telegraf Documentation](https://docs.influxdata.com/telegraf/)**
* **[Telegraf Plugins](https://docs.influxdata.com/telegraf/latest/plugins/)**
* **[Arc Query Guide](/arc-enterprise/getting-started/)**
* **[Arc Grafana Integration](/arc-enterprise/integrations/grafana/)**
## Next steps [#next-steps]
* **[Query Telegraf metrics](/arc-enterprise/getting-started/)**
* **[Create Grafana dashboards](/arc-enterprise/integrations/grafana/)**
* **[Optimize performance](/arc-enterprise/configuration/overview/)**
# VS Code Extension (/arc-enterprise/integrations/vscode)
Complete development toolkit for Arc Database directly in Visual Studio Code.
## Overview [#overview]
The Arc Database Manager extension provides a full-featured IDE for working with Arc:
* **Connection Management**: Multiple saved connections with secure token storage
* **SQL IntelliSense**: Auto-completion for tables, columns, and SQL functions
* **Interactive Results**: Export to CSV/JSON/Markdown with automatic chart visualization
* **Arc Notebooks**: Mix SQL and Markdown in `.arcnb` files with parameterized queries
* **Schema Explorer**: Browse databases and tables with context menus
* **Data Ingestion**: CSV import wizard and bulk data generator
* **Alerting**: Create query-based alerts with desktop notifications
* **Query Management**: Automatic history and saved queries
* **Dark Mode**: Automatic theme detection and adaptation
## Installation [#installation]
### From VS Code Marketplace [#from-vs-code-marketplace]
1. Open VS Code
2. Click **Extensions** (Ctrl+Shift+X / Cmd+Shift+X)
3. Search for **"Arc Database Manager"**
4. Click **Install**
Or install directly from the marketplace:
* **[Arc Database Manager on VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=basekick-labs.arc-db-manager)**
### From command line [#from-command-line]
```bash
code --install-extension basekick-labs.arc-db-manager
```
## Quick start [#quick-start]
### 1. Connect to Arc server [#1-connect-to-arc-server]
**Option A: From Status Bar**
1. Click **"Arc: Not Connected"** in the status bar
2. Enter connection details:
* Name: `My Arc Server`
* Host: `localhost`
* Port: `8000`
* Protocol: `http` or `https`
3. Enter your authentication token
**Option B: From Command Palette**
1. Press `Ctrl+Shift+P` / `Cmd+Shift+P`
2. Type `Arc: Connect to Server`
3. Follow the prompts
### 2. Get your API token [#2-get-your-api-token]
```bash
# Docker - check logs for admin token
docker logs 2>&1 | grep "Admin token"
# Or create a new token
curl -X POST http://localhost:8000/api/v1/auth/tokens \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "vscode-extension",
"description": "VS Code extension access"
}'
```
### 3. Start querying [#3-start-querying]
1. Press `Ctrl+Shift+P` → `Arc: New Query`
2. Write your SQL query
3. Press `Ctrl+Enter` / `Cmd+Enter` to execute
**Example Query:**
```sql
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(usage_idle) * -1 + 100 AS cpu_usage,
host
FROM prod.cpu
WHERE cpu = 'cpu-total'
AND time > NOW() - INTERVAL '1 hour'
GROUP BY bucket, host
ORDER BY bucket ASC;
```
## Features [#features]
### SQL IntelliSense [#sql-intellisense]
Get auto-completion for:
* Database names
* Table names
* Column names
* SQL functions
* SQL keywords
**How to use:**
* Start typing and IntelliSense will suggest completions
* Press `Ctrl+Space` to manually trigger suggestions
* Navigate with arrow keys, press `Enter` to accept
### Interactive results view [#interactive-results-view]
After executing a query, results are displayed with:
**Export Options:**
* CSV format
* JSON format
* Markdown tables
**Automatic Visualizations:**
* Time-series data is automatically charted
* Line charts for temporal data
* Theme-aware (adapts to VS Code theme)
**Table Features:**
* Sort by clicking column headers
* Filter rows with search
* View execution time and row count
### Arc notebooks [#arc-notebooks]
Create analysis documents mixing SQL and Markdown in `.arcnb` files.
**Create a Notebook:**
1. Press `Ctrl+Shift+P` → `Arc: New Notebook`
2. Save with `.arcnb` extension
**Notebook Features:**
* Mix Markdown documentation with SQL queries
* Parameterized queries with variable substitution
* Execute cells individually or all at once
* Export to Markdown with results
* Auto-save functionality
**Example Notebook:**
```markdown
# CPU Performance Analysis
This notebook analyzes CPU usage patterns over time.
Variables:
- interval = 1 HOUR
- threshold = 80
- database = prod
## Average CPU Usage
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(usage_user) as avg_cpu,
host
FROM ${database}.cpu
WHERE time > NOW() - INTERVAL ${interval}
AND usage_user > ${threshold}
GROUP BY bucket, host
ORDER BY bucket DESC;
## Results
The query shows periods where CPU exceeded ${threshold}% in the last ${interval}.
```
**Variable Syntax:**
* Define variables in YAML frontmatter
* Reference with `${variable_name}`
* Variables are replaced before execution
### Schema explorer [#schema-explorer]
Browse your Arc databases and tables in the sidebar.
**Features:**
* Hierarchical view of databases and tables
* Connection status indicator
* Visual refresh button
**Right-Click Context Menus:**
**On Tables:**
* **Show Table Schema** - View column names and types
* **Preview Data** - Show first 100 rows
* **Show Table Statistics** - Row count and size
* **Generate SELECT Query** - Create basic query
* **Query Last Hour** - Filter to recent data
* **Query Today** - Filter to today's data
**Example: Show Table Schema**
```text
Right-click table → Show Table Schema
Result:
┌────────────┬──────────┐
│ Column │ Type │
├────────────┼──────────┤
│ time │ TIMESTAMP│
│ host │ VARCHAR │
│ usage_idle │ DOUBLE │
│ usage_user │ DOUBLE │
└────────────┴──────────┘
```
### Data ingestion [#data-ingestion]
#### CSV import wizard [#csv-import-wizard]
Import CSV files directly into Arc with guided setup.
**Steps:**
1. Press `Ctrl+Shift+P` → `Arc: Import CSV`
2. Select your CSV file
3. Configure import settings:
* Auto-detect delimiter and headers
* Select timestamp column
* Choose target database
* Set batch size
**Performance:**
* Uses MessagePack columnar format
* Progress tracking for large files
* Batch processing support
**Example:**
```text
Import Settings:
- File: metrics.csv
- Delimiter: , (auto-detected)
- Timestamp Column: time
- Database: prod
- Measurement: custom_metrics
- Batch Size: 10,000
Result: 250,000 rows imported in 3.2 seconds
```
#### Bulk data generator [#bulk-data-generator]
Generate test data for development and testing.
**Presets:**
1. **CPU Metrics** - System CPU usage data
2. **Memory Metrics** - Memory usage statistics
3. **Network Metrics** - Network traffic data
4. **IoT Sensor Data** - Temperature, humidity sensors
5. **Custom Schema** - Define your own fields
**Steps:**
1. Press `Ctrl+Shift+P` → `Arc: Generate Test Data`
2. Select preset
3. Configure:
* Number of rows
* Target database
* Time range
**Performance:**
* Realistic sample data
* Configurable patterns
### Alerting & monitoring [#alerting--monitoring]
Create alerts based on query results with desktop notifications.
**Create an Alert:**
1. Press `Ctrl+Shift+P` → `Arc: Create Alert`
2. Configure alert:
* Name
* SQL query
* Condition type
* Threshold value
* Check interval
**Condition Types:**
* Greater than
* Less than
* Equals
* Not equals
* Contains
**Example Alert:**
```yaml
Name: High CPU Usage
Query: SELECT AVG(usage_user) as cpu FROM prod.cpu WHERE time > NOW() - INTERVAL '5 minutes'
Condition: greater_than
Threshold: 80
Interval: 60s
```
**Alert Features:**
* Desktop notifications when triggered
* Alert history tracking
* Enable/disable without deletion
* Minimum check interval: 10 seconds
### Query management [#query-management]
**Query History:**
* Every executed query is automatically saved
* View execution time and row counts
* Quick re-run from history
* Search through past queries
**Saved Queries:**
* Bookmark frequently used queries
* Organize by tags or folders
* Quick access from sidebar
**Access:**
1. Open Arc sidebar
2. Navigate to **Query History** or **Saved Queries**
3. Click query to view or re-run
### Token management [#token-management]
Manage Arc authentication tokens directly from VS Code.
**Features:**
* Create new tokens
* Rotate existing tokens
* Delete tokens
* Verify token validity
* Secure storage in system keychain
**Access:**
1. Press `Ctrl+Shift+P`
2. Type `Arc: Manage Tokens`
3. Select action
## Commands [#commands]
Access all commands via Command Palette (`Ctrl+Shift+P` / `Cmd+Shift+P`):
### Connection [#connection]
* `Arc: Connect to Server`
* `Arc: Disconnect`
* `Arc: Verify Token`
* `Arc: Manage Tokens`
### Queries [#queries]
* `Arc: New Query`
* `Arc: Execute Query` (Ctrl+Enter / Cmd+Enter)
* `Arc: Save Current Query`
* `Arc: Open Query History`
### Notebooks [#notebooks]
* `Arc: New Notebook`
* `Arc: Execute Notebook Cell`
* `Arc: Execute All Cells`
* `Arc: Export Notebook to Markdown`
### Data [#data]
* `Arc: Import CSV`
* `Arc: Generate Test Data`
### Alerts [#alerts]
* `Arc: Create Alert`
* `Arc: View Alerts`
* `Arc: Enable/Disable Alert`
### Explorer [#explorer]
* `Arc: Refresh Explorer`
* `Arc: Show Table Schema`
* `Arc: Preview Table Data`
* `Arc: Show Table Statistics`
## Keyboard shortcuts [#keyboard-shortcuts]
| Command | Windows/Linux | macOS |
| --------------- | ------------------------------- | ------------------------------ |
| Execute Query | `Ctrl+Enter` | `Cmd+Enter` |
| New Query | `Ctrl+Shift+P` → Arc: New Query | `Cmd+Shift+P` → Arc: New Query |
| Command Palette | `Ctrl+Shift+P` | `Cmd+Shift+P` |
| Toggle Sidebar | `Ctrl+B` | `Cmd+B` |
## Extension settings [#extension-settings]
Configure extension defaults in VS Code settings:
```json
{
"arc.defaultHost": "localhost",
"arc.defaultPort": 8000,
"arc.defaultProtocol": "http"
}
```
**Available Settings:**
| Setting | Description | Default |
| --------------------- | ----------------------- | ----------- |
| `arc.defaultHost` | Default Arc server host | `localhost` |
| `arc.defaultPort` | Default Arc server port | `8000` |
| `arc.defaultProtocol` | Default protocol | `http` |
## Use cases [#use-cases]
### Development & testing [#development--testing]
**Generate Test Data:**
```text
1. Arc: Generate Test Data
2. Select: CPU Metrics
3. Rows: 100,000
4. Database: dev
5. Time Range: Last 24 hours
Result: Realistic CPU metrics for testing
```
**Query the Data:**
```sql
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(usage_user) as avg_cpu
FROM dev.cpu
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;
```
### Data analysis [#data-analysis]
Create analysis notebooks (`.arcnb`) with:
* Documentation in Markdown
* Parameterized SQL queries
* Inline results and charts
* Export to Markdown reports
**Example Use Case:**
```markdown
# Weekly Performance Report
Variables:
- week_start = 2025-01-10
- database = prod
## CPU Trends
SELECT
DATE_TRUNC('day', time) as day,
AVG(usage_user) as avg_cpu
FROM ${database}.cpu
WHERE time >= '${week_start}'
GROUP BY day
ORDER BY day;
```
### Production monitoring [#production-monitoring]
**Create Alert:**
```yaml
Name: High Memory Usage
Query: SELECT AVG(used_percent) FROM prod.mem WHERE time > NOW() - INTERVAL '5 minutes'
Condition: greater_than
Threshold: 90
Interval: 60s
→ Desktop notification when memory exceeds 90%
```
### Data migration [#data-migration]
**Import CSV Files:**
```text
1. Arc: Import CSV
2. Select file: server_metrics.csv
3. Auto-detect: delimiter, headers
4. Set timestamp column: timestamp
5. Target: prod.imported_metrics
6. Batch size: 10,000
→ Import complete with progress tracking
```
## Performance [#performance]
* **Query Results**: Displays up to 1,000 rows instantly
* **CSV Import**: streams via MessagePack columnar format with progress tracking
* **Data Generator**: produces realistic sample data with configurable patterns
* **Batch Processing**: Handles millions of rows with progress tracking
## Troubleshooting [#troubleshooting]
### Cannot connect to Arc server [#cannot-connect-to-arc-server]
```bash
# 1. Verify Arc is running
curl http://localhost:8000/health
# 2. Check connection details
- Host: localhost
- Port: 8000
- Protocol: http
# 3. Verify token
Arc: Verify Token (from Command Palette)
```
### Query timeout [#query-timeout]
**Solutions:**
1. Add time filters:
```sql
WHERE time > NOW() - INTERVAL '1 hour'
```
2. Add `LIMIT` clause:
```sql
LIMIT 1000
```
3. Check Arc server performance:
```bash
curl http://localhost:8000/api/v1/compaction/trigger \
-H "Authorization: Bearer $ARC_TOKEN"
```
### CSV import fails [#csv-import-fails]
**Common Issues:**
1. **Encoding**: Ensure UTF-8 encoding
2. **Delimiter**: Verify delimiter is correct (auto-detect usually works)
3. **File Size**: Try smaller file first to test
4. **Timestamp Format**: Ensure timestamp column is recognized
**Check Import Settings:**
```text
File encoding: UTF-8
Delimiter: , (comma)
Headers: First row
Timestamp column: time
Format: ISO 8601 or Unix timestamp
```
### Extension not activating [#extension-not-activating]
1. **Check VS Code version**: Requires 1.85.0 or higher
2. **View Output**: View → Output → Arc Database Manager
3. **Reload Window**: Ctrl+Shift+P → Reload Window
4. **Reinstall**: Uninstall and reinstall extension
### IntelliSense not working [#intellisense-not-working]
1. **Refresh Schema**: Right-click in Arc Explorer → Refresh
2. **Reconnect**: Disconnect and reconnect to server
3. **Check Connection**: Ensure server is connected (status bar)
## Requirements [#requirements]
* **VS Code**: Version 1.85.0 or higher
* **Arc Database**: Running instance (v1.0.0+)
* **Authentication Token**: Valid Arc API token
## Release notes [#release-notes]
### 0.2.0 - Latest [#020---latest]
**New Features:**
* Auto-qualified table names in queries
* Right-click queries include database prefix (e.g., `prod.cpu`)
**Improvements:**
* Fixed query generation to read metadata correctly
* All context menu queries now work without manual editing
### 0.1.9 [#019]
**⚠️ Breaking Changes:**
* Updated all API endpoints to `/api/v1/` prefix
* Requires Arc v1.0.0 or later
* Not compatible with pre-v1.0 Arc servers
**Migration:**
1. Upgrade Arc to v1.0.0+
2. Update extension to v0.1.9
3. Reconnect to Arc server
## Resources [#resources]
* **[VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=basekick-labs.arc-db-manager)**
* **[GitHub Repository](https://github.com/basekick-labs/arc-vscode-extension)**
* **[Arc Documentation](https://docs.basekick.net)**
* **[Report Issues](https://github.com/basekick-labs/arc-vscode-extension/issues)**
## Next steps [#next-steps]
* **[Getting Started with Arc](/arc-enterprise/getting-started/)** - Install and configure Arc
* **[Query API Reference](/arc-enterprise/api-reference/overview/)** - Learn Arc SQL
* **[Grafana Integration](/arc-enterprise/integrations/grafana/)** - Build dashboards
* **[Telegraf Integration](/arc-enterprise/integrations/telegraf/)** - Collect system metrics
***
**Enjoy using Arc Database Manager!**
Made with ❤️ by [Basekick Labs](https://github.com/basekick-labs)
# Automated Scheduling (/arc-enterprise/operations/automated-scheduling)
Automatically execute continuous queries and retention policies on configurable schedules. Eliminate manual data lifecycle management and build efficient data pipelines.
## Overview [#overview]
Arc OSS provides [continuous queries](/arc/data-lifecycle/continuous-queries/) and [retention policies](/arc/data-lifecycle/retention-policies/) with manual API-triggered execution. Arc Enterprise adds automatic scheduling — define your schedules once, and Arc handles execution automatically.
**Two schedulers:**
| Scheduler | Purpose | Default Schedule |
| ----------------------- | ----------------------------------------------------- | -------------------------- |
| **CQ Scheduler** | Runs continuous queries at their configured intervals | Per-CQ interval |
| **Retention Scheduler** | Enforces retention policies on a cron schedule | Daily at 3am (`0 3 * * *`) |
## CQ scheduler [#cq-scheduler]
The CQ Scheduler automatically executes continuous queries at their configured intervals. Each continuous query runs independently on its own schedule.
### How it works [#how-it-works]
1. Define continuous queries with intervals via the [CQ API](/arc/data-lifecycle/continuous-queries/)
2. Enable the CQ scheduler (requires enterprise license)
3. Arc automatically executes each CQ at its configured interval
4. Results are written to the destination measurement
### Configuration [#configuration]
The CQ Scheduler is enabled when continuous queries are enabled and a valid enterprise license is present:
```toml
[continuous_query]
enabled = true
```
```bash
ARC_CONTINUOUS_QUERY_ENABLED=true
```
Each continuous query defines its own execution interval when created through the API.
## Retention scheduler [#retention-scheduler]
The Retention Scheduler automatically enforces retention policies on a cron schedule, deleting data that has exceeded its retention period.
### How it works [#how-it-works-1]
1. Define retention policies via the [Retention API](/arc/data-lifecycle/retention-policies/)
2. Enable the retention scheduler (requires enterprise license)
3. Arc evaluates all active policies on the configured schedule
4. Expired data is automatically deleted
### Configuration [#configuration-1]
```toml
[retention]
enabled = true
[scheduler]
retention_schedule = "0 3 * * *" # Cron: daily at 3am
```
```bash
ARC_RETENTION_ENABLED=true
ARC_SCHEDULER_RETENTION_SCHEDULE="0 3 * * *"
```
The schedule uses standard 5-field cron syntax: `minute hour day-of-month month day-of-week`.
| Schedule | Meaning |
| ------------- | ----------------------------- |
| `0 3 * * *` | Daily at 3:00 AM |
| `0 */6 * * *` | Every 6 hours |
| `0 2 * * 0` | Weekly on Sunday at 2:00 AM |
| `30 1 1 * *` | Monthly on the 1st at 1:30 AM |
## Data lifecycle pipeline [#data-lifecycle-pipeline]
Combine CQ and retention scheduling to build a complete data lifecycle pipeline:
```text
Raw Data (1-second resolution)
│
│ CQ: 1-minute aggregation (runs every minute)
▼
1-Minute Data
│
│ CQ: 1-hour aggregation (runs every hour)
▼
1-Hour Data
│
│ CQ: 1-day aggregation (runs daily)
▼
1-Day Data
Retention Schedule (runs daily at 3am):
├── Delete raw data older than 7 days
├── Delete 1-minute data older than 30 days
├── Delete 1-hour data older than 365 days
└── Keep 1-day data indefinitely
```
### Example setup [#example-setup]
**1. Create continuous queries for downsampling:**
```bash
# 1-minute aggregation
curl -X POST http://localhost:8000/api/v1/continuous-queries \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "downsample_1min",
"database": "telemetry",
"source_measurement": "sensors_raw",
"destination_measurement": "sensors_1min",
"query": "SELECT time_bucket('\''1 minute'\'', timestamp) as timestamp, device_id, AVG(temperature) as temperature, MAX(pressure) as pressure FROM sensors_raw WHERE timestamp >= $start AND timestamp < $end GROUP BY 1, 2",
"interval": "1m",
"enabled": true
}'
# 1-hour aggregation
curl -X POST http://localhost:8000/api/v1/continuous-queries \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "downsample_1hr",
"database": "telemetry",
"source_measurement": "sensors_1min",
"destination_measurement": "sensors_1hr",
"query": "SELECT time_bucket('\''1 hour'\'', timestamp) as timestamp, device_id, AVG(temperature) as temperature, MAX(pressure) as pressure FROM sensors_1min WHERE timestamp >= $start AND timestamp < $end GROUP BY 1, 2",
"interval": "1h",
"enabled": true
}'
```
**2. Create retention policies:**
```bash
# Delete raw data after 7 days
curl -X POST http://localhost:8000/api/v1/retention \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "raw_7d",
"database": "telemetry",
"measurement": "sensors_raw",
"retention_days": 7,
"enabled": true
}'
# Delete 1-minute data after 30 days
curl -X POST http://localhost:8000/api/v1/retention \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "1min_30d",
"database": "telemetry",
"measurement": "sensors_1min",
"retention_days": 30,
"enabled": true
}'
```
With enterprise scheduling enabled, these queries and policies run automatically — no cron jobs, no external orchestration.
## Best practices [#best-practices]
1. **Schedule retention during off-peak hours** — File deletion generates I/O. The default 3am schedule avoids impacting daytime workloads.
2. **Add buffer days to retention policies** — Use the `buffer_days` parameter in retention policies to provide a safety margin before deletion.
3. **Test CQ queries manually first** — Before enabling automatic execution, run your continuous query SQL manually to verify correct results.
4. **Combine with tiered storage** — Use [tiered storage](/arc-enterprise/data-lifecycle/tiered-storage/) to move data to cold storage before retention deletes it, keeping long-term archives at low cost.
5. **Monitor CQ execution** — Check Arc logs for CQ execution results and errors. Failed CQ executions are logged at WARN level.
## Next steps [#next-steps]
* [Continuous Queries](/arc/data-lifecycle/continuous-queries/) — Create and manage continuous queries (OSS docs)
* [Retention Policies](/arc/data-lifecycle/retention-policies/) — Create and manage retention policies (OSS docs)
* [Tiered Storage](/arc-enterprise/data-lifecycle/tiered-storage/) — Combine scheduling with tiered storage for optimal cost management
# Operations (/arc-enterprise/operations)
Day-two concerns for a licensed deployment. Automated scheduling is Enterprise-only and is what turns the data-lifecycle features from manual API calls into something that runs unattended.
Telemetry and profiling behave identically to OSS, but in a cluster you configure them per node — pprof in particular binds to loopback on the node you are diagnosing, not to a cluster endpoint.
# Upgrading pre-26.06.1 API tokens (/arc-enterprise/operations/legacy-token-upgrade)
Applies to Arc Enterprise clusters that carry API-created tokens from before 26.06.1. Before that release, every node kept its own SQLite auth database, so a token created through the API existed only on the node that issued it. 26.06.1 moved token creation and revocation onto the replicated Raft path, but pre-existing tokens were not migrated. They keep working on their home node until an operator replaces them.
Plan a maintenance window and preserve administrative access throughout. Bootstrap tokens configured via `ARC_AUTH_BOOTSTRAP_TOKEN` with the same value on every node are unaffected.
The cluster revoke endpoint (`POST /api/v1/auth/tokens/:id/revoke`) resolves the ID in the replicated FSM, which never held pre-26.06.1 tokens. Two failure modes follow: an unknown ID is an idempotent no-op even though the API reports success, so the legacy token keeps working on its home node; and if a legacy `AUTOINCREMENT` ID collides with a replicated token ID, the same request revokes the replacement token cluster-wide instead. Legacy tokens are removed locally, with the node stopped, as described below.
## Maintenance procedure [#maintenance-procedure]
1. **Inventory active tokens on every node** before upgrading: record node, local token identity, name, permissions, owner, and consumers. Preserve enough non-secret identity to distinguish legacy rows from their replacements; a numeric ID alone is insufficient. Do not export hashes or plaintext into the inventory.
2. **Back up each node's auth database** with the node stopped: the file configured by `auth.db_path`, including SQLite `-wal` and `-shm` sidecars when present. Keep the backup access-restricted. Never copy or delete the live database while the node is writing.
3. **Upgrade all nodes** to 26.06.1 or later. Restore stable leadership and full membership before issuing replicated tokens.
4. **Validate the replicated path** with a short-lived token created through `POST /api/v1/auth/tokens`. Confirm it authenticates on every node and that `arc_cluster_auth_apply_create_total` increments to match on every node. Revoke this new token through `POST /api/v1/auth/tokens/:id/revoke` and verify it is rejected everywhere.
5. **Re-issue each inventoried token** through `POST /api/v1/auth/tokens` on any node, preserving its scope and permissions. Capture the returned plaintext once into the approved secret store. Verify it on every node before rotating any consumer. If an ID collision prevents materialisation, stop and resolve the divergence (see below) before proceeding.
6. **Rotate downstream consumers** (CI secrets, SDKs, dashboards) to the replacement. Verify each consumer against multiple nodes.
7. **Remove the old token locally on every node that holds it.** Stop the node, take a fresh backup, and remove only the positively identified legacy rows from its local auth database, or use the supported rebuild from replicated FSM state. Do not select rows by ID alone, and do not remove a replicated replacement. If identity cannot be established, stop and seek support rather than guessing. Restart and verify that replacement tokens still authenticate and legacy tokens do not. Keep quorum available during node maintenance, or use a planned full-cluster outage. Repeat steps 5 through 7 one token at a time.
8. **Verify completion**: every active consumer uses a replacement, apply counters converge across nodes, `arc_cluster_auth_rejected_total` is stable, and every legacy token is rejected on every node.
## Divergence and rollback [#divergence-and-rollback]
On a collision or a growing `arc_cluster_auth_rejected_total`, stop the affected node and preserve its backup. Reconcile only the conflicting legacy rows, or rebuild the node's local auth database from authoritative FSM state before it rejoins. Never remove Raft state to repair the SQLite cache, and do not invent SQL cleanup by token ID.
Restoring a pre-upgrade database can re-enable legacy credentials and diverge from replicated state. Treat rollback as a coordinated maintenance operation: keep the affected node out of client traffic until its state and credentials have been reconciled and verified. Retain backups until the completed migration is accepted.
## Background [#background]
The [26.06.1 release notes](https://github.com/Basekick-Labs/arc/blob/main/RELEASE_NOTES_2026.06.1.md) cover the auth replication design, ID-collision detection, and the related metrics. The source issue is [#457](https://github.com/Basekick-Labs/arc/issues/457); its original suggestion to revoke legacy tokens through the API does not apply to unmigrated rows.
# Profiling with pprof (/arc-enterprise/operations/profiling)
Arc exposes Go's built-in `net/http/pprof` profiler — heap, goroutine, CPU, allocations, blocking, mutex, and execution-trace endpoints — for diagnosing memory pressure, hot CPU paths, goroutine leaks, and deadlocks in production. The endpoints are **opt-in** and bound to `localhost` by default; exposing them anywhere else requires a deliberate two-step configuration.
The opt-in pprof listener ships in Arc v26.06.1 ([PR #443](https://github.com/Basekick-Labs/arc/pull/443), [GHSA-j93g-rp6m-j32m](https://github.com/Basekick-Labs/arc/security/advisories/GHSA-j93g-rp6m-j32m)). Prior versions registered pprof on the public API port without authentication — upgrade and adopt the env-var gate below.
A reachable `/debug/pprof/*` endpoint leaks process internals: in-flight SQL strings and msgpack records (via heap dumps), goroutine stacks, environment variables on some Go versions, and lets any caller pin a CPU core for arbitrary seconds via `/debug/pprof/profile?seconds=N`. Treat the pprof listener like a root shell — bind to loopback, restrict by firewall, and turn it off when you're done debugging.
## Why pprof is off by default [#why-pprof-is-off-by-default]
Pre-v26.06.1, `/debug/pprof/*` was mounted on Arc's public Fiber app — no token, no allowlist. An unauthenticated network caller could fetch heap dumps containing recent query text and ingested records. The hardening PR removed pprof from the public app entirely and moved it to a separate listener that only starts when the `ARC_DEBUG_PPROF` env var is set.
The new design has three properties:
1. **Off by default** — `ARC_DEBUG_PPROF` unset means no socket is opened, no goroutine is spawned, the endpoints don't exist on Arc's process.
2. **Loopback-bound by default** — even with `ARC_DEBUG_PPROF=1`, the listener binds to `127.0.0.1:6060` unless you explicitly override.
3. **Two-step opt-in for non-loopback** — binding to any non-loopback address (`0.0.0.0:6060`, a public IP, etc.) requires both `ARC_DEBUG_PPROF_ADDR` AND `ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1`, so a typo in the bind address can't accidentally expose the endpoint cross-host.
## Configuration [#configuration]
All configuration is via environment variables — pprof is a debugging surface, not a runtime feature, so there's no `[debug]` block in `arc.toml`.
| Variable | Default | Description |
| ------------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ARC_DEBUG_PPROF` | unset (off) | Set to `1`, `true`, `yes`, or `on` to enable the pprof listener. Any other value (including unset) leaves it off. |
| `ARC_DEBUG_PPROF_ADDR` | `127.0.0.1:6060` | Bind address for the pprof listener. Accepts any form `net.Listen("tcp", …)` accepts — `127.0.0.1:6060`, `localhost:6060`, `[::1]:6060`, `0.0.0.0:6060`, etc. |
| `ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK` | unset (off) | Required when `ARC_DEBUG_PPROF_ADDR` is non-loopback. Set to `1`/`true`/`yes`/`on`. Without it, Arc logs an error and refuses to start the pprof listener. |
## Enabling pprof on a single node [#enabling-pprof-on-a-single-node]
The common case — investigate a single production node from the same host via SSH and a local port-forward:
```bash
# On the node you want to profile:
ARC_DEBUG_PPROF=1 ./arc
```
Arc emits a startup warning:
```text
WARN ARC_DEBUG_PPROF is set — pprof endpoints are exposed on this address.
Restrict access via firewall or unset ARC_DEBUG_PPROF in production.
addr=127.0.0.1:6060
```
From your laptop, SSH-tunnel the port:
```bash
ssh -L 6060:127.0.0.1:6060 user@node
```
Then point `go tool pprof` at `localhost:6060` on your laptop. See [Profiling Workflows](#profiling-workflows) below.
### With docker-compose [#with-docker-compose]
```yaml
services:
arc-writer:
image: basekick/arc:latest
environment:
ARC_DEBUG_PPROF: "1"
# No host port mapping for 6060 — the listener stays inside the container.
# Use `docker exec` or a sidecar to reach it.
```
To reach the in-container listener:
```bash
docker exec -it arc-writer wget -qO heap.pprof http://127.0.0.1:6060/debug/pprof/heap
docker cp arc-writer:/heap.pprof ./
go tool pprof -http=:8080 heap.pprof
```
### With Kubernetes [#with-kubernetes]
```yaml
env:
- name: ARC_DEBUG_PPROF
value: "1"
```
Then port-forward:
```bash
kubectl port-forward arc-writer-0 6060:6060
```
`kubectl port-forward` only listens on the local machine, so the pprof endpoint stays loopback-bound on the Arc pod AND on your laptop simultaneously. No cluster-network exposure.
## Exposing pprof cross-host (discouraged) [#exposing-pprof-cross-host-discouraged]
There are cases where loopback isn't enough — for example, a remote profiler that can't open an SSH tunnel, or a multi-tenant box where the operator workstation isn't on the Arc host. Arc supports this with a deliberate two-step opt-in:
```bash
ARC_DEBUG_PPROF=1 \
ARC_DEBUG_PPROF_ADDR=0.0.0.0:6060 \
ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1 \
./arc
```
Without `ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1`, Arc logs an **error** and refuses to start the pprof listener — the rest of Arc continues to run normally, but pprof stays off:
```text
ERROR ARC_DEBUG_PPROF=1 with a non-loopback ARC_DEBUG_PPROF_ADDR requires
ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1; refusing to start pprof listener
addr=0.0.0.0:6060
```
When the second opt-in IS set and Arc binds to a non-loopback address, the startup log line is escalated to **error** level (instead of warn) so default alerting policies notice the cross-host exposure on this node:
```text
ERROR ARC_DEBUG_PPROF is set — pprof endpoints are exposed on this address.
Restrict access via firewall or unset ARC_DEBUG_PPROF in production.
addr=0.0.0.0:6060
```
The pprof listener has no authentication. Anyone who can reach `0.0.0.0:6060` (or whatever address you bound) can fetch heap dumps containing recent query text and ingested records, dump goroutine stacks, and pin CPU cores. Restrict by network ACL, security group, or iptables before turning this on. Unset all three env vars the moment you're done.
## Profiling workflows [#profiling-workflows]
Once the listener is reachable at `http://localhost:6060` (whether direct or via SSH/kubectl port-forward), `go tool pprof` does the rest. The recipes below assume Go 1.20+.
### Heap (memory) [#heap-memory]
The most common case — Arc's RSS is high and you want to know what's holding it.
```bash
# Live snapshot:
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/heap
# Save for later analysis:
curl -o heap.pprof http://localhost:6060/debug/pprof/heap
go tool pprof -http=:8080 heap.pprof
```
The `-http=:8080` flag launches the interactive web UI at `http://localhost:8080` — flame graph, top callers, source view. Without it you get the CLI prompt.
Common starting commands at the pprof CLI prompt:
```text
(pprof) top20 # 20 largest in-use allocations by bytes
(pprof) top20 -cum # 20 largest by cumulative (function + callees)
(pprof) list # source-level breakdown of one function
```
### CPU profile [#cpu-profile]
Capture 30 seconds of CPU activity:
```bash
go tool pprof -http=:8080 'http://localhost:6060/debug/pprof/profile?seconds=30'
```
The `seconds` parameter is configurable — 30s is a reasonable default. **Don't go above \~300s** unless you know what you're doing: each in-flight capture holds a connection open and consumes scheduler overhead. Arc's pprof listener has a 10-minute write timeout as the hard ceiling.
### Goroutines [#goroutines]
Diagnose a goroutine leak or deadlock:
```bash
# Summary (top goroutine call sites + counts):
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=1' | head -50
# Full stacks for every goroutine (text):
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=2' > goroutines.txt
# Or via pprof for the UI:
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/goroutine
```
A healthy idle Arc writer typically has \~50–200 goroutines (Fiber workers, WAL writer, ingest shards, compaction scheduler, Raft loops). Thousands of goroutines stuck on the same `chan receive` or `sync.Mutex.Lock` is the diagnostic signature of a stall.
### Execution trace [#execution-trace]
Captures every scheduler event for `N` seconds — useful for diagnosing latency spikes:
```bash
curl -o trace.out 'http://localhost:6060/debug/pprof/trace?seconds=5'
go tool trace -http=:8080 trace.out
```
The trace UI shows per-goroutine timelines, GC pauses, and network/syscall waits. Use sparingly — even 5 seconds of trace produces \~10–50 MB of data on a busy writer.
### Block & mutex profiles [#block--mutex-profiles]
By default these profiles are zero-rate (Go runtime samples nothing). To enable, you'd need to call `runtime.SetBlockProfileRate` / `runtime.SetMutexProfileFraction` from inside Arc — currently not exposed via env var. If you need block/mutex profiles, open an issue describing the problem you're chasing and we'll add the knobs.
## Operational notes [#operational-notes]
### Startup logging [#startup-logging]
When `ARC_DEBUG_PPROF` is unset, Arc emits nothing at startup about pprof. The listener is genuinely absent — no port, no handlers, no log noise.
When set, a single warn-level (loopback) or error-level (non-loopback) line names the bind address and reminds you to restrict access. Grep for `ARC_DEBUG_PPROF is set` in your logs to find nodes that left it on accidentally.
### Shutdown behavior [#shutdown-behavior]
Arc registers pprof with the same shutdown priority as the main HTTP server. On `SIGTERM` / `SIGINT`, the pprof listener closes **immediately** — in-flight captures (especially long `/debug/pprof/profile?seconds=N` requests) are aborted. This is deliberate: a long pprof capture would otherwise hold the cluster's shared shutdown budget and risk skipping downstream hooks (WAL flush, storage close, auth close), which is a data-loss path on what the operator expected to be a graceful exit.
If your capture was killed by shutdown, just re-run it after Arc restarts.
### Port conflicts [#port-conflicts]
If the configured bind address is already in use, Arc logs an **error** and continues without the pprof listener — Arc itself doesn't fail to start. Look for:
```text
ERROR ARC_DEBUG_PPROF=1 but failed to bind pprof listener; continuing without pprof
addr=127.0.0.1:6060 error="listen tcp 127.0.0.1:6060: bind: address already in use"
```
Common causes:
* A previous Arc process didn't release the port (`lsof -nP -iTCP:6060`).
* Another Go service on the host already runs pprof on `:6060` (the Go-runtime convention).
* A non-Arc service grabbed the port.
Resolve the conflict and restart Arc, or set `ARC_DEBUG_PPROF_ADDR` to a different port.
## Security checklist [#security-checklist]
Before enabling pprof on a production node:
* [ ] `ARC_DEBUG_PPROF_ADDR` is loopback (default) **or** the host is firewalled to allow only your jumphost / operator workstation.
* [ ] If non-loopback, `ARC_DEBUG_PPROF_ALLOW_NON_LOOPBACK=1` is set deliberately (not by env-var inheritance from a parent process).
* [ ] You have a plan to unset `ARC_DEBUG_PPROF` when the investigation is done — pprof should not be left on indefinitely.
* [ ] On Kubernetes / docker-compose, the pprof port is **not** in the service's port list or compose `ports:` block — only reachable via `kubectl port-forward` or `docker exec`.
* [ ] Heap dumps you save (`heap.pprof`, `goroutines.txt`, `trace.out`) are treated as sensitive: they contain in-flight query text and ingested records. Don't paste them into public issues; share via your team's secure channel.
## Reference [#reference]
* Source: [`cmd/arc/debug_pprof.go`](https://github.com/Basekick-Labs/arc/blob/main/cmd/arc/debug_pprof.go) — the listener and the two-step gate.
* PR that introduced the gate: [#443](https://github.com/Basekick-Labs/arc/pull/443).
* Advisory: [GHSA-j93g-rp6m-j32m](https://github.com/Basekick-Labs/arc/security/advisories/GHSA-j93g-rp6m-j32m).
* Upstream Go docs: [`net/http/pprof`](https://pkg.go.dev/net/http/pprof) and [`runtime/pprof`](https://pkg.go.dev/runtime/pprof).
# Telemetry (/arc-enterprise/operations/telemetry)
Arc sends anonymous usage telemetry to help improve the project. This page explains what data is collected, how it's used, and how to opt out.
## Overview [#overview]
Arc collects minimal, anonymous usage statistics to help the development team understand:
* How Arc is being deployed (operating systems, hardware configurations)
* Which Arc versions are in active use
* Basic system characteristics for optimization and testing
Arc does not collect any personally identifiable information, user data, database contents, queries, or performance metrics.
## What is collected [#what-is-collected]
Arc sends the following anonymous data every 24 hours:
### Instance information [#instance-information]
* **instance\_id**: A random UUID generated on first run
* Stored in `./data/.instance_id`
* Unique per Arc installation
* Not linked to any personal information
* **timestamp**: When the telemetry report was generated (UTC)
* **arc\_version**: The running version number (e.g., `0.1.0`)
### System information [#system-information]
* **os**: Operating system details
* Name (e.g., "Linux", "macOS", "Windows")
* Version (e.g., "Ubuntu 22.04", "macOS 14.0")
* Architecture (e.g., "x86\_64", "arm64")
* Platform (e.g., "linux", "darwin")
* **cpu**: CPU characteristics
* Physical cores
* Logical cores (threads)
* Frequency in MHz
* **memory**: System memory
* Total RAM in gigabytes
### Example payload [#example-payload]
```json
{
"instance_id": "550e8400-e29b-41d4-a716-446655440000",
"timestamp": "2024-01-20T10:30:00Z",
"arc_version": "0.1.0",
"os": {
"name": "Linux",
"version": "Ubuntu 22.04",
"architecture": "x86_64",
"platform": "linux"
},
"cpu": {
"physical_cores": 8,
"logical_cores": 16,
"frequency_mhz": 3400
},
"memory": {
"total_gb": 32
}
}
```
## What is NOT collected [#what-is-not-collected]
Arc explicitly avoids collecting:
* **User Data**: No usernames, emails, or personal information
* **Database Contents**: No table names, schemas, or data
* **Query Information**: No SQL queries or query patterns
* **Network Information**: No IP addresses or hostnames
* **Credentials**: No API keys, passwords, or tokens
* **File Paths**: No directory structures or file names
* **Performance Metrics**: No query times, throughput, or resource usage
* **Custom Configuration**: No application-specific settings
## How it works [#how-it-works]
### Telemetry schedule [#telemetry-schedule]
1. **First Transmission**: 1 minute after Arc starts
2. **Subsequent Transmissions**: Every 24 hours
3. **Primary Worker Only**: Only the primary worker process sends telemetry (multi-worker deployments send one report)
### Endpoint [#endpoint]
Telemetry is sent to: `telemetry.basekick.net`
### Network behavior [#network-behavior]
* If the telemetry endpoint is unreachable, Arc logs a warning but continues operating normally
* Failed transmissions are retried during the next scheduled transmission
* No telemetry data is queued or persisted locally
### Startup logging [#startup-logging]
Arc logs telemetry status on startup:
**When Enabled**:
```yaml
INFO: Telemetry enabled. Sending anonymous usage data to telemetry.basekick.net every 24 hours.
```
**When Disabled**:
```yaml
INFO: Telemetry disabled via configuration.
```
## Disabling telemetry [#disabling-telemetry]
You can opt out of telemetry in two ways:
### Option 1: configuration file [#option-1-configuration-file]
Edit your `arc.toml` file and add:
```toml
[telemetry]
enabled = false
```
**Full Example**:
```toml
[server]
host = "0.0.0.0"
port = 8000
[telemetry]
enabled = false
```
### Option 2: environment variable [#option-2-environment-variable]
Set the environment variable before starting Arc:
```bash
export ARC_TELEMETRY_ENABLED=false
```
**With Docker**:
```bash
docker run -e ARC_TELEMETRY_ENABLED=false arc:latest
```
**With Docker Compose**:
```yaml
services:
arc:
image: arc:latest
environment:
- ARC_TELEMETRY_ENABLED=false
```
### Verification [#verification]
After configuring, start Arc and check the logs:
```yaml
INFO: Telemetry disabled via configuration.
```
If you see this message, telemetry is successfully disabled.
## Why telemetry? [#why-telemetry]
### Benefits to the project [#benefits-to-the-project]
Anonymous telemetry helps the Arc team:
1. **Prioritize Platform Support**: Understand which operating systems and architectures to focus on
2. **Test on Real Hardware**: Know what CPU and memory configurations are common
3. **Track Version Adoption**: See how quickly users upgrade to new releases
4. **Plan Deprecations**: Identify when old versions are no longer in use
### Privacy considerations [#privacy-considerations]
Arc's telemetry is designed with privacy as a priority:
* **Anonymous**: No linkage to individuals or organizations
* **Minimal**: Only essential system characteristics
* **Transparent**: Full disclosure of what is collected
* **Optional**: Easy opt-out with no functionality loss
* **No Tracking**: No cookies, fingerprinting, or cross-site tracking
## Frequently asked questions [#frequently-asked-questions]
### Does telemetry affect performance? [#does-telemetry-affect-performance]
No. Telemetry runs asynchronously and has negligible performance impact:
* Transmission occurs once per 24 hours
* Payload is \~500 bytes
* Network timeout is short (5 seconds)
* Failed transmissions don't block Arc operations
### Can I verify what's being sent? [#can-i-verify-whats-being-sent]
Yes. You can inspect the telemetry payload by:
1. **Network Inspection**: Use tools like Wireshark or tcpdump to capture the request
2. **Source Code**: Review the telemetry implementation in the Arc repository
3. **Logging**: Enable debug logging to see telemetry payloads (future feature)
### What happens to the data? [#what-happens-to-the-data]
Telemetry data is:
* Stored securely on Basekick infrastructure
* Aggregated for statistical analysis
* Not shared with third parties
* Not used for commercial purposes
* Retained for a limited time (90 days)
### Will Arc work if telemetry is blocked? [#will-arc-work-if-telemetry-is-blocked]
Yes. Arc functions identically whether telemetry is enabled or disabled. If the telemetry endpoint is unreachable (firewall, network issues), Arc logs a warning and continues normally.
### Why not make it opt-in? [#why-not-make-it-opt-in]
We believe in transparency and easy opt-out rather than opt-in because:
* Telemetry helps improve the product for everyone
* Data collected is truly anonymous and minimal
* Opt-out is simple and clearly documented
* Many users don't discover opt-in options
However, we respect your choice and make opting out straightforward.
### Does Arc Enterprise have different telemetry? [#does-arc-enterprise-have-different-telemetry]
No. Both Arc OSS and Arc Enterprise use identical telemetry collection. Arc Enterprise customers can request custom telemetry configurations for their deployments.
## Privacy policy [#privacy-policy]
For detailed information about how Basekick handles data, see our [Privacy Policy](https://basekick.net/privacy?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise) (Coming Soon).
## Support [#support]
If you have questions or concerns about telemetry:
* [Discord Community](https://discord.gg/nxnWfUxsdm)
* [GitHub Issues](https://github.com/basekick-labs/arc/issues)
* Email: [privacy@basekick.net](mailto:privacy@basekick.net)
# Performance Benchmarks (/arc-enterprise/performance/benchmarks)
Benchmark results for Arc are published on the Basekick blog rather than in these docs, so that every figure stays tied to the hardware, dataset, and Arc version it was measured on.
## Published results [#published-results]
Start with the ClickBench summary, which covers methodology, dataset, and how Arc was configured for each run:
* **[Arc on ClickBench](https://basekick.net/blog/arc-fastest-timeseries-database-clickbench?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)** — methodology and headline results
Per-database comparisons:
* [vs InfluxDB](https://basekick.net/blog/arc-clickbench-vs-influxdb?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)
* [vs TimescaleDB](https://basekick.net/blog/arc-clickbench-vs-timescaledb?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)
* [vs DuckDB](https://basekick.net/blog/arc-clickbench-vs-duckdb?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)
* [vs CrateDB](https://basekick.net/blog/arc-clickbench-vs-cratedb?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)
* [vs StarRocks](https://basekick.net/blog/arc-clickbench-vs-starrocks?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)
* [vs Elasticsearch](https://basekick.net/blog/arc-clickbench-vs-elasticsearch?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)
Additional runs:
* [Cold-run results](https://basekick.net/blog/arc-clickbench-cold-runs?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise) — first-query latency against object storage
* [Log benchmark](https://basekick.net/blog/arc-log-benchmark-2026?utm_source=docs\&utm_medium=referral\&utm_campaign=arc-enterprise)
## Reproduce locally [#reproduce-locally]
The benchmark harness ships in the Arc repository:
```bash
git clone https://github.com/basekick-labs/arc.git
cd arc
make bench
```
## Benchmarking an Enterprise cluster [#benchmarking-an-enterprise-cluster]
Published numbers are measured on a single node. A clustered deployment adds variables that dominate the result, so size from your own measurements rather than from the blog figures:
* **Node roles.** Writers, readers, and compactors are benchmarked separately — a reader's query throughput is unrelated to a writer's ingest ceiling. See [Clustering](/arc-enterprise/configuration/clustering/).
* **Storage topology.** Shared object storage and local storage with peer replication have different latency profiles. See [Deployment patterns](/arc-enterprise/configuration/deployment-patterns/).
* **Tiered storage.** Queries that reach cold-tier data pay a retrieval cost that hot-tier queries do not. See [Tiered storage](/arc-enterprise/data-lifecycle/tiered-storage/).
* **Query governance.** Rate limits and row limits cap throughput by design; benchmark with the limits you intend to run. See [Query governance](/arc-enterprise/query/query-governance/).
## Next steps [#next-steps]
* **[Getting started](/arc-enterprise/getting-started/)** — run Arc Enterprise locally
* **[Configuration](/arc-enterprise/configuration/overview/)** — tune for your workload
* **[Python SDK](/arc-enterprise/sdks/python/)** — client for driving load
# Performance (/arc-enterprise/performance)
Benchmark figures live on the Basekick blog rather than in these docs, so each number stays attached to the hardware, dataset, and Arc version it was measured on.
Published results are single-node. An Enterprise cluster introduces variables that dominate them, so size from your own measurements.
# Query (/arc-enterprise/query)
Both features on this page are Enterprise-only, and they answer two different questions about a shared cluster.
Governance is preventive: it caps what any one token may consume, so a runaway dashboard cannot starve other tenants. Query management is diagnostic: it shows what is running right now and lets you cancel it.
# Query Governance (/arc-enterprise/query/query-governance)
Control resource usage with per-token rate limits, query quotas, and row limits. Protect your cluster from runaway queries and ensure fair resource allocation across teams.
## Overview [#overview]
Query governance enforces limits at the API token level:
* **Rate limits** — Maximum queries per minute and per hour
* **Query quotas** — Maximum queries per hour and per day
* **Row limits** — Maximum rows returned per query
* **Per-token policies** — Override defaults for specific tokens
* **Usage monitoring** — Track current usage and remaining quotas
## Prerequisites [#prerequisites]
* Authentication must be enabled (`ARC_AUTH_ENABLED=true`) — governance is enforced per token
* Arc Enterprise license with query governance feature
## Configuration [#configuration]
### Default limits [#default-limits]
Set global defaults that apply to all tokens without a specific policy:
```toml
[governance]
enabled = true
default_rate_limit_per_min = 60 # 0 = unlimited
default_rate_limit_per_hour = 1000
default_max_queries_per_hour = 500
default_max_queries_per_day = 5000
default_max_rows_per_query = 100000
```
### Environment variables [#environment-variables]
```bash
ARC_GOVERNANCE_ENABLED=true
ARC_GOVERNANCE_DEFAULT_RATE_LIMIT_PER_MIN=60
ARC_GOVERNANCE_DEFAULT_RATE_LIMIT_PER_HOUR=1000
ARC_GOVERNANCE_DEFAULT_MAX_QUERIES_PER_HOUR=500
ARC_GOVERNANCE_DEFAULT_MAX_QUERIES_PER_DAY=5000
ARC_GOVERNANCE_DEFAULT_MAX_ROWS_PER_QUERY=100000
```
Setting any limit to `0` means unlimited. If you want governance enabled but with no default restrictions, set all defaults to `0` and create explicit policies for tokens that need limits.
## Enforcement behavior [#enforcement-behavior]
When a limit is exceeded, Arc responds with:
| Scenario | HTTP Status | Behavior |
| --------------------- | ----------------------- | ---------------------------------------- |
| Rate limit exceeded | `429 Too Many Requests` | Includes `Retry-After` header |
| Query quota exhausted | `429 Too Many Requests` | Quota resets at the next period boundary |
| Max rows exceeded | `200 OK` | Returns partial results with a warning |
**Rate limit response example:**
```json
{
"success": false,
"error": "Rate limit exceeded. Try again in 45 seconds.",
"retry_after": 45
}
```
The response includes a `Retry-After` HTTP header that clients can use for automatic backoff.
## Per-token policies [#per-token-policies]
Override default limits for specific tokens. This is useful for giving higher limits to critical services or stricter limits to external integrations.
### Create policy [#create-policy]
```bash
curl -X POST http://localhost:8000/api/v1/governance/policies \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"token_id": 1,
"rate_limit_per_minute": 120,
"rate_limit_per_hour": 5000,
"max_queries_per_hour": 2000,
"max_queries_per_day": 20000,
"max_rows_per_query": 500000
}'
```
**Response:**
```json
{
"success": true,
"data": {
"token_id": 1,
"rate_limit_per_minute": 120,
"rate_limit_per_hour": 5000,
"max_queries_per_hour": 2000,
"max_queries_per_day": 20000,
"max_rows_per_query": 500000,
"created_at": "2026-02-13T10:00:00Z",
"updated_at": "2026-02-13T10:00:00Z"
}
}
```
### List all policies [#list-all-policies]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/governance/policies
```
### Get policy for token [#get-policy-for-token]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/governance/policies/1
```
### Update policy [#update-policy]
```bash
curl -X PUT http://localhost:8000/api/v1/governance/policies/1 \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"rate_limit_per_minute": 200,
"max_rows_per_query": 1000000
}'
```
### Delete policy [#delete-policy]
Removes the per-token policy. The token reverts to the global defaults.
```bash
curl -X DELETE http://localhost:8000/api/v1/governance/policies/1 \
-H "Authorization: Bearer $ADMIN_TOKEN"
```
## Usage monitoring [#usage-monitoring]
Check current usage and remaining quotas for any token.
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/governance/usage/1
```
**Response:**
```json
{
"success": true,
"data": {
"token_id": 1,
"current_minute_count": 15,
"rate_limit_per_minute": 120,
"current_hour_count": 342,
"max_queries_per_hour": 2000,
"current_day_count": 1580,
"max_queries_per_day": 20000,
"remaining_minute": 105,
"remaining_hour": 1658,
"remaining_day": 18420
}
}
```
## Best practices [#best-practices]
1. **Set conservative defaults** — Start with moderate limits (e.g., 60/min, 500/hour) and increase for tokens that need more.
2. **Give critical services higher limits** — Create explicit policies for ingestion and dashboard tokens that need higher throughput.
3. **Use row limits for external integrations** — Prevent third-party tools from pulling excessive data with `max_rows_per_query`.
4. **Monitor usage patterns** — Use the usage API to identify tokens approaching their limits before they start getting throttled.
5. **Pair with query management** — Use [query management](/arc-enterprise/query/query-management/) to identify which queries consume the most resources.
## Next steps [#next-steps]
* [Query Management](/arc-enterprise/query/query-management/) — Monitor and cancel running queries
* [Audit Logging](/arc-enterprise/security/audit-logging/) — Track all governance enforcement events
# Query Management (/arc-enterprise/query/query-management)
Monitor active queries in real time, review query history, and cancel long-running queries. Full visibility into your query workload for debugging and capacity planning.
## Overview [#overview]
Query management provides:
* **Active query monitoring** — See all currently running queries with their duration, SQL, and resource usage
* **Query history** — Review recently completed queries with execution details
* **Query cancellation** — Cancel long-running or runaway queries on demand
* **Diagnostic details** — View parallel execution status and partition counts per query
## Configuration [#configuration]
### TOML [#toml]
```toml
[query_management]
enabled = true
history_size = 100 # Number of completed queries to keep in history
```
### Environment variables [#environment-variables]
```bash
ARC_QUERY_MANAGEMENT_ENABLED=true
ARC_QUERY_MANAGEMENT_HISTORY_SIZE=100
```
## API reference [#api-reference]
All query management endpoints require admin authentication.
### List active queries [#list-active-queries]
View all currently running queries:
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/queries/active
```
**Response:**
```json
{
"success": true,
"data": [
{
"id": "q-abc123",
"sql": "SELECT * FROM production.sensors WHERE timestamp > NOW() - INTERVAL '1 hour'",
"status": "running",
"token_name": "grafana-readonly",
"remote_addr": "10.0.1.50:54321",
"started_at": "2026-02-13T10:30:00Z",
"duration_ms": 2500,
"is_parallel": true,
"partition_count": 4
},
{
"id": "q-def456",
"sql": "SELECT COUNT(*) FROM analytics.events GROUP BY event_type",
"status": "running",
"token_name": "dashboard-token",
"remote_addr": "10.0.1.51:54322",
"started_at": "2026-02-13T10:30:01Z",
"duration_ms": 1200,
"is_parallel": false,
"partition_count": 1
}
]
}
```
### View query history [#view-query-history]
Review recently completed queries:
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/queries/history
```
**Response:**
```json
{
"success": true,
"data": [
{
"id": "q-ghi789",
"sql": "SELECT AVG(temperature) FROM production.sensors GROUP BY device_id",
"status": "completed",
"token_name": "analytics-token",
"remote_addr": "10.0.1.52:54323",
"started_at": "2026-02-13T10:29:00Z",
"duration_ms": 850,
"is_parallel": true,
"partition_count": 3
}
]
}
```
### Get query details [#get-query-details]
Inspect a specific query by ID:
```bash
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/queries/q-abc123
```
### Cancel a query [#cancel-a-query]
Stop a long-running or runaway query:
```bash
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/queries/q-abc123
```
**Response:**
```json
{
"success": true,
"data": {
"message": "Query cancelled",
"id": "q-abc123"
}
}
```
## Use cases [#use-cases]
### Debugging slow queries [#debugging-slow-queries]
1. Check active queries to find long-running operations
2. Review the SQL and execution details
3. Cancel if the query is consuming excessive resources
4. Optimize the query and retry
```bash
# Find queries running longer than expected
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/queries/active
# Cancel a specific runaway query
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
http://localhost:8000/api/v1/queries/q-abc123
```
### Capacity planning [#capacity-planning]
Review query history to understand workload patterns:
* Which tokens generate the most queries?
* What are typical query durations?
* How many queries run in parallel?
* Are queries hitting multiple partitions (indicating large time ranges)?
### Incident response [#incident-response]
During performance incidents:
1. List active queries to identify the cause
2. Cancel offending queries immediately
3. Review history to understand what changed
4. Apply [query governance](/arc-enterprise/query/query-governance/) policies to prevent recurrence
## Best practices [#best-practices]
1. **Set an appropriate history size** — The default (100) works for most deployments. Increase it if you need more historical context for debugging.
2. **Monitor active queries in dashboards** — Poll the active queries endpoint from your monitoring system to detect long-running queries early.
3. **Combine with query governance** — Use governance policies to prevent runaway queries automatically, and query management for visibility and manual intervention.
4. **Review parallel query patterns** — Queries with high partition counts span large time ranges. Consider adding time bounds to improve performance.
## Next steps [#next-steps]
* [Query Governance](/arc-enterprise/query/query-governance/) — Automatically enforce rate limits and quotas
* [Audit Logging](/arc-enterprise/security/audit-logging/) — Track query patterns for compliance
# SDKs (/arc-enterprise/sdks)
Arc's client libraries talk to the same REST API in OSS and Enterprise deployments. Against a cluster you point the client at a load-balanced or reader endpoint and authenticate with a token whose permissions come from [RBAC](/arc-enterprise/security/rbac/).
# Audit Logging (/arc-enterprise/security/audit-logging)
Track every significant operation in your Arc deployment. Audit logging captures authentication attempts, data access, configuration changes, and infrastructure events for compliance and security monitoring.
## Overview [#overview]
Arc Enterprise audit logging provides:
* **Comprehensive event capture** — Authentication, data operations, RBAC changes, and infrastructure events
* **Query and filter API** — Search logs by event type, actor, database, and time range
* **Configurable retention** — Automatic cleanup of old audit entries
* **Non-blocking** — Audit events are captured asynchronously with zero impact on request latency
## Event types [#event-types]
| Category | Events | Description |
| ----------------------- | ------------------------------------------------------------------ | ------------------------------------------------ |
| **Authentication** | `auth.failed` | Failed authentication attempts (401/403) |
| **Token Management** | `token.created`, `token.deleted`, `token.rotated` | API token lifecycle events |
| **RBAC** | `rbac.org.created`, `rbac.team.updated`, `rbac.role.deleted`, etc. | Organization, team, role, and membership changes |
| **Data Operations** | `data.query`, `data.write`, `data.import`, `data.delete` | Data read/write operations |
| **Database Management** | `database.created`, `database.deleted` | Database lifecycle events |
| **Infrastructure** | `mqtt.*`, `compaction.triggered`, `tiering.*` | System and background operations |
| **API** | `api.POST`, `api.PUT`, `api.DELETE` | Catch-all for other API operations |
## Configuration [#configuration]
### TOML [#toml]
```toml
[audit_log]
enabled = true
retention_days = 90 # Auto-cleanup entries older than this
include_reads = false # Log GET/query requests (high volume)
```
### Environment variables [#environment-variables]
```bash
ARC_AUDIT_LOG_ENABLED=true
ARC_AUDIT_LOG_RETENTION_DAYS=90
ARC_AUDIT_LOG_INCLUDE_READS=false
```
Enabling `include_reads` logs every query and GET request. This can generate significant log volume in high-throughput environments. Enable it only when needed for compliance or debugging, and consider a shorter `retention_days` when active.
## API reference [#api-reference]
All audit endpoints require admin authentication.
### Query audit logs [#query-audit-logs]
```bash
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/audit/logs?limit=20"
```
**Filter parameters:**
| Parameter | Type | Description |
| ------------ | ------ | -------------------------------------------------------- |
| `event_type` | string | Filter by event type (e.g., `auth.failed`, `data.write`) |
| `actor` | string | Filter by actor (token name or ID) |
| `database` | string | Filter by database name |
| `since` | string | Start time (RFC3339, e.g., `2026-02-01T00:00:00Z`) |
| `until` | string | End time (RFC3339) |
| `limit` | int | Maximum results (default: 50) |
| `offset` | int | Pagination offset |
**Examples:**
```bash
# Failed authentication attempts in the last 24 hours
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/audit/logs?event_type=auth.failed&since=2026-02-12T00:00:00Z"
# All write operations to the production database
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/audit/logs?event_type=data.write&database=production"
# Recent RBAC changes
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/audit/logs?event_type=rbac&limit=50"
# Activity by a specific token
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/audit/logs?actor=telegraf-writer&limit=100"
```
**Response:**
```json
{
"success": true,
"data": [
{
"id": 1542,
"event_type": "data.write",
"actor": "telegraf-writer",
"database": "production",
"detail": "measurement=cpu, records=5000",
"ip_address": "10.0.1.50",
"timestamp": "2026-02-13T10:30:00Z"
},
{
"id": 1541,
"event_type": "auth.failed",
"actor": "unknown",
"detail": "invalid token",
"ip_address": "192.168.1.100",
"timestamp": "2026-02-13T10:29:55Z"
}
],
"total": 1542,
"limit": 20,
"offset": 0
}
```
### Get audit statistics [#get-audit-statistics]
Aggregate event counts by type, useful for dashboards and alerting.
```bash
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/audit/stats"
```
**With time range:**
```bash
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/audit/stats?since=2026-02-01T00:00:00Z&until=2026-02-13T23:59:59Z"
```
**Response:**
```json
{
"success": true,
"data": {
"auth.failed": 23,
"data.write": 15420,
"data.query": 8340,
"token.created": 5,
"rbac.role.created": 3,
"database.created": 2
}
}
```
## Compliance use cases [#compliance-use-cases]
### SOC 2 [#soc-2]
SOC 2 requires logging of access to systems and data. Arc audit logging captures:
* Who accessed the system (actor/token identification)
* What they did (event type and detail)
* When it happened (timestamp)
* Where the request came from (IP address)
### HIPAA [#hipaa]
For healthcare data, enable `include_reads = true` to log all data access, including queries. Set `retention_days` according to your HIPAA retention requirements (typically 6 years).
### Security monitoring [#security-monitoring]
Monitor `auth.failed` events to detect brute-force attempts. Use the stats API to set up alerts when failed authentication counts exceed normal thresholds.
## Best practices [#best-practices]
1. **Start with writes only** — Keep `include_reads = false` (default) and enable read logging only when compliance requires it.
2. **Set appropriate retention** — 90 days is a good default. Adjust based on your compliance requirements (HIPAA may require years).
3. **Monitor failed auth attempts** — Set up alerts on `auth.failed` events to detect unauthorized access attempts.
4. **Pair with RBAC** — Use audit logs to verify that RBAC permissions are configured correctly by reviewing who accessed what data.
5. **Export for long-term analysis** — For retention beyond what Arc stores, periodically export audit logs to your SIEM or log management system.
## Next steps [#next-steps]
* [RBAC](/arc-enterprise/security/rbac/) — Control who can access your data
* [Query Governance](/arc-enterprise/query/query-governance/) — Add rate limits and quotas
# Cluster Security (/arc-enterprise/security/cluster-security)
Secure inter-node communication in Arc Enterprise clusters with shared secret authentication and TLS encryption.
## Overview [#overview]
Arc Enterprise clustering supports two complementary security layers:
* **Shared secret authentication** — Prevents unauthorized nodes from joining the cluster using HMAC-SHA256 challenge-response
* **TLS encryption** — Encrypts all inter-node traffic including coordinator messages, WAL replication, peer file replication, and Raft consensus
Both features are opt-in and can be used independently or together. When disabled, cluster behavior is unchanged from previous versions.
For production deployments, enable **both** shared secret and TLS. The shared secret prevents unauthorized joins, while TLS encrypts all traffic between nodes. Using shared secret without TLS means the HMAC tokens are visible on the network.
## Shared secret authentication [#shared-secret-authentication]
When a shared secret is configured, every node joining the cluster must prove it knows the secret. The joining node computes an HMAC-SHA256 signature over a random nonce, its node ID, the cluster name, and a timestamp. The leader validates the signature before accepting the join.
* Timestamps are checked within a 5-minute tolerance to prevent replay attacks
* The shared secret itself is never sent over the network — only the HMAC signature
* All nodes in the cluster must be configured with the same secret
### Configuration [#configuration]
```toml
[cluster]
enabled = true
cluster_name = "production"
shared_secret = "my-secure-cluster-secret-key"
```
```bash
ARC_CLUSTER_ENABLED=true
ARC_CLUSTER_CLUSTER_NAME=production
ARC_CLUSTER_SHARED_SECRET=my-secure-cluster-secret-key
```
Use a strong, randomly generated secret (at least 32 characters). Store it securely using environment variables or a secrets manager — avoid committing it to version control. In Kubernetes, use a Secret resource and reference it in your pod spec.
### What happens on join [#what-happens-on-join]
1. The joining node generates a cryptographically random nonce (32 bytes)
2. It computes `HMAC-SHA256(secret, nonce:nodeID:clusterName:timestamp)`
3. The nonce, timestamp, and HMAC are included in the join request
4. The leader recomputes the HMAC with its own copy of the secret
5. If the HMACs match and the timestamp is within tolerance, the join is accepted
6. If not, the join is rejected with an authentication error
## TLS encryption [#tls-encryption]
When TLS is enabled, all inter-node TCP connections are encrypted. This covers:
| Connection | Description |
| --------------------- | --------------------------------------------------------------------------------------------------- |
| Coordinator listener | Accepts join requests, heartbeats, and leave notifications |
| Coordinator dialer | Connects to seed nodes during cluster discovery |
| WAL replication | Streams write-ahead log entries from writer to reader nodes |
| Peer file replication | Fetches Parquet files from peer nodes (SHA-256 verified) so each node converges to the full dataset |
| Raft consensus | Leader election and log replication between voters |
### Configuration [#configuration-1]
```toml
[cluster]
enabled = true
cluster_name = "production"
shared_secret = "my-secure-cluster-secret-key"
# TLS for inter-node communication
tls_enabled = true
tls_cert_file = "/etc/arc/cluster-cert.pem"
tls_key_file = "/etc/arc/cluster-key.pem"
tls_ca_file = "/etc/arc/cluster-ca.pem" # Optional: for mutual TLS
```
```bash
ARC_CLUSTER_ENABLED=true
ARC_CLUSTER_CLUSTER_NAME=production
ARC_CLUSTER_SHARED_SECRET=my-secure-cluster-secret-key
# TLS for inter-node communication
ARC_CLUSTER_TLS_ENABLED=true
ARC_CLUSTER_TLS_CERT_FILE=/etc/arc/cluster-cert.pem
ARC_CLUSTER_TLS_KEY_FILE=/etc/arc/cluster-key.pem
ARC_CLUSTER_TLS_CA_FILE=/etc/arc/cluster-ca.pem
```
### Configuration reference [#configuration-reference]
| Setting | Default | Description |
| --------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tls_enabled` | `false` | Enable TLS for all inter-node communication |
| `tls_cert_file` | — | Path to the TLS certificate file (PEM format). Required when `tls_enabled=true`. |
| `tls_key_file` | — | Path to the TLS private key file (PEM format). Required when `tls_enabled=true`. |
| `tls_ca_file` | — | Optional CA certificate for verifying peer certificates. When set, enables mutual TLS (mTLS) — each node verifies the other's certificate against this CA. |
### Certificate requirements [#certificate-requirements]
* TLS 1.2 minimum is enforced
* Certificates must be in PEM format
* Arc validates certificates at startup:
* **Expired certificates** generate a warning but do not block startup, allowing operators to rotate without downtime
* **Certificates expiring within 30 days** generate an early warning
* When `tls_ca_file` is set, both client and server sides verify peer certificates against the CA
## Generating certificates [#generating-certificates]
For testing or internal deployments, you can generate self-signed certificates:
```bash
# Generate a CA key and certificate
openssl genrsa -out cluster-ca-key.pem 4096
openssl req -new -x509 -key cluster-ca-key.pem -sha256 \
-subj "/CN=arc-cluster-ca" -days 3650 -out cluster-ca.pem
# Generate a node certificate signed by the CA
openssl genrsa -out cluster-key.pem 4096
openssl req -new -key cluster-key.pem \
-subj "/CN=arc-cluster-node" -out cluster.csr
# Sign with the CA (include SANs for all node addresses)
openssl x509 -req -in cluster.csr \
-CA cluster-ca.pem -CAkey cluster-ca-key.pem -CAcreateserial \
-days 365 -sha256 \
-extfile <(printf "subjectAltName=DNS:*.arc-cluster.svc.cluster.local,DNS:localhost,IP:127.0.0.1") \
-out cluster-cert.pem
```
Use a wildcard SAN like `*.arc-cluster.svc.cluster.local` to cover all pods in a headless Service. This way, certificates remain valid when pods are rescheduled to different Kubernetes nodes. Avoid using pod IPs in SANs — they change on every reschedule.
For production, use your organization's PKI or a tool like [cert-manager](https://cert-manager.io/) to automate certificate issuance and rotation.
## Kubernetes deployment example [#kubernetes-deployment-example]
In Kubernetes, store the shared secret and TLS certificates as Secrets and mount them into the pod:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: arc-cluster-secret
type: Opaque
stringData:
shared-secret: "your-strong-random-secret-here"
---
apiVersion: v1
kind: Secret
metadata:
name: arc-cluster-tls
type: kubernetes.io/tls
data:
tls.crt:
tls.key:
ca.crt:
```
Reference in your StatefulSet or Deployment:
```yaml
containers:
- name: arc
env:
- name: ARC_CLUSTER_SHARED_SECRET
valueFrom:
secretKeyRef:
name: arc-cluster-secret
key: shared-secret
- name: ARC_CLUSTER_TLS_ENABLED
value: "true"
- name: ARC_CLUSTER_TLS_CERT_FILE
value: "/etc/arc-tls/tls.crt"
- name: ARC_CLUSTER_TLS_KEY_FILE
value: "/etc/arc-tls/tls.key"
- name: ARC_CLUSTER_TLS_CA_FILE
value: "/etc/arc-tls/ca.crt"
volumeMounts:
- name: cluster-tls
mountPath: /etc/arc-tls
readOnly: true
volumes:
- name: cluster-tls
secret:
secretName: arc-cluster-tls
```
## Troubleshooting [#troubleshooting]
### Join rejected: Shared secret required [#join-rejected-shared-secret-required]
The leader has `shared_secret` configured but the joining node does not. Ensure all nodes use the same `shared_secret` value.
### Join rejected: Authentication failed [#join-rejected-authentication-failed]
The joining node's shared secret does not match the leader's, or the system clocks are more than 5 minutes apart. Verify the secret is identical on all nodes and that NTP is configured.
### TLS handshake failure [#tls-handshake-failure]
* Verify certificate and key files exist and are readable by the Arc process
* Check that certificates are not expired (`openssl x509 -in cert.pem -noout -dates`)
* If using mutual TLS (`tls_ca_file`), ensure all nodes' certificates are signed by the same CA
### Certificate expiration warnings at startup [#certificate-expiration-warnings-at-startup]
Arc logs a warning if the cluster TLS certificate expires within 30 days. Rotate the certificate before it expires to avoid connection failures. If using cert-manager, configure automatic renewal.
## Next steps [#next-steps]
* [Clustering & High Availability](/arc-enterprise/configuration/clustering/) — Node roles, Raft consensus, and deployment patterns
* [RBAC](/arc-enterprise/security/rbac/) — Role-based access control for API authentication
* [Audit Logging](/arc-enterprise/security/audit-logging/) — Track cluster operations for compliance
# Security (/arc-enterprise/security)
Arc OSS authenticates with API tokens that are effectively all-or-nothing. Arc Enterprise adds structure above them and a trust boundary below them.
Above: [RBAC](/arc-enterprise/security/rbac/) attaches organizations, teams, and roles to those tokens, down to per-measurement permissions, and [audit logging](/arc-enterprise/security/audit-logging/) records who did what. Below: nodes authenticate to each other with a shared secret and encrypt inter-node traffic with TLS, so cluster membership is not implied by network reachability alone.
# Role-Based Access Control (RBAC) (/arc-enterprise/security/rbac)
Manage access to your Arc deployment with organizations, teams, and granular permissions down to the measurement level.
## Overview [#overview]
Arc Enterprise RBAC builds on top of Arc's token-based authentication to add organizational structure and fine-grained permissions:
```text
Organization (e.g., "Acme Corp")
└── Team (e.g., "Data Engineering")
└── Role (e.g., "production-readwrite")
├── Database: "production" → [read, write, delete]
└── Database: "analytics" → [read]
└── Measurements: ["metrics_*", "events_*"]
```
**Key capabilities:**
* **Organizations** — Top-level grouping for your company or division
* **Teams** — Group users by function (engineering, analytics, operations)
* **Roles** — Define permissions per database with optional measurement restrictions
* **Measurement-level permissions** — Restrict access to specific measurements using wildcard patterns
* **Backward compatible** — Existing OSS token permissions continue to work
## Prerequisites [#prerequisites]
* Authentication must be enabled (`ARC_AUTH_ENABLED=true`)
* Arc Enterprise license with RBAC feature
## Permission model [#permission-model]
Permissions are defined at the role level and apply to specific databases:
| Permission | Description |
| ---------- | ----------------------------- |
| `read` | Query data from the database |
| `write` | Write data to the database |
| `delete` | Delete data from the database |
| `admin` | Full administrative access |
Roles can optionally restrict access to specific measurements within a database using wildcard patterns (e.g., `metrics_*` matches `metrics_cpu`, `metrics_memory`, etc.).
## API reference [#api-reference]
All RBAC endpoints require admin authentication.
### Organizations [#organizations]
#### Create organization [#create-organization]
```bash
curl -X POST http://localhost:8000/api/v1/rbac/organizations \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Acme Corp",
"description": "Main organization"
}'
```
**Response:**
```json
{
"success": true,
"data": {
"id": 1,
"name": "Acme Corp",
"description": "Main organization",
"created_at": "2026-02-13T10:00:00Z",
"updated_at": "2026-02-13T10:00:00Z"
}
}
```
#### List organizations [#list-organizations]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/rbac/organizations
```
#### Get organization [#get-organization]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/rbac/organizations/1
```
#### Update organization [#update-organization]
```bash
curl -X PATCH http://localhost:8000/api/v1/rbac/organizations/1 \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"description": "Updated description"}'
```
#### Delete organization [#delete-organization]
```bash
curl -X DELETE http://localhost:8000/api/v1/rbac/organizations/1 \
-H "Authorization: Bearer $ADMIN_TOKEN"
```
### Teams [#teams]
#### Create team [#create-team]
```bash
curl -X POST http://localhost:8000/api/v1/rbac/organizations/1/teams \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Data Engineering",
"description": "Data engineering team"
}'
```
**Response:**
```json
{
"success": true,
"data": {
"id": 1,
"organization_id": 1,
"name": "Data Engineering",
"description": "Data engineering team",
"created_at": "2026-02-13T10:00:00Z",
"updated_at": "2026-02-13T10:00:00Z"
}
}
```
#### List teams [#list-teams]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/rbac/organizations/1/teams
```
#### Get team [#get-team]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/rbac/teams/1
```
#### Update team [#update-team]
```bash
curl -X PATCH http://localhost:8000/api/v1/rbac/teams/1 \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"description": "Updated team description"}'
```
#### Delete team [#delete-team]
```bash
curl -X DELETE http://localhost:8000/api/v1/rbac/teams/1 \
-H "Authorization: Bearer $ADMIN_TOKEN"
```
### Roles [#roles]
#### Create role [#create-role]
```bash
curl -X POST http://localhost:8000/api/v1/rbac/teams/1/roles \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "production-readwrite",
"database_pattern": "production",
"permissions": ["read", "write"]
}'
```
**Response:**
```json
{
"success": true,
"data": {
"id": 1,
"team_id": 1,
"name": "production-readwrite",
"database_pattern": "production",
"permissions": ["read", "write"],
"created_at": "2026-02-13T10:00:00Z",
"updated_at": "2026-02-13T10:00:00Z"
}
}
```
Use `*` as the database pattern to grant permissions across all databases. For example, `"database_pattern": "*"` with `"permissions": ["read"]` grants read access to every database.
#### List roles [#list-roles]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/rbac/teams/1/roles
```
#### Get role [#get-role]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/rbac/roles/1
```
#### Update role [#update-role]
```bash
curl -X PATCH http://localhost:8000/api/v1/rbac/roles/1 \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"permissions": ["read", "write", "delete"]}'
```
#### Delete role [#delete-role]
```bash
curl -X DELETE http://localhost:8000/api/v1/rbac/roles/1 \
-H "Authorization: Bearer $ADMIN_TOKEN"
```
### Measurement permissions [#measurement-permissions]
Restrict a role to specific measurements within its database pattern.
#### Add measurement permission [#add-measurement-permission]
```bash
curl -X POST http://localhost:8000/api/v1/rbac/roles/1/measurements \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"measurement_pattern": "metrics_*"
}'
```
#### List measurement permissions [#list-measurement-permissions]
```bash
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
http://localhost:8000/api/v1/rbac/roles/1/measurements
```
#### Remove measurement permission [#remove-measurement-permission]
```bash
curl -X DELETE http://localhost:8000/api/v1/rbac/roles/1/measurements/1 \
-H "Authorization: Bearer $ADMIN_TOKEN"
```
## Walkthrough: Setting up RBAC [#walkthrough-setting-up-rbac]
This example sets up a typical organization with two teams and different access levels.
### Step 1: create the organization [#step-1-create-the-organization]
```bash
curl -X POST http://localhost:8000/api/v1/rbac/organizations \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name": "Acme Corp"}'
```
### Step 2: create teams [#step-2-create-teams]
```bash
# Data Engineering team — full access
curl -X POST http://localhost:8000/api/v1/rbac/organizations/1/teams \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name": "Data Engineering"}'
# Analytics team — read-only access
curl -X POST http://localhost:8000/api/v1/rbac/organizations/1/teams \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name": "Analytics"}'
```
### Step 3: create roles [#step-3-create-roles]
```bash
# Data Engineering: read/write/delete on production database
curl -X POST http://localhost:8000/api/v1/rbac/teams/1/roles \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "production-full",
"database_pattern": "production",
"permissions": ["read", "write", "delete"]
}'
# Analytics: read-only on production, restricted to specific measurements
curl -X POST http://localhost:8000/api/v1/rbac/teams/2/roles \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "production-readonly",
"database_pattern": "production",
"permissions": ["read"]
}'
```
### Step 4: restrict measurements (optional) [#step-4-restrict-measurements-optional]
```bash
# Analytics team can only see metrics_* and events_* measurements
curl -X POST http://localhost:8000/api/v1/rbac/roles/2/measurements \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"measurement_pattern": "metrics_*"}'
curl -X POST http://localhost:8000/api/v1/rbac/roles/2/measurements \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"measurement_pattern": "events_*"}'
```
## Best practices [#best-practices]
1. **Principle of least privilege** — Start with minimal permissions and expand as needed. Use read-only roles as the default for analytics users.
2. **Use measurement restrictions** — When teams only need access to specific data, restrict by measurement pattern rather than granting full database access.
3. **Use wildcard patterns carefully** — Database pattern `*` grants access to all databases. Use specific patterns when possible.
4. **Pair with audit logging** — Enable [audit logging](/arc-enterprise/security/audit-logging/) to track RBAC changes and access patterns.
5. **Plan your hierarchy** — Design your organization and team structure before implementation. A typical pattern is one organization per company, teams per department or function.
## Next steps [#next-steps]
* [Audit Logging](/arc-enterprise/security/audit-logging/) — Track all access and changes for compliance
* [Query Governance](/arc-enterprise/query/query-governance/) — Add rate limits and quotas per token
# Configuration (/launchpad/administration/configuration)
All configuration is via environment variables. The only strictly required one is `LAUNCHPAD_JWT_SECRET`, without which the app refuses to start in production. Everything else has a sensible default or is optional. Email can also be configured in the UI (the first-run wizard, or the **Email** section under **Settings**), which is stored in the database and takes precedence over the email env vars.
## Required [#required]
| Variable | Purpose |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `LAUNCHPAD_JWT_SECRET` | Secret used to sign session JWTs. Generate one with `openssl rand -hex 32`. **The app won't start in production without it.** In development it falls back to a well-known placeholder, which must never be used in production. |
## Recommended [#recommended]
| Variable | Default | Purpose |
| ----------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LAUNCHPAD_BASE_URL` | none; see below | Public base URL of this deployment. Used for links in emails and as the WebAuthn (passkey) origin. Set it to the URL users actually visit. |
| `LAUNCHPAD_DB_PATH` | `./data/launchpad.db` | Path to the SQLite database file. The Docker image sets this to `/app/data/launchpad.db`. |
| `LAUNCHPAD_DOMAIN` | `arc.localhost` | Display domain shown in the UI. Not wired through the Helm chart or the bundled Compose file; set it via `extraEnv` if you need it. |
| `LAUNCHPAD_ALLOW_PRIVATE_ENDPOINTS` | `false` | Allow registering Arc endpoints on private/localhost addresses. Must be the literal string `true` to take effect. See [below](#private-endpoints). |
This one value drives several behaviors that silently break if it's wrong for how you serve the app:
* **Session cookies**: the `secure` flag tracks the URL's scheme. On a plain-HTTP deployment served over `http://`, `secure` is off so the cookie persists; a mismatch can make login "succeed" then bounce back to the login page.
* **CSRF / `ORIGIN`**: the Docker image derives `ORIGIN` from `LAUNCHPAD_BASE_URL` when `ORIGIN` isn't already set; a wrong value causes form actions (including finishing setup) to fail with a 403.
* **Passkeys & email links**: WebAuthn is bound to this origin, and email links are built from it.
There is no single global default. `npm run dev` serves on `http://localhost:5173` and the dev-mode fallbacks assume it; the Docker image and Helm chart both default to `http://localhost:3000`. Set it explicitly to the exact scheme + host + port users hit (e.g. `http://localhost:3000` or `https://launchpad.example.com`).
## adapter-node / reverse proxy [#adapter-node--reverse-proxy]
Launchpad is a SvelteKit app built with `adapter-node`. It validates the `Origin` header on POST form submissions (CSRF). The Docker image handles this automatically by deriving `ORIGIN` from `LAUNCHPAD_BASE_URL`. For other cases:
| Variable | Purpose |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `ORIGIN` | The public origin, e.g. `http://localhost:3000`. Set this if you run `node build` directly (not via the Docker image). |
| `PROTOCOL_HEADER` | e.g. `x-forwarded-proto`, for a reverse proxy that terminates TLS and forwards the original scheme. |
| `HOST_HEADER` | e.g. `x-forwarded-host`, for a reverse proxy that rewrites the host. |
| `PORT` | Port the server listens on. Default `3000`. |
See the [adapter-node environment variables](https://svelte.dev/docs/kit/adapter-node#Environment-variables-ORIGIN) reference for details.
## Private endpoints [#private-endpoints]
By default Launchpad **rejects** Arc endpoints that resolve to a private, loopback, or link-local address (`localhost`, `127.0.0.1`, `10.x`, `192.168.x`, `*.internal`, cloud metadata, …). This is an SSRF safeguard: the built-in proxy forwards requests to whatever endpoint you register, so untrusted endpoints must not be able to reach internal services.
Set `LAUNCHPAD_ALLOW_PRIVATE_ENDPOINTS=true` only when your Arc server is intentionally on a private network reachable from the Launchpad host (e.g. the same box, the same Docker network, or the same Kubernetes cluster).
Even when it's enabled, the proxy still resolves the target hostname and connects to a pinned IP address rather than re-resolving at connect time, which bounds exposure to DNS rebinding. Resolutions are cached briefly, so the pin is refreshed periodically rather than on literally every request.
## Email (optional) [#email-optional]
Email is normally configured in the UI. As an alternative, operators can set it via env vars (the DB setting wins if both are present). Without any email config, transactional emails (invites, verification, password reset) are **printed to the server console** instead of being sent.
**Mailgun** (both `MAILGUN_API_KEY` and `MAILGUN_DOMAIN` are needed to activate it):
| Variable | Purpose |
| ----------------- | --------------------------------------------------------------------------------- |
| `MAILGUN_API_KEY` | Mailgun API key. |
| `MAILGUN_DOMAIN` | Sending domain. |
| `MAILGUN_API_URL` | `https://api.mailgun.net` (US, the default) or `https://api.eu.mailgun.net` (EU). |
**SMTP (any provider)** — setting `SMTP_HOST` activates it:
| Variable | Purpose |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `SMTP_HOST` | SMTP server host. |
| `SMTP_PORT` | Port. Default `587`. |
| `SMTP_SECURE` | `true` for port 465 (implicit TLS), else `false`. |
| `SMTP_USER` / `SMTP_PASS` | Credentials. |
| `EMAIL_FROM` | From header, e.g. `Arc Launchpad `. Defaults to a `noreply@` address derived from the provider settings. |
## Other optional integrations [#other-optional-integrations]
| Variable | Purpose |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `PUBLIC_TURNSTILE_SITE_KEY` / `TURNSTILE_SECRET_KEY` | Cloudflare Turnstile CAPTCHA on signup. Skipped if unset. |
| `GCHAT_OPS_WEBHOOK_URL` | Google Chat webhook for ops alerting on silent server-side failures. |
| `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` | Credentials for the Google OAuth callback routes. Setting them does not by itself add a Google button to the login page. |
## Put it behind TLS [#put-it-behind-tls]
For anything beyond local testing, run Launchpad behind a reverse proxy that terminates TLS, and set `LAUNCHPAD_BASE_URL` to your public HTTPS URL so cookies, CSRF, email links, and passkey origins all line up. If you're already using Traefik for Arc, the same pattern extends cleanly to Launchpad; see the [Traefik + Let's Encrypt guide](https://basekick.net/blog/arc-traefik?utm_source=docs\&utm_medium=referral\&utm_campaign=launchpad).
# Teams & organizations (/launchpad/administration/teams-and-organizations)
Launchpad is multi-tenant: users belong to **organizations**, and connections (Arc servers) are shared within an organization. This lets a team share access to the same instances with role-based permissions.
## Organizations [#organizations]
An **organization** is a tenant: a group of members who share a set of Arc connections. Every account gets a personal organization automatically; super-admins can create and review more from the **Orgs** page, which only they can see.
* **Create organization**: give it a name; you become the owner. You can invite a different owner from the org's member list afterward.
* **All organizations**: each org lists its members, their roles, and how many instances it holds.
Switch the active organization from the selector at the top of the sidebar. Connections and console access are scoped to the active org.
## Members and roles [#members-and-roles]
Invite people into an organization from the **Team** page: enter an email address, pick a role, and **Send invite**.
| Role | Can do |
| ---------- | ------------------------------------------------------------------------------- |
| **Owner** | Full access. Manage instances, team, and roles. |
| **Admin** | Connect and remove instances. Invite members and viewers. |
| **Member** | Connect and remove instances. View the team. |
| **Viewer** | Read-only. View instances and the team, but cannot connect or modify instances. |
Owners and admins get the management surface: they can invite people, and they alone can create and change alerts. Only an owner can change another member's role, and the owner role can't be handed out through an invitation. **Viewer** is the read-only role; a **member** can still connect and remove instances.
Roles govern what you can do *in Launchpad*. What any of them can do on a given Arc instance also depends on the **token** registered for that connection: a token without admin permission limits the connection to querying and schema browsing no matter your role. See [Connecting to Arc](/launchpad/getting-started/connecting-to-arc/#getting-the-arc-admin-token).
## Platform users & super-admins [#platform-users--super-admins]
The **Platform users** section (Orgs page) manages accounts across the whole deployment:
* **Invite a new user by email**: a personal organization is created for the new user on acceptance.
* **Super-admin**: super-admins can manage all organizations, not just their own. Grant this only to operators who should see everything.
## Signup and invitations [#signup-and-invitations]
The **first account you create becomes the admin.** After that, self-service signup is closed by default; additional users join **by invitation only**. It's your deployment; you decide who's in it.
Invitations are delivered by email if you've configured a provider (Mailgun or SMTP). Without one, invitation links are printed to the server console instead; copy them from the logs and share them directly. See [First-run setup](/launchpad/getting-started/first-run-setup/#configure-email-optional).
## Authentication [#authentication]
Launchpad uses local authentication, with no external identity provider required:
* **Email + password**: bcrypt-hashed, with strength requirements.
* **MFA (TOTP)**: optional time-based one-time-password second factor, with recovery codes.
* **Passkeys (WebAuthn)**: optional passwordless / hardware-key sign-in.
Auth endpoints (signup, login, password reset, MFA, and WebAuthn) are rate-limited per IP or per account. Manage your own MFA and passkeys under **Settings**.
WebAuthn is bound to an origin. Make sure `LAUNCHPAD_BASE_URL` matches the URL users actually visit, or passkey registration/authentication will fail. See [Configuration](/launchpad/administration/configuration/).
# Connecting to Arc (/launchpad/getting-started/connecting-to-arc)
Launchpad doesn't host databases; you point it at an Arc server you already run. Each server you register is called a **connection** (or instance) in the UI.
## Add a connection [#add-a-connection]
From the sidebar, go to **Instances**, then click **Connect Instance**. That opens the **Connect an Arc server** form.
Fill in:
| Field | Description |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Name** | Optional label to identify this server (e.g. "Production", "Arc running in Docker"). |
| **Arc server URL** | Base URL of the Arc HTTP API, e.g. `https://arc.example.com:8000`. On a shared Docker network this is `http://arc:8000`. |
| **Admin / API token** | A bearer token with access to this Arc server. Stored securely and used to authenticate queries and admin actions. |
Click **Connect**. Launchpad verifies the endpoint and token, then the connection appears on your Instances page.
## Getting the Arc admin token [#getting-the-arc-admin-token]
If Arc has auth enabled, you need an admin (or sufficiently scoped) token. On a fresh Arc instance, an admin token is generated on first run; grab it from the logs:
```bash
docker logs arc | grep -i "admin token"
```
If your Arc runs with auth disabled, any non-empty value is accepted but unused; admin actions still work against that instance.
The token you register determines what Launchpad can do on that instance. An **admin-scoped** token unlocks the operational surface (tokens, retention, alerts, continuous queries, MQTT). A read-only token can still run SQL and browse schemas, but management tabs that require admin access will say so.
## Reaching Arc on a private network [#reaching-arc-on-a-private-network]
By default Launchpad rejects Arc endpoints that resolve to a private, loopback, or link-local address (`localhost`, `127.0.0.1`, `10.x`, `192.168.x`, `*.internal`, cloud metadata, …). This is an SSRF safeguard.
If your Arc server legitimately runs on a private network reachable from the Launchpad host (the same box, the same Docker network, or the same Kubernetes cluster), set `LAUNCHPAD_ALLOW_PRIVATE_ENDPOINTS=true`. Even then, the proxy resolves-and-pins the target IP per request, so it stays safe against DNS rebinding. See [Configuration](/launchpad/administration/configuration/#private-endpoints).
Some URL notes for containerized setups:
* **Same Docker network:** use the service name, e.g. `http://arc:8000`.
* **Arc on the host, Launchpad in a container:** `localhost` inside the container is the container itself. Reach a host-side Arc via `http://host.docker.internal:8000` or the host's LAN IP.
## Open the console [#open-the-console]
Once connected, click the instance to open its console. From there you can run SQL and manage the instance. See [Using the Console](/launchpad/using-the-console/sql-console/).
# First-run setup (/launchpad/getting-started/first-run-setup)
The first time you open Launchpad, it funnels you to a one-time setup wizard. **The first account you create becomes the administrator.** After that, self-service signup is closed by default; additional users join by invitation only.
## Create the admin account [#create-the-admin-account]
Open your Launchpad URL (e.g. **[http://localhost:3000](http://localhost:3000)**). You'll see the setup wizard.
Fill in:
* **First name** / **Last name**
* **Email**: becomes your login and the owner of the first organization
* **Password**: 8 to 128 characters, with an uppercase letter and a number or symbol
Click **Continue**.
## Configure email (optional) [#configure-email-optional]
The second step lets you configure an email provider for invitations, verification, and password resets.
Choose **None**, **SMTP**, or **Mailgun**:
* **None**: email is skipped. Invitation and reset links are printed to the server console instead of being sent. This is fine for a single-admin or evaluation deployment.
* **SMTP**: provide the from address, host, port, TLS setting, and (optionally) username and password.
* **Mailgun**: provide the from address, sending domain, API key, and the API base URL for your region.
Either provider lets you send a test message before committing, so you can confirm delivery works.
Email configuration is best-effort and never blocks admin creation; you can set or change it later under **Settings**, in the **Email** section.
Click **Finish setup** to create your admin account. You'll be signed in and dropped on the dashboard.
Once an account exists, the setup wizard is permanently closed and redirects to the login page. There's no way to re-run it against an existing database.
## Next [#next]
Now connect your first Arc server: [Connecting to Arc](/launchpad/getting-started/connecting-to-arc/).
# Installation (/launchpad/getting-started/installation)
Launchpad ships as a container image, a Helm chart, and source. Pick the path that fits your environment. The only strictly required configuration is `LAUNCHPAD_JWT_SECRET`, without which the app refuses to start in production.
## Docker Compose (with Arc) [#docker-compose-with-arc]
The fastest way to try Launchpad is to bring up both Arc and Launchpad together. Create a `docker-compose.yml`:
```yaml
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
container_name: arc
restart: unless-stopped
ports:
- "8000:8000"
environment:
- STORAGE_BACKEND=local
volumes:
- arc-data:/app/data
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8000/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 15s
launchpad:
image: ghcr.io/basekick-labs/launchpad:latest
container_name: launchpad
restart: unless-stopped
depends_on:
- arc
ports:
- "3000:3000"
environment:
# REQUIRED: signs session tokens. Generate with: openssl rand -hex 32
- LAUNCHPAD_JWT_SECRET=${LAUNCHPAD_JWT_SECRET:?set LAUNCHPAD_JWT_SECRET}
# Public URL of this deployment (email links + passkey origin).
- LAUNCHPAD_BASE_URL=${LAUNCHPAD_BASE_URL:-http://localhost:3000}
- LAUNCHPAD_DB_PATH=/app/data/launchpad.db
# Arc is on the same Docker network (a private address), so allow it.
- LAUNCHPAD_ALLOW_PRIVATE_ENDPOINTS=true
volumes:
- launchpad-data:/app/data
healthcheck:
test: ["CMD", "node", "-e", "require('http').get('http://127.0.0.1:3000/',r=>process.exit(r.statusCode<500?0:1)).on('error',()=>process.exit(1))"]
interval: 30s
timeout: 5s
retries: 3
start_period: 15s
volumes:
arc-data:
launchpad-data:
```
Bring it up:
```bash
LAUNCHPAD_JWT_SECRET=$(openssl rand -hex 32) docker compose up -d
```
Then open **[http://localhost:3000](http://localhost:3000)** and continue with [First-run setup](/launchpad/getting-started/first-run-setup/).
By default Launchpad **rejects** Arc endpoints that resolve to a private, loopback, or link-local address. This is an SSRF safeguard, since its proxy forwards requests to whatever endpoint you register. Because Arc here lives on the same Docker network, the compose sets `LAUNCHPAD_ALLOW_PRIVATE_ENDPOINTS=true`. See [Configuration](/launchpad/administration/configuration/#private-endpoints).
## Standalone Docker [#standalone-docker]
Point Launchpad at an Arc you already run elsewhere:
```bash
docker run -p 3000:3000 \
-e LAUNCHPAD_JWT_SECRET=$(openssl rand -hex 32) \
-e LAUNCHPAD_BASE_URL=http://localhost:3000 \
-v launchpad-data:/app/data \
ghcr.io/basekick-labs/launchpad:latest
```
The SQLite database is written to `/app/data/launchpad.db`; mount a volume there to persist it.
Set `LAUNCHPAD_BASE_URL` to the public URL you actually serve on. It also drives the `ORIGIN` used for CSRF and the `secure` flag on session cookies, so a mismatch can cause login or form actions to fail. See [Configuration](/launchpad/administration/configuration/).
## Helm (Kubernetes) [#helm-kubernetes]
```bash
helm install launchpad oci://ghcr.io/basekick-labs/charts/launchpad \
--set jwtSecret=$(openssl rand -hex 32) \
--set baseUrl=https://launchpad.example.com
```
Common values:
| Value | Default | Purpose |
| ------------------ | ----------------------- | -------------------------------------------------------------------- |
| `jwtSecret` | `""` | **Required** unless `existingSecret` is set. Signs session tokens. |
| `existingSecret` | `""` | Name of a pre-created Secret holding `LAUNCHPAD_JWT_SECRET` instead. |
| `baseUrl` | `http://localhost:3000` | Public URL (email links + passkey origin). |
| `persistence.size` | `1Gi` | PVC size for the SQLite database. |
| `ingress.enabled` | `false` | Enable to expose via an Ingress. |
See `helm/launchpad/values.yaml` in the repo for the full list.
## From source [#from-source]
Requires Node.js 20+.
```bash
git clone https://github.com/Basekick-Labs/launchpad
cd launchpad
npm install
cp .env.example .env # then edit .env: set LAUNCHPAD_JWT_SECRET at minimum
npm run dev # http://localhost:5173
```
For a production build:
```bash
npm run build
LAUNCHPAD_JWT_SECRET=$(openssl rand -hex 32) node build
```
The server listens on `$PORT` (default `3000`).
## Next steps [#next-steps]
* [First-run setup](/launchpad/getting-started/first-run-setup/): create the admin account
* [Connecting to Arc](/launchpad/getting-started/connecting-to-arc/): register your first Arc server
* [Configuration](/launchpad/administration/configuration/): the full environment-variable reference
# Alerts (/launchpad/using-the-console/alerts)
Alerts watch your data and notify a webhook when a condition is met. Launchpad evaluates them on a schedule, querying the connected Arc instance for the value. Create and manage them from the **Alerts** tab.
## Create an alert [#create-an-alert]
Click **Create Alert** and configure:
| Field | Description |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Alert Name** | A label for the alert. |
| **Condition** | The comparison to apply against a threshold: greater than, less than, equals, not equals, or contains. The alert fires when it matches. |
| **Check Interval** | How often the condition is evaluated. The minimum accepted interval is 1 minute. |
| **Webhook URL** | Optional endpoint to notify when the alert fires (e.g. Slack, Discord, or a custom receiver). Must be a public HTTPS URL. |
| **Enable alert immediately** | Whether it starts active or paused. |
The notification message is generated for you from the rule: the alert name, the observed value, the condition, and the threshold.
## Monitor and manage [#monitor-and-manage]
Each alert card shows its condition, check interval, last value, and how many times it has triggered. Expand **Recent Triggers** for the time, value, and message of past firings.
From the same card you can **Edit**, **Disable** / **Enable**, **Test Now**, or **Delete** an alert. **Test Webhook** sends a sample payload so you can confirm the receiver works before the alert ever fires.
Unlike the other management tabs, alert rules live in Launchpad, so access is governed by your **organization role**: creating, editing, and deleting alerts requires the owner or admin role in the active organization. See [Teams & organizations](/launchpad/administration/teams-and-organizations/).
Alerts are evaluated and delivered by **Launchpad**, not by Arc. Launchpad queries Arc for the value, compares it against the threshold, and posts to the webhook itself. The webhook must therefore be reachable from the Launchpad host, and the rule and its history live in Launchpad's database rather than in Arc.
# Continuous queries (/launchpad/using-the-console/continuous-queries)
Continuous queries roll up and downsample data on a schedule: read from a source, aggregate over a window, and materialize the result into a destination measurement. They're how you keep long-term storage cheap while preserving the summaries you actually query.
## Create a continuous query [#create-a-continuous-query]
From the **Continuous Queries** tab, click **Create Query** and configure:
| Field | Description |
| ------------------------------ | ------------------------------------------------------------------------------- |
| **Database** | The database the query runs in. |
| **Source Measurement** | The measurement to read from. |
| **Destination Measurement** | Where the aggregated result is written. |
| **Interval** | How often the query runs, chosen from the interval picker. |
| **Delete Source After (days)** | Optional. Drop the raw source data this many days after it has been aggregated. |
| **Retention Days** | Optional. Drop the rolled-up data after this many days too, to cap storage. |
| **Enable query immediately** | Whether it runs on a schedule, or is created paused. |
The query body must reference the `{start_time}` and `{end_time}` placeholders, which Launchpad substitutes with the bounds of each run's window.
## Run and manage [#run-and-manage]
Each continuous query can be **Edited**, **Executed Now** on demand, paused with **Pause** / **Activate**, or **Deleted**. The execute dialog offers a dry run that previews the query without writing anything, so you can check a rollup before it materializes. Run one manually to backfill or test it, then enable it to run continuously.
Continuous queries depend on Arc's support for them on that instance. If the tab shows "Continuous Queries Not Available", the connected Arc version or configuration doesn't expose the feature.
Managing continuous queries requires an admin-scoped connection.
# Logs & monitoring (/launchpad/using-the-console/logs-and-monitoring)
Two console tabs cover observability, but they observe different things. **Log Viewer** is for the logs you *store in Arc*. **Monitoring** is Arc's own self-observability.
## Log viewer [#log-viewer]
The **Log Viewer** tab reads the log data you've ingested into Arc: application logs, structured events, anything you write to a log measurement. Point it at your log tables and slice through them without writing the SQL by hand.
It adds tooling on top of the raw rows:
* **Patterns**: groups recurring shapes across your log lines so you can spot what's noisy. It needs a detectable message field, and is unavailable without one.
* **Traces**: pulls the entries belonging to one trace together and renders them as a waterfall, for following a request across services. It needs a `trace_id` field.
This is the right tab when you need to see what *your application* is doing, as recorded in Arc.
## Monitoring [#monitoring]
The **Monitoring** tab is Arc's own self-observability: how the *instance itself* is doing, right alongside your data.
It shows the instance's operational signals: ingestion throughput, query activity, and internal metrics. Use it to answer "is Arc healthy and keeping up?" rather than "what's in my data?".
- **Log Viewer** → the logs your systems write *into* Arc.
- **Monitoring** → how the Arc server is performing.
# MQTT ingestion (/launchpad/using-the-console/mqtt-ingestion)
If you're pulling sensor and IoT data over MQTT, the **MQTT** tab manages the whole ingestion path from the browser, with no broker config files and no restarts. Each **subscription** tells Arc which broker to connect to, which topics to consume, and where to land the data.
## Add a subscription [#add-a-subscription]
Click **Add Subscription** and configure the broker connection:
### Broker connection [#broker-connection]
| Field | Description |
| -------------------------- | --------------------------------------------------------------------------------------------------------- |
| **Name** | A label for the subscription. |
| **Broker URL** | e.g. `tcp://broker:1883`. Accepted schemes: `tcp://`, `ssl://`, `ws://`, `wss://`, `mqtt://`, `mqtts://`. |
| **Client ID** | Optional MQTT client identifier. |
| **Username / Password** | Optional broker credentials. |
| **Target Database** | The Arc database that consumed messages land in. |
| **QoS** | Quality of service: 0 (at most once), 1 (at least once), or 2 (exactly once). |
| **Auto-start on Arc boot** | Bring the subscription up automatically with the instance. |
### Topics and routing [#topics-and-routing]
* **Topics**: one topic per line. The MQTT wildcards `+` and `#` are allowed, so `sensors/#` works.
* **Topic → database mapping**: optional `topic = database` lines that route different topics into different databases, so each stream lands where it belongs.
### Secure transport (TLS / mTLS) [#secure-transport-tls--mtls]
Enable TLS for encrypted broker connections, including full **mTLS**. Give the **CA certificate path**, **client cert path**, and **client key path** as paths on the Arc host (e.g. `/etc/arc/ca.pem`), not uploads. A **Skip TLS verification** option exists for testing against self-signed brokers.
### Connection tuning [#connection-tuning]
Under **Advanced**, keep-alive, connect timeout, and reconnect back-off (min and max) are all configurable in seconds, so you can match the subscription to your broker's behavior.
## Live stats and lifecycle [#live-stats-and-lifecycle]
A running subscription card shows the broker, target database, topic count, and **live stats for the current session**: messages **Received**, **Failed**, **Bytes**, **Reconnects**, and the **Last message** time.
A running subscription offers **Stop** and **Pause**; a stopped one offers **Start**. **Restart**, **Edit**, and **Delete** are always available.
A subscription must be stopped to change its configuration: stop it, edit, then start it again.
The MQTT tab is always present, but it only manages subscriptions when MQTT is enabled on that Arc instance; otherwise it says so and points you at the `[mqtt]` section of `arc.toml`. Managing subscriptions also requires the owner or admin role in the active organization.
# Retention policies (/launchpad/using-the-console/retention-policies)
Retention policies delete aged data automatically, so storage doesn't grow without bound. Launchpad drives Arc's admin API to create and run them from the **Retention** tab, with no config files and no curl.
## Create a policy [#create-a-policy]
Click **Create Policy** and configure:
| Field | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------- |
| **Database** | Which database the policy applies to. |
| **Measurement** | Optional. A specific measurement (table); leave it as **All measurements** to cover the whole database. |
| **Retention Period (days)** | How long to keep data. Anything older than the cutoff becomes eligible for deletion. |
| **Buffer Period (days)** | An optional grace window beyond the retention period before data is actually removed. |
| **Enable policy immediately** | Whether the policy runs on its own, or is created paused. |
## Dry-run before you delete [#dry-run-before-you-delete]
Before anything is removed, Launchpad runs a **dry-run** for you. Opening **Execute Now** first shows **Dry-Run Results**: the records and files the policy *would* delete, the cutoff date, and the affected measurements, so you're never guessing about the blast radius. Review it, then confirm to execute for real. When the dry-run finds nothing to delete, the execute button stays disabled.
## Run and manage [#run-and-manage]
Each policy card shows its target (database + measurement), retention period, buffer period, last execution time, and last-deleted count. From the card you can:
* **Edit**: change the periods or scope
* **Pause** / **Activate**: toggle whether it runs automatically
* **Execute Now**: dry-run, then run it on demand
* **Delete**: remove the policy
An expanded policy card also lists its past runs.
Retention management drives Arc's admin API, so the connection must use an admin-scoped token. See [Connecting to Arc](/launchpad/getting-started/connecting-to-arc/#getting-the-arc-admin-token).
If the tab reports that retention policies are not available, retention isn't enabled on that Arc instance. Enable the `retention` section in its `arc.toml` and reconnect.
# SQL console (/launchpad/using-the-console/sql-console)
The SQL console is where you run queries against a connected Arc instance. Open a connection from **Instances** and you land on the console, with a tab bar across the top: **SQL Console · Log Viewer · Monitoring · Retention · Continuous Queries · Alerts · MQTT · Tokens**.
## Schema explorer [#schema-explorer]
The left panel has two tabs:
* **Tables**: a tree of your databases and their measurements (tables). Click a database to expand it and see its tables; the active database is highlighted. Use the **+** to create a database and the refresh icon to reload the tree.
* **History**: your recent queries, so you can re-run or tweak past work.
Selecting a database sets it as the query target (shown as a chip in the top-right of the editor).
## Running a query [#running-a-query]
Type SQL in the editor and click **Execute** (or press the shortcut shown on the button). Select part of the query first and a **Run Selection** button appears, which runs only the highlighted statement.
```sql
SELECT * FROM citibike_trips LIMIT 100;
```
Arc speaks standard analytical SQL, so window functions, CTEs, and joins all work. See the [Arc SQL reference](/arc/) for the full dialect.
## Multi-statement scripts [#multi-statement-scripts]
Run several statements in one go and the results panel gives each its own tab, so you can page through the output of a script without splitting it up by hand.
## Exporting results [#exporting-results]
Below the result grid, export the current result set:
* **CSV**: download as a `.csv` file
* **JSON**: download as JSON
* **Markdown**: copy a Markdown table to the clipboard (handy for issues and docs)
* **Show Chart**: render the result as a quick chart. It appears only when the result has a numeric column to plot.
## Tips [#tips]
* The status bar at the bottom shows which connection you're on and its Arc endpoint, alongside the row count and execution time of the last query.
* Query history is per-connection, so switching instances gives you that instance's history.
# API tokens (/launchpad/using-the-console/tokens)
The **Tokens** tab manages the API tokens your ingestion pipelines and applications use to authenticate against Arc. This is the full credential lifecycle (create, scope, disable, and revoke) from the browser.
## Create a token [#create-a-token]
Click **Create Token** and provide:
| Field | Description |
| --------------- | -------------------------------------------------------------------------------- |
| **Name** | Required. Identifies the token in the list. |
| **Description** | Optional note on what the token is for (e.g. "Telegraf ingest", "Grafana read"). |
| **Permissions** | Independent checkboxes, so you grant exactly what the consumer needs. |
| **Expiration** | How long the token stays valid, or never expires. |
The four permissions are:
| Permission | Grants |
| ---------- | -------------------------------------------- |
| **Read** | Query data from databases. |
| **Write** | Insert and write data to databases. |
| **Delete** | Delete data from databases. |
| **Admin** | Manage tokens, settings, and admin features. |
New tokens default to **Read** and **Write**.
When the token is created it's shown **once**: copy it immediately and store it securely. The plaintext can't be retrieved later, so a lost token has to be replaced rather than recovered.
## Manage tokens [#manage-tokens]
The token list shows each token's name, description, and permissions. From there you can:
* **Revoke**: disable a token without deleting it, keeping the record so you can re-enable it later.
* **Enable**: reactivate a revoked token.
* **Delete**: permanently remove it.
Rotating a credential is: create the new token, roll it out to the consumer, then delete the old one.
Creating and managing tokens requires an **admin-scoped** connection. If your connection uses a read-only token, the tab will show "Admin Permission Required"; register an admin token for that instance to manage credentials. See [Connecting to Arc](/launchpad/getting-started/connecting-to-arc/#getting-the-arc-admin-token).
Give each consumer its own token, with only the permissions it needs (an ingest pipeline rarely needs **Delete** or **Admin**) and a clear description. That way you can disable or rotate one consumer without touching the others.
# tsm2arc (/arc/migration/tsm2arc)
[tsm2arc](https://github.com/Basekick-Labs/tsm2arc) (Apache-2.0) migrates **InfluxDB 1.x (1.7/1.8) and 2.x (2.0–2.7)** data into Arc by reading TSM **and WAL** files **directly off disk**. No running `influxd` is required, which makes it the right tool when the InfluxDB data sits on cold or unmounted volumes (an EBS snapshot, a decommissioned server's disk) that can be mounted read-only but are not served by any InfluxDB instance.
The on-disk TSM/WAL format is the same across 1.x and 2.x, and tsm2arc auto-detects the layout. For 2.x it resolves bucket IDs to readable names from `influxd.bolt` and skips InfluxDB's internal system buckets (`_monitoring`, `_tasks`). InfluxDB 3.x stores Parquet rather than TSM, so for 3.x use the [Line Protocol export/import path](/arc/migration/influxdb/#step-3-migrate-historical-data) instead.
Live ingestion moves with a URL change (Arc speaks Line Protocol natively; dual-write from Telegraf and cut over). For history: tsm2arc for 1.x/2.x datasets, especially large ones on cold volumes; Line Protocol export/import for 3.x or small datasets. The [InfluxDB migration guide](/arc/migration/influxdb/) covers the whole journey; these pages are the full tsm2arc reference.
## What it does [#what-it-does]
InfluxDB stores each field of a point as a separate TSM key, each with its own timestamp and value stream. tsm2arc:
1. Parses the TSM index (header/index/footer) of each file.
2. Decodes every block with native Go implementations of the TSM codecs (timestamp, float, integer, unsigned, boolean, string), validated against the real InfluxDB encoder and cross-checked against real 2.7 data.
3. **Rejoins fields by (series, timestamp)** so multi-field points are reconstructed as single Line Protocol lines. (This is the correctness gap in `influx_inspect export`, which emits one line per field.)
4. Streams gzipped, size-bounded chunks into Arc's `/api/v1/import/lp` endpoint, in parallel across shards, with a SQLite checkpoint for crash-safe resume.
Capabilities at a glance:
* Native TSM reader, field rejoin, LP encode, and `--dry-run`
* Chunked gzip POST to Arc `/api/v1/import/lp` with per-database routing
* SQLite checkpoint with crash-safe, cursor-seeking resume
* WAL (`.wal`) reader, merged with TSM per shard
* Parallel workers (`--workers`) with live progress reporting
* InfluxDB 2.x layout auto-detection and bucket-name resolution
* Measurement rename map and invalid-name policy with a checkpoint audit trail
* Index-only shard profiling (`--analyze`), shareable with `--redact`
## Install [#install]
Download a prebuilt binary from [Releases](https://github.com/Basekick-Labs/tsm2arc/releases) (Linux and macOS on amd64/arm64, Windows on amd64), or:
```bash
# from source (Go 1.25+)
go install github.com/basekick-labs/tsm2arc/cmd/tsm2arc@latest
# container (multi-arch, linux amd64/arm64)
docker run --rm ghcr.io/basekick-labs/tsm2arc:latest --version
```
Each release ships an SBOM (SPDX) and `checksums.txt`.
## Quick start: Dry run first [#quick-start-dry-run-first]
A dry run discovers shards, decodes every block, reconstructs points, and prints per-database counts plus sample Line Protocol, **without writing to Arc**. It is the safe first contact with the source data.
```bash
# InfluxDB 1.x: point at the data dir or its parent (layout auto-detected)
tsm2arc --datadir /var/lib/influxdb --dry-run --sample 10
# InfluxDB 2.x: point at the v2 root; engine/data, engine/wal, and
# influxd.bolt (for bucket names) are auto-detected
tsm2arc --datadir /var/lib/influxdb2 --dry-run --sample 10
```
Then load:
```bash
tsm2arc \
--datadir /mnt/influxdb/data \
--waldir /mnt/influxdb/wal \
--arc-url https://arc.example.net \
--token "$ARC_TOKEN" \
--verbose
```
InfluxDB does not flush the write-ahead log to TSM on shutdown, so small or recently written shards can live entirely in `.wal` files. Without `--waldir`, that data is silently skipped. For 2.x the WAL directory is auto-detected from `engine/wal`; for 1.x pass it explicitly. When a point exists in both TSM and WAL, the WAL value wins (last-write-wins, matching InfluxDB and Arc compaction).
## Where to go next [#where-to-go-next]
The repo's [DESIGN.md](https://github.com/Basekick-Labs/tsm2arc/blob/main/docs/DESIGN.md) covers the internals: the verified Arc ingest constraints, the resume protocol, and the duplicate-bounding argument.
The extraction side produces standard InfluxDB Line Protocol; the Arc sink is just the first sink. The TSM/WAL decoder is Apache-2.0 and contributions of new sinks (ClickHouse, QuestDB, TimescaleDB) are welcome. See [CONTRIBUTING.md](https://github.com/Basekick-Labs/tsm2arc/blob/main/CONTRIBUTING.md).
# Performance and scaling (/arc/migration/tsm2arc/performance)
The knobs that matter, in order: `--workers` (sized against the **Arc node**, not the migration host), `--chunk-bytes` (per-request memory on both sides), and for shard-shaped bottlenecks, `--shard-split` with `--merge-memory`. None of them affect correctness or resume; all of them are safe to change between runs except the shaping flags the checkpoint fingerprints (see the [runbook](/arc/migration/tsm2arc/runbook/#if-it-stops-just-resume)).
## Size `--workers` against the Arc node [#size---workers-against-the-arc-node]
`--workers N` migrates N shards concurrently (default 2). Shards are fully independent, each with its own chunk sequence and checkpoint rows, so this scales cleanly. The binding constraint is the **Arc node's memory**: Arc's import endpoint buffers each request fully in memory while parsing, so peak transient Arc-side memory is roughly:
```text
workers × (~1 to 1.3 GB) at the default 450 MB chunk size
```
Guidance:
* The default `2` is safe for almost any Arc node.
* If the Arc node has RAM headroom, raise it (4 to 8) for throughput. The big dedicated migration host is rarely the bottleneck; Arc is.
* On Arc 429s or memory pressure, lower `--workers` (the tool backs off on 429 automatically, but fewer workers reduces peak pressure). Lowering `--chunk-bytes` (e.g. `200MB`) also reduces per-request memory, at the cost of more requests; note it is a checkpoint-fingerprinted shaping flag, so change it between migrations, not mid-checkpoint.
## Migration-host memory [#migration-host-memory]
* **Extraction** (`--dry-run` and the read side of a load) streams one series at a time and, within a series, one TSM block at a time. Peak heap is a few MiB and does **not** grow with the shard, the dataset, or the largest series (measured at 3.7/3.8/3.9 MiB for series of 500K/2M/8M values).
* **The load adds the chunk buffers**: each worker holds up to `--chunk-bytes` of raw Line Protocol being accumulated plus, since 0.1.5, a second chunk in flight to Arc (extraction and upload overlap by default). Budget roughly `workers × 2 × chunk-bytes` (e.g. `4 × 2 × 450 MB ≈ 3.6 GB`). `--pipeline=false` reverts to serial send and halves that. This dominates the host budget.
* **The index cache** (`--index-cache`, default 2 GiB per in-flight shard, 0 disables) keeps parsed TSM file indexes so per-series file reopens do not re-parse them, a large CPU cost on shards with many series. Worst case adds `workers × index-cache` to the host budget. If the run prints "budget full; raise --index-cache", extraction is paying re-parse CPU; raise it if the host has headroom.
* **File descriptors**: extraction holds one handle per TSM file containing the series currently being merged. Files with non-overlapping time ranges are merged in separate passes, so this is normally one or two; if every file in a shard spans the same time range, budget `workers × files-per-shard` and raise `ulimit -n`.
If the process is OOM-killed during extraction, `--workers` and `--chunk-bytes` are the wrong knobs; they bound the load buffer, not the read. Versions 0.1.3 and earlier held one whole series in memory at roughly 8 to 32 times its compressed on-disk size, so a single very large series could exhaust any instance. Upgrade to 0.1.4+, where extraction memory is flat.
`--start`/`--end` bound work as well as output: blocks outside the window are skipped straight from the TSM index and never read or decoded.
## Intra-shard parallelism [#intra-shard-parallelism]
When a few large shards dominate wall-clock time and cores sit idle, `--shard-split N` merges up to N windows of a shard concurrently while emitting **byte-identical output**: resume, checkpoints, and the audit trail are unaffected, and N may change between a crash and its resume.
Run [`--analyze`](/arc/migration/tsm2arc/runbook/#profile-the-shards-optional) first. It profiles whether each shard's file time ranges partition into windows (SPLIT-FRIENDLY) or overlap fully (OVERLAPPING). On fully overlapping generations, every window still holds the whole run's decoded blocks and the admission budget serializes the tasks, so the flag buys little there.
`--merge-memory` is required with `--shard-split > 1`: a concurrent merge holds roughly one decoded block (\~64 KiB) per (file × field) stream, so wide shards cost GiBs per concurrent task. Size it against free RAM:
```text
(host RAM − workers × 2 × chunk-bytes − index caches) / workers
```
then set `--shard-split` to 2 to 4. A task whose estimate exceeds the budget runs alone (the serial memory profile); that is never an error, just no extra parallelism for that task.
## Spread load across a multi-writer cluster [#spread-load-across-a-multi-writer-cluster]
If Arc Enterprise runs several writer nodes behind a standard Kubernetes `Service`, be aware that a ClusterIP Service balances **per TCP connection**, not per request. tsm2arc keeps HTTP connections alive and reuses them for its large sequential POSTs, so a handful of long-lived connections each stay pinned to whichever pod they first dialed. One writer can end up taking nearly all the import traffic while the others idle.
The fix is on the routing layer, not the client: put an **L7 (HTTP-aware) load balancer** in front of the writers (an ingress controller, Envoy or HAProxy, or a cloud ALB), which balances each request independently. With that in place, import traffic spreads evenly across writers with no tsm2arc changes. See [Deployment patterns](/arc-enterprise/configuration/deployment-patterns/) for the Enterprise-side picture.
# Migration runbook (/arc/migration/tsm2arc/runbook)
This is the operator guide for migrating InfluxDB 1.x/2.x data into Arc with tsm2arc, covering the common case: terabytes of InfluxDB data on a cold or unmounted volume that no running `influxd` serves. For flag reference and install, see the [tsm2arc overview](/arc/migration/tsm2arc/); for sizing and throughput, see [Performance and scaling](/arc/migration/tsm2arc/performance/).
## Before you start [#before-you-start]
* [ ] The InfluxDB data volume is mounted **read-only** on the migration host (`mount -o ro /dev/xvdf1 /mnt/influx`). tsm2arc only ever reads the source files, but a read-only mount means a mistake cannot damage them.
* [ ] You know the InfluxDB root or data path (1.x: `.../influxdb` or `.../data`; 2.x: the v2 root containing `engine/` and `influxd.bolt`).
* [ ] You have an **admin-tier** Arc API token (the import endpoint requires admin) and the network path to the Arc base URL is open.
* [ ] You have somewhere durable for the SQLite checkpoint file (tiny, KB to MB).
## Understand the source layout [#understand-the-source-layout]
tsm2arc auto-detects 1.x vs 2.x; the TSM/WAL file format is identical between them, only the directory layout differs.
```text
# InfluxDB 1.x
/data////*.tsm # compacted data
/wal////*.wal # un-flushed data
# InfluxDB 2.x (default root ~/.influxdbv2)
/engine/data//autogen//*.tsm
/engine/wal//autogen//*.wal
/influxd.bolt # bucket id-to-name map
```
On a cold-volume migration, copy `influxd.bolt` alongside the `engine/` directory (or point `--bolt` at it). Without it, tsm2arc cannot recover bucket names: buckets migrate under their 16-hex IDs and the `_monitoring`/`_tasks` system buckets cannot be skipped (it warns loudly). Resume is robust either way, since the checkpoint keys on the stable bucket ID, but keep the bolt available for the whole migration so Arc database names stay consistent.
Confirm what you have before running anything:
```bash
ls /mnt/influx/data # databases present
find /mnt/influx/data/ -name '*.tsm' | head # shards and TSM files
find /mnt/influx/wal/ -name '*.wal' -size +0c # non-empty WAL segments
```
## Dry run: The safe first contact [#dry-run-the-safe-first-contact]
```bash
tsm2arc --datadir /mnt/influx/data --waldir /mnt/influx/wal --dry-run --sample 10
```
Check the output:
* **Databases** match what you expect. `_internal` (InfluxDB's own monitoring DB) is skipped by default; pass `--include-internal` only if you truly want it.
* **points / fields / keys** are non-zero and roughly the expected magnitude; **skipped-keys** should be 0 or explainable.
* **Time range** looks sane. Pre-1970 timestamps are supported and show as negative epoch dates.
* **Sample Line Protocol** lines look right: measurement names, tags, field types (`i` integer, `u` unsigned, quoted strings, booleans).
* **`INVALID:` lines** list measurement names Arc would reject, with point counts, so you can author renames before the load (next section).
If an expected database is missing, check whether its data is WAL-only and whether you passed `--waldir`.
## Profile the shards (optional) [#profile-the-shards-optional]
`--analyze` reads only the TSM indexes and finishes in seconds: nothing is decoded, nothing is sent. It prints per-shard series/file/key counts and, for the largest merge runs, whether the files' time ranges partition into windows or overlap fully. Run it when discussing throughput, since the profile determines which optimizations help your data shape (see [shard-level parallelism](/arc/migration/tsm2arc/performance/#intra-shard-parallelism)).
```bash
tsm2arc --datadir /mnt/influx/data --analyze
```
If the analysis has to leave your organization (a support ticket, a GitHub issue), add `--redact`: database, retention policy, and series names are replaced with stable hashed pseudonyms (`series_3f9a2c1b04d7`). The report keeps every number but carries no internal identifiers, and the pseudonyms are stable across runs and machines, so a conversation can keep referring to the same series.
## Scope the migration (optional) [#scope-the-migration-optional]
Migrate a subset first to validate the round-trip end to end:
```bash
# one database
tsm2arc --datadir /mnt/influx/data --waldir /mnt/influx/wal --database-filter telemetry --dry-run
# a time window (RFC3339, UTC)
tsm2arc --datadir /mnt/influx/data --waldir /mnt/influx/wal \
--start 2024-01-01T00:00:00Z --end 2024-02-01T00:00:00Z --dry-run
```
`--start`/`--end` skip out-of-window TSM blocks straight from the index, so a window bounds read work and time, not just output. A large source can be migrated in sequential windows; use a **separate `--checkpoint` file per window** (the tool enforces this, refusing to resume a checkpoint whose window changed).
Each source database maps to an Arc database of the same name; rename with `--db-map old=new` (repeatable).
## Handle measurement names Arc rejects [#handle-measurement-names-arc-rejects]
Arc accepts only measurement names matching `^[a-zA-Z][a-zA-Z0-9_-]*$` (the dot is Arc's `database.measurement` separator in queries and RBAC grant keys). InfluxDB is far more permissive, and dotted `.` names are common. tsm2arc validates **client-side, before sending**, so a bad name cannot end a multi-hour load with a mid-flight Arc 400.
1. **Dry-run first.** Every name Arc would reject is listed with its point count.
2. **Author a rename map** with deterministic targets you choose:
```text
# renames.map: one old=new per line; # comments allowed
edge-prod.gateway_services=edge_prod_gateway_services
qa.node-b=qa_node_b
```
Pass it with `--measurement-map-file renames.map` (or inline, repeatable, with `--measurement-map`). Targets are validated at startup, so a typo fails immediately rather than mid-load.
3. **Pick the policy for anything still invalid** with `--on-invalid-measurement`:
* `fail` (default): abort with an actionable error before sending. Keep this with a map; it guarantees nothing unmapped slips through.
* `skip`: drop those points, keep loading, report names and point counts at the end. Use it to land the good data now and deal with stragglers later.
* `map`: deterministic auto-rename (disallowed characters become `_`, an `m_` prefix if the name does not start with a letter). Beware that distinct names can collide after sanitizing (`a.b` and `a_b` both become `a_b`) and would merge; prefer an explicit map when names sit close together.
Every rename and skip is **recorded in the checkpoint** (table `measurement_actions`: source db, shard, source name, final name, origin, point count) and summarized when the run finishes, so renames are auditable and skipped data is on record rather than quietly missing:
```bash
sqlite3 /var/lib/tsm2arc/migration.checkpoint.db \
'SELECT source_db, measurement, action, renamed_to, origin, SUM(points)
FROM measurement_actions GROUP BY source_db, measurement'
```
A name like `has-hyphen` is valid, Arc accepts it at write time and tsm2arc migrates it, but Arc versions before 26.09.1 cannot reference it in SQL at all. From 26.09.1 on, quote it: `FROM "has-hyphen"` (with the `x-arc-database` header) or `FROM "db"."has-hyphen"`. If the target Arc predates 26.09.1 and cannot be upgraded first, rename at migration time (`--measurement-map 'has-hyphen=has_hyphen'`). Data migrated with hyphens before an upgrade is stored correctly and becomes queryable as soon as Arc is upgraded, with no re-migration.
## Run the migration [#run-the-migration]
```bash
export ARC_TOKEN=''
tsm2arc \
--datadir /mnt/influx/data \
--waldir /mnt/influx/wal \
--arc-url https://arc.example.net \
--token "$ARC_TOKEN" \
--workers 4 \
--checkpoint /var/lib/tsm2arc/migration.checkpoint.db \
--verbose
```
A heartbeat line reports progress while it runs:
```text
[12/40 shards] 3821 chunks, 18402991 rows, 4210.5 MB raw — 38211 rows/s, 9.4 MB/s (480s)
```
* Put the `--checkpoint` file somewhere durable and **keep it**: it is how resume works, and its audit tables are your record of exactly what was sent.
* Run one process; scale with `--workers` (size it against the Arc node, see [Performance and scaling](/arc/migration/tsm2arc/performance/)). Never run two processes against the same checkpoint file.
* Transient failures (429, 5xx, network) retry with exponential backoff; 4xx errors are permanent and abort the run.
## If it stops: Just resume [#if-it-stops-just-resume]
Re-run the **exact same command**. tsm2arc skips shards already fully migrated, and for a partially migrated shard it **seeks** to the stored cursor: series before it are never read, already-sent TSM blocks are skipped at the index without being decoded, and sending resumes from the first un-acknowledged chunk within seconds to minutes. Chunk progress commits only after Arc returns 2xx, and Arc's import handler flushes to storage before returning, so 2xx means durably persisted.
Resume requires the same shaping flags: the checkpoint fingerprints `--chunk-bytes`, `--start`, `--end`, `--db-map`, `--precision`, the measurement-map flags, and `--on-invalid-measurement`. Changing any of them would misalign chunk boundaries, so tsm2arc refuses with `checkpoint was created with different settings` rather than corrupting the migration. To change a shaping flag, start a fresh `--checkpoint` (a full re-migration).
**Duplicates:** a clean, uninterrupted run produces **zero** duplicates. The only duplication window is a crash between Arc persisting a chunk and the tool recording it; on resume that single chunk is re-sent. Arc compaction collapses the duplicate for tag-bearing series automatically; tagless series can retain at most one chunk of duplicate rows per shard per crash (bounded and attributable, the `--verbose` log names the re-sent chunk).
## Verify counts before declaring success [#verify-counts-before-declaring-success]
Count reconciliation is how you know the migration is complete and correct.
```bash
# Tool side: re-run --dry-run with the SAME flags (including --waldir/--start/--end)
tsm2arc --datadir --waldir --dry-run --sample 0
# Arc side: count rows per measurement, with the same time bounds if you used a window
curl -s -X POST "https://arc.example.net/api/v1/query" \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT count(*) FROM ."}'
```
* **Tag-bearing data**: after Arc compaction runs, counts should match exactly.
* **Tagless data**: a small positive delta on Arc's side is resume-overlap duplicates (bounded per the previous section); a clean run shows no delta.
* **A short count on Arc's side** is the signal to investigate: check the `--verbose` log for `WARN` lines, non-zero `skipped-keys`, or an errored shard, then resume.
* If you loaded with `--on-invalid-measurement=skip`, subtract the skipped point counts (printed at run end and stored in `measurement_actions`) before comparing. If you used a rename map, query Arc under the **renamed** names.
Also confirm WAL coverage if you did not pass `--waldir` (`find -name '*.wal' -size +0c`; non-empty segments mean un-migrated data, so re-run with `--waldir` and the same checkpoint), and spot-check a few series' min/max time and sampled values against the dry-run output.
## Cleanup [#cleanup]
Keep the checkpoint until the migration is **verified**; delete it only to force a full re-migration. Unmount the read-only source volume, and rotate the Arc admin token if it was placed on a shared host.
## Troubleshooting [#troubleshooting]
| Symptom | Likely cause | Action |
| ------------------------------------------------ | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `no shards with TSM/WAL data found` | data is WAL-only and `--waldir` omitted, or wrong `--datadir` | pass `--waldir` (auto for 2.x); point `--datadir` at the InfluxDB root or data dir |
| A database/bucket is missing from output | WAL-only without `--waldir`, filtered out, or (2.x) a system bucket | add `--waldir`; check `--database-filter`; confirm it is not a system bucket |
| 2.x buckets show as 16-hex IDs | `influxd.bolt` missing or unreadable | provide `--bolt` or copy `influxd.bolt` next to `engine/` |
| `arc 401` / permanent error | token not admin-tier or wrong | use an admin token |
| `invalid measurement name …` before sending | source names violate Arc's rule (dots etc.) | dry-run to list them, then `--measurement-map`/`--measurement-map-file`, or `--on-invalid-measurement=skip\|map` |
| `arc 413` | `--chunk-bytes` too large for Arc's cap | keep `--chunk-bytes` under 500MB (the 450MB default is safe) |
| Repeated `arc 429` then backoff | Arc under load / too many workers | lower `--workers` and/or `--chunk-bytes` |
| Arc node OOM | `--workers` too high for Arc's RAM | lower `--workers`; see the [memory math](/arc/migration/tsm2arc/performance/) |
| `checkpoint was created with different settings` | resuming with a changed shaping flag | restore the original flags, or use a fresh `--checkpoint` |
| Run aborts on a corrupt TSM file | damaged source file | note the file from the error; use `--database-filter`/`--start`/`--end` to skip the affected range, then handle it separately |
| Resume re-sends everything | wrong or missing `--checkpoint` path | always point `--checkpoint` at the same durable file |
# Data Management (/arc/sdks/python/data-management)
How to manage data lifecycle, authentication, and administrative tasks using the Python SDK.
## Overview [#overview]
The SDK provides clients for managing Arc's data lifecycle features:
| Client | Purpose | Use Case |
| --------------------------- | ------------------ | ------------------------------- |
| `client.retention` | Retention policies | Delete old data on a schedule |
| `client.continuous_queries` | Continuous queries | Downsample and aggregate data |
| `client.delete` | Delete operations | Remove data matching conditions |
| `client.auth` | Authentication | Manage API tokens |
**Important:** In Arc OSS, retention policies and continuous queries do **not** run automatically. You must execute them manually or set up an external scheduler (cron, Airflow, etc.).
Automatic scheduling is planned for **Arc Enterprise** (2026).
See [Scheduling with External Tools](#scheduling-with-external-tools) for how to automate execution.
## Retention policies [#retention-policies]
Retention policies define rules for deleting data older than a specified age. Use them to:
* Control storage costs
* Comply with data retention regulations
* Remove stale data on a schedule
### Create a policy [#create-a-policy]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
policy = client.retention.create(
name="logs-30d",
database="default",
retention_days=30,
measurement="logs", # Optional: applies to specific measurement
buffer_days=7, # Optional: keep extra days as safety buffer
)
print(f"Created policy: {policy.name} (id={policy.id})")
```
### Parameters [#parameters]
| Parameter | Type | Required | Description |
| ---------------- | ----- | -------- | ---------------------------------------------- |
| `name` | `str` | Yes | Unique name for the policy |
| `database` | `str` | Yes | Target database |
| `retention_days` | `int` | Yes | Delete data older than N days |
| `measurement` | `str` | No | Limit to specific measurement (all if omitted) |
| `buffer_days` | `int` | No | Extra buffer days before deletion |
### List policies [#list-policies]
```python
policies = client.retention.list()
for p in policies:
status = "active" if p.is_active else "inactive"
measurement = p.measurement or "all measurements"
print(f"{p.name}: {p.retention_days} days on {measurement} ({status})")
```
### Execute a policy [#execute-a-policy]
Always use `dry_run=True` first to preview what will be deleted:
```python
# Preview deletion
result = client.retention.execute(policy.id, dry_run=True)
print(f"Would delete {result.deleted_count} rows")
# Execute for real (requires confirm=True for large deletes)
result = client.retention.execute(policy.id, dry_run=False, confirm=True)
print(f"Deleted {result.deleted_count} rows")
```
### Update a policy [#update-a-policy]
```python
client.retention.update(
policy.id,
retention_days=60, # Change retention period
is_active=False, # Disable the policy
)
```
### Delete a policy [#delete-a-policy]
```python
client.retention.delete(policy.id)
```
### Full example [#full-example]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Create policy for logs
policy = client.retention.create(
name="logs-retention",
database="default",
retention_days=30,
measurement="logs",
)
# Create policy for metrics (keep longer)
metrics_policy = client.retention.create(
name="metrics-retention",
database="default",
retention_days=90,
measurement="metrics",
)
# List all policies
for p in client.retention.list():
print(f" - {p.name}: {p.retention_days} days")
# Dry run to see what would be deleted
result = client.retention.execute(policy.id, dry_run=True)
print(f"\nDry run: would delete {result.deleted_count} rows")
```
See [Retention Policies](/arc/data-lifecycle/retention-policies/) for more details on how retention works in Arc.
## Continuous queries [#continuous-queries]
Continuous queries (CQs) define aggregation rules that transform data from one measurement to another. Use them to:
* Downsample high-resolution data to save storage
* Pre-compute aggregations for faster dashboard queries
* Create materialized views of your data
CQs define *what* to aggregate and *where* to store results. The `interval` parameter documents the intended frequency, but you must trigger execution manually or via an external scheduler. See [Scheduling with External Tools](#scheduling-with-external-tools).
### Create a continuous query [#create-a-continuous-query]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
cq = client.continuous_queries.create(
name="cpu-hourly-avg",
database="default",
source_measurement="cpu",
destination_measurement="cpu_1h",
query="""
SELECT
time_bucket('1 hour', time) as time,
host,
avg(usage_idle) as usage_idle,
avg(usage_system) as usage_system,
max(usage_user) as max_usage_user
FROM default.cpu
GROUP BY 1, 2
""",
interval="1h",
description="Hourly CPU averages per host",
)
print(f"Created CQ: {cq.name} (id={cq.id})")
```
### Parameters [#parameters-1]
| Parameter | Type | Required | Description |
| ------------------------- | ----- | -------- | -------------------------------------------- |
| `name` | `str` | Yes | Unique name for the CQ |
| `database` | `str` | Yes | Target database |
| `source_measurement` | `str` | Yes | Measurement to read from |
| `destination_measurement` | `str` | Yes | Measurement to write results to |
| `query` | `str` | Yes | SQL aggregation query |
| `interval` | `str` | Yes | Execution interval (e.g., `1h`, `15m`, `1d`) |
| `description` | `str` | No | Human-readable description |
### Query guidelines [#query-guidelines]
Your CQ query should:
* Use `time_bucket()` to aggregate time into intervals
* Include `time` as the first column in SELECT and GROUP BY
* Use aggregate functions (`avg`, `sum`, `count`, `min`, `max`, etc.)
* Reference the source measurement with `database.measurement` syntax
### List continuous queries [#list-continuous-queries]
```python
cqs = client.continuous_queries.list(database="default")
for cq in cqs:
status = "active" if cq.is_active else "inactive"
print(f"{cq.name}: {cq.source_measurement} → {cq.destination_measurement}")
print(f" Interval: {cq.interval} ({status})")
```
### Manual execution [#manual-execution]
Execute a CQ manually for a specific time range:
```python
# Dry run first
result = client.continuous_queries.execute(
cq.id,
start_time="2024-01-01T00:00:00Z",
end_time="2024-01-02T00:00:00Z",
dry_run=True,
)
print(f"Would process {result.records_read or 0} records")
print(f"Would write {result.records_written or 0} records")
# Execute for real
result = client.continuous_queries.execute(
cq.id,
start_time="2024-01-01T00:00:00Z",
end_time="2024-01-02T00:00:00Z",
dry_run=False,
)
```
### Update a CQ [#update-a-cq]
```python
client.continuous_queries.update(
cq.id,
interval="30m", # Change interval
is_active=False, # Pause the CQ
)
```
### Delete a CQ [#delete-a-cq]
```python
client.continuous_queries.delete(cq.id)
```
### Full example [#full-example-1]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Create hourly rollup
hourly_cq = client.continuous_queries.create(
name="cpu-hourly",
database="default",
source_measurement="cpu",
destination_measurement="cpu_1h",
query="""
SELECT
time_bucket('1 hour', time) as time,
host,
avg(usage_idle) as avg_idle,
min(usage_idle) as min_idle,
max(usage_idle) as max_idle
FROM default.cpu
GROUP BY 1, 2
""",
interval="1h",
)
# Create daily rollup from hourly data
daily_cq = client.continuous_queries.create(
name="cpu-daily",
database="default",
source_measurement="cpu_1h",
destination_measurement="cpu_1d",
query="""
SELECT
time_bucket('1 day', time) as time,
host,
avg(avg_idle) as avg_idle,
min(min_idle) as min_idle,
max(max_idle) as max_idle
FROM default.cpu_1h
GROUP BY 1, 2
""",
interval="1d",
)
print("Created CQ hierarchy: cpu → cpu_1h → cpu_1d")
```
See [Continuous Queries](/arc/data-lifecycle/continuous-queries/) for more details.
## Delete operations [#delete-operations]
Delete data matching specific conditions. Use this for:
* Removing erroneous data
* Deleting data for specific hosts or time ranges
* GDPR/compliance data removal
### Delete with conditions [#delete-with-conditions]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# ALWAYS dry_run first!
result = client.delete.delete(
database="default",
measurement="logs",
where="time < '2024-01-01' AND level = 'debug'",
dry_run=True,
)
print(f"Would delete {result.deleted_count} rows")
print(f"Affected files: {result.affected_files}")
# Execute deletion (requires confirm=True)
result = client.delete.delete(
database="default",
measurement="logs",
where="time < '2024-01-01' AND level = 'debug'",
dry_run=False,
confirm=True,
)
print(f"Deleted {result.deleted_count} rows")
```
### Parameters [#parameters-2]
| Parameter | Type | Required | Description |
| ------------- | ------ | -------- | -------------------------------------------- |
| `database` | `str` | Yes | Target database |
| `measurement` | `str` | Yes | Target measurement |
| `where` | `str` | Yes | SQL WHERE clause (without "WHERE" keyword) |
| `dry_run` | `bool` | No | Preview only, don't delete (default: `True`) |
| `confirm` | `bool` | No | Required for large deletes |
### Common delete patterns [#common-delete-patterns]
```python
# Delete old data
client.delete.delete(
database="default",
measurement="logs",
where="time < '2024-01-01'",
dry_run=False,
confirm=True,
)
# Delete data for a specific host
client.delete.delete(
database="default",
measurement="metrics",
where="host = 'decommissioned-server'",
dry_run=False,
confirm=True,
)
# Delete data in a time range
client.delete.delete(
database="default",
measurement="events",
where="time BETWEEN '2024-01-15' AND '2024-01-16'",
dry_run=False,
confirm=True,
)
```
See [Delete Operations](/arc/data-lifecycle/delete-operations/) for more details.
## Authentication [#authentication]
Manage API tokens for accessing Arc.
### Verify current token [#verify-current-token]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
result = client.auth.verify()
if result.valid:
print(f"Token name: {result.token_info.name}")
print(f"Permissions: {result.permissions}")
print(f"Created: {result.token_info.created_at}")
else:
print("Token is invalid or expired")
```
### Create a new token [#create-a-new-token]
```python
result = client.auth.create_token(
name="my-app-token",
description="Token for my application",
permissions=["read", "write"],
)
# IMPORTANT: Save this token - it's only shown once!
print(f"New token: {result.token}")
print(f"Token ID: {result.token_id}")
```
### Available permissions [#available-permissions]
| Permission | Description |
| ---------- | -------------------------------------- |
| `read` | Query data |
| `write` | Write/ingest data |
| `admin` | Manage tokens, retention policies, CQs |
### List tokens [#list-tokens]
```python
tokens = client.auth.list_tokens()
for t in tokens:
print(f"{t.name} (id={t.id})")
print(f" Created: {t.created_at}")
print(f" Last used: {t.last_used_at or 'never'}")
```
### Rotate a token [#rotate-a-token]
Generate a new token value while keeping the same token ID and permissions:
```python
result = client.auth.rotate_token(token_id=123)
# IMPORTANT: Save the new token - the old one is now invalid!
print(f"New token: {result.new_token}")
```
### Revoke a token [#revoke-a-token]
```python
client.auth.revoke_token(token_id=123)
print("Token revoked")
```
## Error handling [#error-handling]
All data management operations can raise specific exceptions:
```python
import os
from arc_client import ArcClient
from arc_client.exceptions import (
ARC_TOKEN = os.environ["ARC_TOKEN"]
ArcError,
ArcNotFoundError,
ArcValidationError,
ArcAuthenticationError,
)
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
try:
client.retention.delete(999) # Non-existent policy
except ArcNotFoundError:
print("Policy not found")
try:
client.retention.create(
name="", # Invalid name
database="default",
retention_days=-1, # Invalid days
)
except ArcValidationError as e:
print(f"Validation error: {e}")
try:
client.auth.create_token(name="test", permissions=["admin"])
except ArcAuthenticationError:
print("Current token doesn't have permission to create tokens")
```
## Async support [#async-support]
All data management operations have async equivalents:
```python
import os
import asyncio
from arc_client import AsyncArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
async def main():
async with AsyncArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Retention
policy = await client.retention.create(
name="async-policy",
database="default",
retention_days=30,
)
# CQs
cq = await client.continuous_queries.create(
name="async-cq",
database="default",
source_measurement="cpu",
destination_measurement="cpu_1h",
query="SELECT time_bucket('1 hour', time) as time, avg(usage) as usage FROM default.cpu GROUP BY 1",
interval="1h",
)
# Delete
result = await client.delete.delete(
database="default",
measurement="logs",
where="time < '2024-01-01'",
dry_run=True,
)
# Auth
verify = await client.auth.verify()
asyncio.run(main())
```
## Best practices [#best-practices]
### 1. Always dry run first [#1-always-dry-run-first]
For any destructive operation (delete, retention execution), always preview first:
```python
# ✅ Good: Preview before executing
result = client.delete.delete(..., dry_run=True)
print(f"Would delete {result.deleted_count} rows")
# Review the count, then execute
# ❌ Bad: Delete without preview
result = client.delete.delete(..., dry_run=False, confirm=True)
```
### 2. Use descriptive names [#2-use-descriptive-names]
```python
# ✅ Good: Descriptive names
client.retention.create(name="logs-30d-cleanup", ...)
client.continuous_queries.create(name="cpu-hourly-avg-by-host", ...)
# ❌ Bad: Generic names
client.retention.create(name="policy1", ...)
```
### 3. Document your CQ queries [#3-document-your-cq-queries]
```python
# ✅ Good: Include description
cq = client.continuous_queries.create(
name="cpu-hourly",
description="Hourly CPU averages per host. Used by main dashboard.",
...
)
```
### 4. Secure token management [#4-secure-token-management]
```python
# ✅ Good: Use environment variables
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
token = os.getenv("ARC_TOKEN")
client = ArcClient(host="localhost", token=token)
# ❌ Bad: Hardcode tokens
client = ArcClient(host="localhost", token="arc_abc123...")
```
## Scheduling with external tools [#scheduling-with-external-tools]
Since Arc OSS doesn't include a built-in scheduler, you need to trigger retention policies and continuous queries externally. Here are several approaches:
### Simple Python script with cron [#simple-python-script-with-cron]
Create a script that executes your policies and CQs:
```python title="arc_scheduler.py"
#!/usr/bin/env python3
"""Execute Arc retention policies and continuous queries."""
import logging
from datetime import datetime, timedelta
from arc_client import ArcClient
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def run_retention_policies(client: ArcClient):
"""Execute all active retention policies."""
policies = client.retention.list()
for policy in policies:
if not policy.is_active:
continue
logger.info(f"Executing retention policy: {policy.name}")
try:
result = client.retention.execute(policy.id, dry_run=False, confirm=True)
logger.info(f" Deleted {result.deleted_count} rows")
except Exception as e:
logger.error(f" Failed: {e}")
def run_continuous_queries(client: ArcClient):
"""Execute all active continuous queries for the last interval."""
cqs = client.continuous_queries.list()
for cq in cqs:
if not cq.is_active:
continue
# Calculate time range based on CQ interval
end_time = datetime.utcnow()
# Parse interval (e.g., "1h" -> 1 hour)
interval_hours = parse_interval_hours(cq.interval)
start_time = end_time - timedelta(hours=interval_hours)
logger.info(f"Executing CQ: {cq.name} ({start_time} to {end_time})")
try:
result = client.continuous_queries.execute(
cq.id,
start_time=start_time.isoformat() + "Z",
end_time=end_time.isoformat() + "Z",
dry_run=False,
)
logger.info(f" Processed {result.records_read or 0} records")
except Exception as e:
logger.error(f" Failed: {e}")
def parse_interval_hours(interval: str) -> int:
"""Parse interval string to hours (e.g., '1h' -> 1, '1d' -> 24)."""
if interval.endswith("h"):
return int(interval[:-1])
elif interval.endswith("d"):
return int(interval[:-1]) * 24
elif interval.endswith("m"):
return max(1, int(interval[:-1]) // 60)
return 1
def main():
import os
with ArcClient(
host=os.getenv("ARC_HOST", "localhost"),
port=int(os.getenv("ARC_PORT", "8000")),
token=os.getenv("ARC_TOKEN"),
) as client:
logger.info("Starting scheduled Arc maintenance...")
run_retention_policies(client)
run_continuous_queries(client)
logger.info("Scheduled maintenance complete")
if __name__ == "__main__":
main()
```
Schedule with cron (run hourly):
```bash
# Edit crontab
crontab -e
# Add entry to run every hour
0 * * * * cd /path/to/project && ARC_TOKEN=your-token python arc_scheduler.py >> /var/log/arc_scheduler.log 2>&1
```
### Using APScheduler [#using-apscheduler]
For more control, use [APScheduler](https://apscheduler.readthedocs.io/) to run different tasks at different intervals:
```python title="arc_scheduler_advanced.py"
#!/usr/bin/env python3
"""Advanced Arc scheduler with APScheduler."""
import os
import logging
from apscheduler.schedulers.blocking import BlockingScheduler
from arc_client import ArcClient
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def get_client():
return ArcClient(
host=os.getenv("ARC_HOST", "localhost"),
token=os.getenv("ARC_TOKEN"),
)
def execute_hourly_cqs():
"""Run CQs with 1h interval."""
logger.info("Running hourly CQs...")
with get_client() as client:
for cq in client.continuous_queries.list():
if cq.is_active and cq.interval == "1h":
try:
client.continuous_queries.execute(cq.id, dry_run=False)
logger.info(f" Executed: {cq.name}")
except Exception as e:
logger.error(f" Failed {cq.name}: {e}")
def execute_daily_retention():
"""Run retention policies once per day."""
logger.info("Running daily retention...")
with get_client() as client:
for policy in client.retention.list():
if policy.is_active:
try:
result = client.retention.execute(
policy.id, dry_run=False, confirm=True
)
logger.info(f" {policy.name}: deleted {result.deleted_count} rows")
except Exception as e:
logger.error(f" Failed {policy.name}: {e}")
if __name__ == "__main__":
scheduler = BlockingScheduler()
# Run hourly CQs every hour at :05
scheduler.add_job(execute_hourly_cqs, "cron", minute=5)
# Run retention policies daily at 3:00 AM
scheduler.add_job(execute_daily_retention, "cron", hour=3, minute=0)
logger.info("Arc scheduler started. Press Ctrl+C to exit.")
scheduler.start()
```
Install and run:
```bash
pip install apscheduler
python arc_scheduler_advanced.py
```
### Docker deployment [#docker-deployment]
Run the scheduler as a Docker container alongside Arc:
```yaml title="docker-compose.yml"
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
ports:
- "8000:8000"
volumes:
- arc-data:/app/data
arc-scheduler:
build:
context: .
dockerfile: Dockerfile.scheduler
environment:
- ARC_HOST=arc
- ARC_PORT=8000
- ARC_TOKEN=${ARC_TOKEN}
depends_on:
- arc
restart: unless-stopped
volumes:
arc-data:
```
```dockerfile title="Dockerfile.scheduler"
FROM python:3.11-slim
WORKDIR /app
RUN pip install arc-tsdb-client apscheduler
COPY arc_scheduler_advanced.py .
CMD ["python", "arc_scheduler_advanced.py"]
```
### Kubernetes CronJob [#kubernetes-cronjob]
For Kubernetes deployments, use a CronJob:
```yaml title="arc-scheduler-cronjob.yaml"
apiVersion: batch/v1
kind: CronJob
metadata:
name: arc-maintenance
spec:
schedule: "0 * * * *" # Every hour
jobTemplate:
spec:
template:
spec:
containers:
- name: arc-scheduler
image: python:3.11-slim
command:
- /bin/sh
- -c
- |
pip install -q arc-tsdb-client
python -c "
from arc_client import ArcClient
import os
with ArcClient(
host=os.environ['ARC_HOST'],
token=os.environ['ARC_TOKEN']
) as client:
# Execute retention policies
for p in client.retention.list():
if p.is_active:
client.retention.execute(p.id, dry_run=False, confirm=True)
# Execute continuous queries
for cq in client.continuous_queries.list():
if cq.is_active:
client.continuous_queries.execute(cq.id, dry_run=False)
"
env:
- name: ARC_HOST
value: "arc-service"
- name: ARC_TOKEN
valueFrom:
secretKeyRef:
name: arc-secrets
key: token
restartPolicy: OnFailure
```
### Apache Airflow [#apache-airflow]
For complex workflows, use Airflow:
```python title="dags/arc_maintenance.py"
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
default_args = {
"owner": "data-team",
"retries": 3,
"retry_delay": timedelta(minutes=5),
}
def run_retention():
from arc_client import ArcClient
import os
with ArcClient(host=os.getenv("ARC_HOST"), token=os.getenv("ARC_TOKEN")) as client:
for policy in client.retention.list():
if policy.is_active:
client.retention.execute(policy.id, dry_run=False, confirm=True)
def run_hourly_cqs():
from arc_client import ArcClient
import os
with ArcClient(host=os.getenv("ARC_HOST"), token=os.getenv("ARC_TOKEN")) as client:
for cq in client.continuous_queries.list():
if cq.is_active and cq.interval == "1h":
client.continuous_queries.execute(cq.id, dry_run=False)
with DAG(
"arc_maintenance",
default_args=default_args,
schedule_interval="@hourly",
start_date=datetime(2024, 1, 1),
catchup=False,
) as dag:
retention_task = PythonOperator(
task_id="run_retention",
python_callable=run_retention,
)
cq_task = PythonOperator(
task_id="run_hourly_cqs",
python_callable=run_hourly_cqs,
)
retention_task >> cq_task
```
## Next steps [#next-steps]
* **[Data Ingestion](/arc/sdks/python/ingestion/)** - Write data to Arc
* **[Querying](/arc/sdks/python/querying/)** - Query data with DataFrames
* **[Retention Policies](/arc/data-lifecycle/retention-policies/)** - Deep dive on retention
* **[Continuous Queries](/arc/data-lifecycle/continuous-queries/)** - Deep dive on CQs
# Python SDK (/arc/sdks/python)
Official Python SDK for the Arc time-series database.
[](https://badge.fury.io/py/arc-tsdb-client)
[](https://www.python.org/downloads/)
## What is it? [#what-is-it]
The `arc-tsdb-client` package is the official Python client for [Arc](https://github.com/basekick-labs/arc), an open, SQL-native time-series database. It provides a high-level, Pythonic interface for:
* **Writing data** at scale
* **Querying with SQL** and getting results as DataFrames
* **Managing data lifecycle** (retention, aggregation, deletion)
* **Handling authentication** (tokens, permissions)
## Why use the SDK? [#why-use-the-sdk]
While you can interact with Arc's REST API directly, the SDK provides significant advantages:
| Feature | Raw API | Python SDK |
| --------------------- | -------------------- | ---------------------------- |
| Connection management | Manual | Automatic (context managers) |
| Data serialization | Manual MessagePack | Automatic |
| DataFrame support | Convert manually | Native pandas/polars/arrow |
| Buffered writes | Implement yourself | Built-in with auto-batching |
| Error handling | Parse HTTP responses | Typed exceptions |
| Async support | Manual httpx/aiohttp | Built-in `AsyncArcClient` |
| Compression | Configure headers | Automatic gzip |
## Quick example [#quick-example]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Write metrics
client.write.write_columnar(
measurement="cpu",
columns={
"time": [1704067200000000, 1704067260000000],
"host": ["server01", "server01"],
"usage_idle": [95.2, 94.8],
},
)
# Query to pandas
df = client.query.query_pandas(
"SELECT * FROM default.cpu WHERE host = 'server01'"
)
print(df)
```
## Architecture [#architecture]
The SDK is organized into specialized clients for different operations:
```text
ArcClient
├── .write # Data ingestion (WriteClient)
│ ├── write_columnar()
│ ├── write_dataframe()
│ ├── write_line_protocol()
│ └── buffered()
│
├── .query # Data querying (QueryClient)
│ ├── query()
│ ├── query_pandas()
│ ├── query_polars()
│ ├── query_arrow()
│ └── estimate()
│
├── .retention # Retention policies (RetentionClient)
├── .continuous_queries # CQs (ContinuousQueryClient)
├── .delete # Delete operations (DeleteClient)
├── .auth # Authentication (AuthClient)
└── .health() # Health check
```
## Documentation [#documentation]
## Source code [#source-code]
* **Repository**: [github.com/basekick-labs/arc-client-python](https://github.com/basekick-labs/arc-client-python)
* **PyPI**: [pypi.org/project/arc-tsdb-client](https://pypi.org/project/arc-tsdb-client/)
# Data Ingestion (/arc/sdks/python/ingestion)
How to write data to Arc using the Python SDK.
## Overview [#overview]
The SDK provides multiple ways to ingest data, each optimized for different use cases:
| Method | Best For | Performance | Format |
| ----------------------- | ------------------------- | ------------ | -------------------- |
| `write_columnar()` | High-throughput ingestion | Fastest | MessagePack columnar |
| `write_dataframe()` | pandas/polars workflows | Fast | MessagePack columnar |
| `buffered()` | Streaming data | Auto-batched | MessagePack columnar |
| `write_line_protocol()` | InfluxDB compatibility | Moderate | Line protocol text |
All write methods are available on `client.write`.
## Columnar format (recommended) [#columnar-format-recommended]
The fastest way to write data. Data is organized by columns (like a DataFrame) rather than rows, which enables efficient compression and fast serialization.
### Basic usage [#basic-usage]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
client.write.write_columnar(
measurement="cpu",
columns={
"time": [1704067200000000, 1704067260000000, 1704067320000000],
"host": ["server01", "server01", "server01"],
"region": ["us-east", "us-east", "us-east"],
"usage_idle": [95.2, 94.8, 93.1],
"usage_system": [2.1, 2.5, 3.2],
"usage_user": [2.7, 2.7, 3.7],
},
)
```
### Parameters [#parameters]
| Parameter | Type | Required | Description |
| ------------- | ------ | -------- | -------------------------------------------- |
| `measurement` | `str` | Yes | Name of the measurement (like a table) |
| `columns` | `dict` | Yes | Column name → list of values |
| `database` | `str` | No | Target database (default: client's database) |
### Column types [#column-types]
The SDK automatically handles type conversion:
| Python Type | Arc Type | Example |
| ----------- | ------------------------ | --------------------- |
| `int` | Integer | `[1, 2, 3]` |
| `float` | Float | `[1.5, 2.7, 3.9]` |
| `str` | String (tag or field) | `["a", "b", "c"]` |
| `bool` | Boolean | `[True, False, True]` |
| `datetime` | Timestamp (microseconds) | `[datetime.now()]` |
### Timestamps [#timestamps]
The `time` column must contain a **numeric Unix epoch**. Arc auto-detects the
unit by magnitude — seconds, milliseconds, microseconds, or nanoseconds — and
normalizes everything to microseconds internally. Microseconds is the
recommended, unambiguous form:
```python
import time
from datetime import datetime
# From current time
timestamp_us = int(time.time() * 1_000_000)
# From datetime
timestamp_us = int(datetime.now().timestamp() * 1_000_000)
# Multiple timestamps
timestamps = [
1704067200000000, # 2024-01-01 00:00:00 UTC
1704067260000000, # 2024-01-01 00:01:00 UTC
1704067320000000, # 2024-01-01 00:02:00 UTC
]
```
**`time` must be numeric, and must never be null.** Arc stores `time` as a
`TIMESTAMP WITH TIME ZONE` column (UTC). To keep this column type consistent
across every file in a partition — which compaction relies on — the ingest path
**rejects** a `time` column sent as a **string** (e.g. an ISO-8601 datetime
string like `"2024-01-01T00:00:00Z"`) or containing **null** values. Such a
write fails with a clear error rather than being silently accepted:
* ❌ `"time": ["2024-01-01T00:00:00Z"]` — string timestamps are rejected. Convert
to a numeric epoch first: `int(datetime.fromisoformat("2024-01-01T00:00:00+00:00").timestamp() * 1_000_000)`.
* ❌ `"time": [1704067200000000, None]` — null timestamps are rejected (a null
`time` would otherwise be silently routed to the `1970-01-01` partition).
* ✅ `"time": [1704067200000000, 1704067260000000]` — numeric epoch (any unit).
If you previously sent string timestamps and saw them ingest "successfully" on
Arc 26.05.1, note that those writes produced files whose `time` column type
disagreed with normally-ingested files, which prevented those partitions from
compacting. Sending a numeric epoch avoids this entirely.
### Tags vs fields [#tags-vs-fields]
In Arc:
* **Tags** are indexed string columns used for filtering (e.g., `host`, `region`, `sensor_id`)
* **Fields** are the actual metric values (e.g., `temperature`, `cpu_usage`, `count`)
For `write_columnar()`, all columns are sent directly to Arc. Tag/field distinction is handled by the Arc server based on its schema detection.
For `write_dataframe()`, you can explicitly specify which columns are tags using the `tag_columns` parameter.
## DataFrame ingestion [#dataframe-ingestion]
Write directly from pandas or polars DataFrames. The SDK converts DataFrames to columnar format automatically.
### pandas example [#pandas-example]
```python
import os
import pandas as pd
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Create a DataFrame
df = pd.DataFrame({
"time": pd.date_range("2024-01-01", periods=100, freq="1min"),
"host": ["server-01"] * 50 + ["server-02"] * 50,
"region": ["us-east"] * 100,
"cpu_usage": [50 + i * 0.1 for i in range(100)],
"memory_mb": [1024 + i for i in range(100)],
})
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
client.write.write_dataframe(
df,
measurement="server_metrics",
time_column="time", # Column containing timestamps
tag_columns=["host", "region"], # Columns to treat as tags
)
print(f"Wrote {len(df)} rows")
```
### Parameters [#parameters-1]
| Parameter | Type | Required | Description |
| ------------- | ----------- | -------- | -------------------------------- |
| `df` | DataFrame | Yes | pandas or polars DataFrame |
| `measurement` | `str` | Yes | Target measurement name |
| `time_column` | `str` | Yes | Name of the timestamp column |
| `tag_columns` | `list[str]` | No | Columns to treat as indexed tags |
| `database` | `str` | No | Target database |
### Polars example [#polars-example]
```python
import os
import polars as pl
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
df = pl.DataFrame({
"time": pl.datetime_range(
datetime(2024, 1, 1),
datetime(2024, 1, 1, 1),
interval="1m",
eager=True
),
"sensor_id": ["sensor-001"] * 61,
"temperature": [20.0 + i * 0.1 for i in range(61)],
})
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
client.write.write_dataframe(
df,
measurement="temperatures",
time_column="time",
tag_columns=["sensor_id"],
)
```
## Buffered writes [#buffered-writes]
For streaming or high-throughput scenarios, use buffered writes. The buffer automatically batches records and flushes them efficiently.
### Basic usage [#basic-usage-1]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
with client.write.buffered(batch_size=5000, flush_interval=2.0) as buffer:
for i in range(50000):
buffer.write(
measurement="events",
tags={"source": "sensor-001", "type": "temperature"},
fields={"value": 22.5 + i * 0.01},
timestamp=1704067200000000 + (i * 1000),
)
# Auto-flushes on exit
```
### Parameters [#parameters-2]
| Parameter | Type | Default | Description |
| ---------------- | ------- | ------- | ---------------------------------------------- |
| `batch_size` | `int` | 5000 | Flush after N records |
| `flush_interval` | `float` | 5.0 | Flush after N seconds (even if batch not full) |
### How it works [#how-it-works]
1. Records are queued in memory
2. When `batch_size` is reached OR `flush_interval` expires, the buffer flushes
3. On context manager exit, any remaining records are flushed
4. Uses columnar format internally for best performance
### When to use buffered writes [#when-to-use-buffered-writes]
✅ **Use buffered writes when:**
* Processing streaming data (sensors, logs, events)
* Ingesting data in a loop
* You don't know the batch size ahead of time
❌ **Don't use buffered writes when:**
* You already have data in columnar format or DataFrame
* You're writing a single batch (use `write_columnar()` directly)
### Async buffered writes [#async-buffered-writes]
```python
import os
import asyncio
from arc_client import AsyncArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
async def ingest_stream():
async with AsyncArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
async with client.write.buffered(batch_size=5000) as buffer:
async for event in event_stream():
await buffer.write(
measurement="events",
tags={"source": event.source},
fields={"value": event.value},
timestamp=event.timestamp,
)
asyncio.run(ingest_stream())
```
## Line Protocol [#line-protocol]
For compatibility with InfluxDB tooling (Telegraf, etc.), use line protocol format.
### Basic usage [#basic-usage-2]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Single line
client.write.write_line_protocol(
"cpu,host=server01,region=us-east usage_idle=95.2 1704067200000000000"
)
# Multiple lines
lines = [
"cpu,host=server01 usage_idle=95.2,usage_system=2.1",
"cpu,host=server02 usage_idle=87.5,usage_system=4.3",
"mem,host=server01 used_percent=45.2",
]
client.write.write_line_protocol(lines)
```
### Line Protocol format [#line-protocol-format]
```text
,=,... =,... [timestamp]
```
Example breakdown:
```text
cpu,host=server01,region=us-east usage_idle=95.2,usage_system=2.1 1704067200000000000
│ │ │ │
│ │ │ └── timestamp (nanoseconds)
│ │ └── fields (space-separated from tags)
│ └── tags (comma-separated)
└── measurement name
```
### When to use Line Protocol [#when-to-use-line-protocol]
✅ **Use line protocol when:**
* Integrating with Telegraf or other InfluxDB tools
* Migrating from InfluxDB
* You already have data in line protocol format
❌ **Don't use line protocol when:**
* Building new applications (use columnar format)
* Performance is critical (columnar is considerably faster)
## Async ingestion [#async-ingestion]
All write methods have async equivalents:
```python
import os
import asyncio
from arc_client import AsyncArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
async def main():
async with AsyncArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Columnar write
await client.write.write_columnar(
measurement="cpu",
columns={
"time": [1704067200000000],
"host": ["server01"],
"usage": [45.2],
},
)
# DataFrame write
await client.write.write_dataframe(
df, measurement="metrics", time_column="time"
)
# Line protocol
await client.write.write_line_protocol("cpu,host=server01 usage=45.2")
asyncio.run(main())
```
## Error handling [#error-handling]
```python
import os
from arc_client import ArcClient
from arc_client.exceptions import (
ARC_TOKEN = os.environ["ARC_TOKEN"]
ArcIngestionError,
ArcValidationError,
ArcConnectionError,
)
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
try:
client.write.write_columnar(
measurement="cpu",
columns={"time": [1], "value": [1.0]},
)
except ArcValidationError as e:
print(f"Invalid data: {e}")
except ArcIngestionError as e:
print(f"Write failed: {e}")
except ArcConnectionError as e:
print(f"Connection error: {e}")
```
## Best practices [#best-practices]
### 1. Batch your data [#1-batch-your-data]
Send multiple rows per request rather than one at a time:
```python
# ✅ Good: Batch write
client.write.write_columnar(
measurement="cpu",
columns={
"time": [t1, t2, t3, ...], # 1000+ values
"host": [h1, h2, h3, ...],
"value": [v1, v2, v3, ...],
},
)
# ❌ Bad: Individual writes
for record in records:
client.write.write_columnar(
measurement="cpu",
columns={
"time": [record.time],
"host": [record.host],
"value": [record.value],
},
)
```
### 2. Use appropriate batch sizes [#2-use-appropriate-batch-sizes]
* **Small batches** (100-1000): Lower latency, more HTTP overhead
* **Medium batches** (1000-10000): Good balance for most use cases
* **Large batches** (10000+): Best throughput, higher memory usage
### 3. Handle backpressure [#3-handle-backpressure]
For high-throughput scenarios, implement backpressure handling:
```python
import time
def write_with_backoff(client, data, max_retries=3):
for attempt in range(max_retries):
try:
client.write.write_columnar(**data)
return
except ArcIngestionError as e:
if attempt == max_retries - 1:
raise
time.sleep(2 ** attempt) # Exponential backoff
```
## Next steps [#next-steps]
* **[Querying](/arc/sdks/python/querying/)** - Query data and work with DataFrames
* **[Data Management](/arc/sdks/python/data-management/)** - Retention, CQs, and deletion
* **[API Reference](/arc/api-reference/overview/)** - Raw REST API documentation
# Install the Python SDK (/arc/sdks/python/installation)
How to install the Arc Python SDK and its optional dependencies.
## Requirements [#requirements]
* **Python 3.9+** (3.10+ recommended for best performance)
* **Arc server** running and accessible
## Basic installation [#basic-installation]
Install the core SDK with pip:
```bash
pip install arc-tsdb-client
```
This installs the minimal dependencies needed to connect to Arc and write/query data using JSON responses.
**Core dependencies:**
* `httpx` - HTTP client with connection pooling
* `msgpack` - Binary serialization for high-performance writes
## Optional dependencies [#optional-dependencies]
The SDK supports optional extras for DataFrame integration:
### pandas support [#pandas-support]
For `query_pandas()` and `write_dataframe()` with pandas:
```bash
pip install arc-tsdb-client[pandas]
```
**What it adds:**
* `pandas` - DataFrame library
* `pyarrow` - Required for efficient Arrow-to-pandas conversion
**Use when:** You're working in Jupyter notebooks, data science workflows, or need pandas DataFrames.
### Polars support [#polars-support]
For `query_polars()` with the high-performance Polars library:
```bash
pip install arc-tsdb-client[polars]
```
**What it adds:**
* `polars` - Fast DataFrame library written in Rust
* `pyarrow` - Required for Arrow IPC parsing
**Use when:** You need maximum query performance, are processing large datasets, or prefer Polars' API.
### All dependencies [#all-dependencies]
Install everything for full functionality:
```bash
pip install arc-tsdb-client[all]
```
**What it adds:**
* `pandas`
* `polars`
* `pyarrow`
**Use when:** You want access to all features without worrying about which extras you need.
## Using uv (recommended) [#using-uv-recommended]
[uv](https://github.com/astral-sh/uv) is a fast Python package manager. Install the SDK with:
```bash
# Core only
uv add arc-tsdb-client
# With pandas
uv add arc-tsdb-client --extra pandas
# With all extras
uv add arc-tsdb-client --extra all
```
## Verifying installation [#verifying-installation]
Test that the SDK is installed correctly:
```python
import os
from arc_client import ArcClient, AsyncArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Check version
import arc_client
print(f"arc-tsdb-client version: {arc_client.__version__}")
# Test connection
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
health = client.health()
print(f"Server status: {health.status}")
```
## Configuration [#configuration]
### Basic client setup [#basic-client-setup]
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
client = ArcClient(
host="localhost", # Arc server hostname
port=8000, # Arc server port (default: 8000)
token=os.environ["ARC_TOKEN"], # API token (required)
database="default", # Default database for queries
timeout=30.0, # Request timeout in seconds
compression=True, # Enable gzip compression for writes
ssl=False, # Use HTTPS instead of HTTP
verify_ssl=True, # Verify SSL certificates
)
```
### Environment variables [#environment-variables]
You can also configure the client using environment variables:
```bash
export ARC_HOST="localhost"
export ARC_PORT="8000"
export ARC_TOKEN="your-token"
export ARC_DATABASE="default"
```
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
from arc_client import ArcClient
client = ArcClient(
host=os.getenv("ARC_HOST", "localhost"),
port=int(os.getenv("ARC_PORT", "8000")),
token=os.getenv("ARC_TOKEN"),
database=os.getenv("ARC_DATABASE", "default"),
)
```
### Context manager (recommended) [#context-manager-recommended]
Always use the client as a context manager to ensure proper connection cleanup:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
# Sync client
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Use client...
pass
# Connection automatically closed
# Async client
async with AsyncArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Use client...
pass
# Connection automatically closed
```
### Manual connection management [#manual-connection-management]
If you can't use a context manager:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
client = ArcClient(host="localhost", token=os.environ["ARC_TOKEN"])
try:
# Use client...
pass
finally:
client.close()
```
## Troubleshooting [#troubleshooting]
### ImportError: No module named 'pandas' [#importerror-no-module-named-pandas]
You need to install the pandas extra:
```bash
pip install arc-tsdb-client[pandas]
```
### ImportError: No module named 'polars' [#importerror-no-module-named-polars]
You need to install the polars extra:
```bash
pip install arc-tsdb-client[polars]
```
### Connection refused [#connection-refused]
Make sure Arc is running and accessible:
```bash
curl http://localhost:8000/health
```
### Authentication failed [#authentication-failed]
Verify your token is correct:
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
result = client.auth.verify()
if result.valid:
print(f"Token is valid: {result.token_info.name}")
else:
print("Token is invalid")
```
## Next steps [#next-steps]
* **[Data Ingestion](/arc/sdks/python/ingestion/)** - Learn how to write data to Arc
* **[Querying](/arc/sdks/python/querying/)** - Query data and work with DataFrames
* **[Data Management](/arc/sdks/python/data-management/)** - Manage retention, CQs, and more
# Querying with the Python SDK (/arc/sdks/python/querying)
How to query data from Arc using the Python SDK.
## Overview [#overview]
The SDK provides multiple query methods, each returning data in a different format:
| Method | Returns | Best For | Performance |
| ---------------- | -------------------- | --------------------------- | ----------- |
| `query()` | `QueryResult` object | Simple queries, inspection | Good |
| `query_pandas()` | pandas DataFrame | Data science, notebooks | Good |
| `query_polars()` | polars DataFrame | Large datasets, performance | Better |
| `query_arrow()` | PyArrow Table | Zero-copy, interop | Best |
All query methods are available on `client.query`.
## SQL syntax [#sql-syntax]
Arc uses SQL with the table syntax `database.measurement`:
```sql
SELECT * FROM default.cpu WHERE host = 'server01'
```
Arc runs a full analytical SQL engine, so you have access to:
* Window functions
* CTEs (Common Table Expressions)
* `time_bucket()` for time-series aggregation
* JSON functions
* And more
## Basic query (JSON) [#basic-query-json]
The simplest way to query data. Returns a `QueryResult` object with columns and data.
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
result = client.query.query(
"SELECT * FROM default.cpu WHERE time > now() - INTERVAL '1 hour' LIMIT 100"
)
print(f"Columns: {result.columns}")
print(f"Row count: {result.row_count}")
for row in result.data:
print(row)
```
### QueryResult object [#queryresult-object]
| Property | Type | Description |
| ----------- | ------------ | ----------------------- |
| `columns` | `list[str]` | Column names |
| `data` | `list[list]` | Rows as nested lists |
| `row_count` | `int` | Number of rows returned |
### When to use [#when-to-use]
✅ **Use `query()` when:**
* You need to inspect results quickly
* Working with small result sets
* Don't need DataFrame functionality
❌ **Don't use when:**
* Processing large datasets (use Arrow)
* Need DataFrame operations (use pandas/polars)
## pandas DataFrame [#pandas-dataframe]
Returns query results as a pandas DataFrame. Requires `pip install arc-tsdb-client[pandas]`.
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
df = client.query.query_pandas(
"SELECT * FROM default.cpu WHERE host = 'server01' LIMIT 1000"
)
print(df.head())
print(df.dtypes)
# Use pandas operations
avg_by_host = df.groupby("host")["usage_idle"].mean()
print(avg_by_host)
```
### How it works [#how-it-works]
1. Query is sent to Arc
2. Results are returned as Arrow IPC stream
3. Arrow data is converted to pandas DataFrame (zero-copy where possible)
### When to use [#when-to-use-1]
✅ **Use `query_pandas()` when:**
* Working in Jupyter notebooks
* Need pandas-specific operations
* Integrating with pandas-based tools (matplotlib, seaborn, scikit-learn)
❌ **Don't use when:**
* Processing very large datasets (polars is faster)
* Memory is constrained (polars uses less memory)
## Polars DataFrame [#polars-dataframe]
Returns query results as a polars DataFrame. Requires `pip install arc-tsdb-client[polars]`.
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
df = client.query.query_polars(
"SELECT * FROM default.cpu LIMIT 100000"
)
print(df.head())
# Use polars operations (lazy evaluation, parallel execution)
result = (
df.lazy()
.filter(pl.col("usage_idle") > 90)
.group_by("host")
.agg(pl.col("usage_idle").mean().alias("avg_idle"))
.collect()
)
print(result)
```
### Why Polars? [#why-polars]
Polars is a DataFrame library written in Rust that offers:
* **Faster operations**: Especially for large datasets
* **Lower memory usage**: Efficient memory layout
* **Lazy evaluation**: Optimize query plans before execution
* **Parallel execution**: Uses all CPU cores automatically
### When to use [#when-to-use-2]
✅ **Use `query_polars()` when:**
* Processing large datasets (100K+ rows)
* Performance is critical
* You prefer polars' API
* Running in production pipelines
❌ **Don't use when:**
* Need pandas compatibility for downstream tools
* Working in environments that only support pandas
## PyArrow table (zero-copy) [#pyarrow-table-zero-copy]
Returns query results as a PyArrow Table. This is the most efficient option for large datasets.
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
table = client.query.query_arrow(
"SELECT * FROM default.cpu LIMIT 1000000"
)
print(f"Rows: {table.num_rows}")
print(f"Columns: {table.num_columns}")
print(f"Schema: {table.schema}")
# Convert to pandas (zero-copy where possible)
df = table.to_pandas()
# Convert to polars
import polars as pl
df = pl.from_arrow(table)
# Save to Parquet
import pyarrow.parquet as pq
pq.write_table(table, "output.parquet")
```
### Why Arrow? [#why-arrow]
Apache Arrow is a columnar memory format that enables:
* **Zero-copy reads**: Data stays in the same memory layout
* **Interoperability**: Share data between pandas, polars, DuckDB, Spark
* **Efficient serialization**: Arrow IPC format is compact and fast
### When to use [#when-to-use-3]
✅ **Use `query_arrow()` when:**
* Processing very large datasets
* Need to pass data to multiple tools
* Saving results to Parquet files
* Maximum performance is required
## Query estimation [#query-estimation]
Preview the cost of a query before executing it:
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
estimate = client.query.estimate(
"SELECT * FROM default.cpu WHERE time > now() - INTERVAL '30 days'"
)
print(f"Estimated rows: {estimate.estimated_rows}")
print(f"Warning level: {estimate.warning_level}") # none, low, medium, high
if estimate.warning_level == "high":
print("Consider adding filters or LIMIT clause")
```
### Estimate result [#estimate-result]
| Property | Type | Description |
| ---------------- | ----- | ------------------------------- |
| `estimated_rows` | `int` | Approximate row count |
| `warning_level` | `str` | `none`, `low`, `medium`, `high` |
## List measurements [#list-measurements]
Discover what measurements exist in a database:
```python
import os
from arc_client import ArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
measurements = client.query.list_measurements(database="default")
for m in measurements:
print(f"{m.measurement}:")
print(f" Files: {m.file_count}")
print(f" Size: {m.total_size_mb:.1f} MB")
```
## Common query patterns [#common-query-patterns]
### Time-series aggregation [#time-series-aggregation]
Use `time_bucket()` to aggregate data into time intervals:
```python
df = client.query.query_pandas("""
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
host,
AVG(usage_idle) as avg_idle,
MAX(usage_system) as max_system,
COUNT(*) as samples
FROM default.cpu
WHERE time > now() - INTERVAL '1 hour'
GROUP BY bucket, host
ORDER BY bucket DESC
""")
```
### Latest value per host [#latest-value-per-host]
```python
df = client.query.query_pandas("""
SELECT DISTINCT ON (host)
time, host, usage_idle, usage_system
FROM default.cpu
ORDER BY host, time DESC
""")
```
### Percentiles [#percentiles]
```python
df = client.query.query_pandas("""
SELECT
host,
percentile_cont(0.50) WITHIN GROUP (ORDER BY usage_idle) as p50,
percentile_cont(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
percentile_cont(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
FROM default.cpu
WHERE time > now() - INTERVAL '24 hours'
GROUP BY host
""")
```
### Join measurements [#join-measurements]
```python
df = client.query.query_pandas("""
SELECT
c.time,
c.host,
c.usage_idle as cpu_idle,
m.used_percent as mem_used
FROM default.cpu c
JOIN default.mem m
ON c.time = m.time AND c.host = m.host
WHERE c.time > now() - INTERVAL '10 minutes'
ORDER BY c.time DESC
""")
```
## Async queries [#async-queries]
All query methods have async equivalents:
```python
import os
import asyncio
from arc_client import AsyncArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
async def main():
async with AsyncArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# JSON result
result = await client.query.query("SELECT * FROM default.cpu LIMIT 10")
# pandas DataFrame
df = await client.query.query_pandas("SELECT * FROM default.cpu LIMIT 1000")
# Polars DataFrame
pl_df = await client.query.query_polars("SELECT * FROM default.cpu LIMIT 1000")
# Arrow Table
table = await client.query.query_arrow("SELECT * FROM default.cpu LIMIT 10000")
asyncio.run(main())
```
### Concurrent queries [#concurrent-queries]
Run multiple queries in parallel:
```python
import os
import asyncio
from arc_client import AsyncArcClient
ARC_TOKEN = os.environ["ARC_TOKEN"]
async def main():
async with AsyncArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
# Run queries concurrently
cpu_task = client.query.query_pandas("SELECT * FROM default.cpu LIMIT 1000")
mem_task = client.query.query_pandas("SELECT * FROM default.mem LIMIT 1000")
disk_task = client.query.query_pandas("SELECT * FROM default.disk LIMIT 1000")
cpu_df, mem_df, disk_df = await asyncio.gather(cpu_task, mem_task, disk_task)
print(f"CPU: {len(cpu_df)} rows")
print(f"Memory: {len(mem_df)} rows")
print(f"Disk: {len(disk_df)} rows")
asyncio.run(main())
```
## Error handling [#error-handling]
```python
import os
from arc_client import ArcClient
from arc_client.exceptions import (
ARC_TOKEN = os.environ["ARC_TOKEN"]
ArcQueryError,
ArcConnectionError,
ArcAuthenticationError,
)
with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
try:
df = client.query.query_pandas("SELECT * FROM nonexistent.table")
except ArcQueryError as e:
print(f"Query failed: {e}") # Invalid SQL or table not found
except ArcAuthenticationError as e:
print(f"Auth failed: {e}") # Invalid token
except ArcConnectionError as e:
print(f"Connection error: {e}") # Server unreachable
```
## Performance tips [#performance-tips]
### 1. Filter early [#1-filter-early]
Push filters to Arc rather than filtering in Python:
```python
# ✅ Good: Filter in SQL
df = client.query.query_pandas("""
SELECT * FROM default.cpu
WHERE time > now() - INTERVAL '1 hour'
AND host = 'server01'
""")
# ❌ Bad: Fetch all, filter in Python
df = client.query.query_pandas("SELECT * FROM default.cpu")
df = df[df["host"] == "server01"]
```
### 2. Select only needed columns [#2-select-only-needed-columns]
```python
# ✅ Good: Select specific columns
df = client.query.query_pandas("""
SELECT time, host, usage_idle FROM default.cpu
""")
# ❌ Bad: Select all columns
df = client.query.query_pandas("SELECT * FROM default.cpu")
```
### 3. Use LIMIT for exploration [#3-use-limit-for-exploration]
```python
# ✅ Good: Limit during exploration
df = client.query.query_pandas("SELECT * FROM default.cpu LIMIT 100")
```
### 4. Use Arrow for large results [#4-use-arrow-for-large-results]
```python
# For 100K+ rows, Arrow is significantly faster
table = client.query.query_arrow("SELECT * FROM default.cpu")
df = table.to_pandas() # Zero-copy conversion
```
## Next steps [#next-steps]
* **[Data Management](/arc/sdks/python/data-management/)** - Retention, CQs, and deletion
* **[Data Ingestion](/arc/sdks/python/ingestion/)** - Write data to Arc
* **[API Reference](/arc/api-reference/overview/)** - Raw REST API documentation
# Data Management (/arc-enterprise/sdks/python/data-management)
How to manage data lifecycle, authentication, and administrative tasks using the Python SDK.
## Overview [#overview]
The SDK provides clients for managing Arc's data lifecycle features:
| Client | Purpose | Use Case |
| --------------------------- | ------------------ | ------------------------------- |
| `client.retention` | Retention policies | Delete old data on a schedule |
| `client.continuous_queries` | Continuous queries | Downsample and aggregate data |
| `client.delete` | Delete operations | Remove data matching conditions |
| `client.auth` | Authentication | Manage API tokens |
Arc Enterprise runs retention policies and continuous queries automatically on
configurable cron schedules — see
[Automated Scheduling](/arc-enterprise/operations/automated-scheduling/).
The SDK methods below trigger these operations on demand, which is still useful
for backfills, ad-hoc runs, and testing a policy before putting it on a schedule.
See [Scheduling with External Tools](#scheduling-with-external-tools) if you
prefer to drive execution from an external orchestrator (cron, Airflow, etc.).
## Retention policies [#retention-policies]
Retention policies define rules for deleting data older than a specified age. Use them to:
* Control storage costs
* Comply with data retention regulations
* Remove stale data on a schedule
### Create a policy [#create-a-policy]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
policy = client.retention.create(
name="logs-30d",
database="default",
retention_days=30,
measurement="logs", # Optional: applies to specific measurement
buffer_days=7, # Optional: keep extra days as safety buffer
)
print(f"Created policy: {policy.name} (id={policy.id})")
```
### Parameters [#parameters]
| Parameter | Type | Required | Description |
| ---------------- | ----- | -------- | ---------------------------------------------- |
| `name` | `str` | Yes | Unique name for the policy |
| `database` | `str` | Yes | Target database |
| `retention_days` | `int` | Yes | Delete data older than N days |
| `measurement` | `str` | No | Limit to specific measurement (all if omitted) |
| `buffer_days` | `int` | No | Extra buffer days before deletion |
### List policies [#list-policies]
```python
policies = client.retention.list()
for p in policies:
status = "active" if p.is_active else "inactive"
measurement = p.measurement or "all measurements"
print(f"{p.name}: {p.retention_days} days on {measurement} ({status})")
```
### Execute a policy [#execute-a-policy]
Always use `dry_run=True` first to preview what will be deleted:
```python
# Preview deletion
result = client.retention.execute(policy.id, dry_run=True)
print(f"Would delete {result.deleted_count} rows")
# Execute for real (requires confirm=True for large deletes)
result = client.retention.execute(policy.id, dry_run=False, confirm=True)
print(f"Deleted {result.deleted_count} rows")
```
### Update a policy [#update-a-policy]
```python
client.retention.update(
policy.id,
retention_days=60, # Change retention period
is_active=False, # Disable the policy
)
```
### Delete a policy [#delete-a-policy]
```python
client.retention.delete(policy.id)
```
### Full example [#full-example]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
# Create policy for logs
policy = client.retention.create(
name="logs-retention",
database="default",
retention_days=30,
measurement="logs",
)
# Create policy for metrics (keep longer)
metrics_policy = client.retention.create(
name="metrics-retention",
database="default",
retention_days=90,
measurement="metrics",
)
# List all policies
for p in client.retention.list():
print(f" - {p.name}: {p.retention_days} days")
# Dry run to see what would be deleted
result = client.retention.execute(policy.id, dry_run=True)
print(f"\nDry run: would delete {result.deleted_count} rows")
```
See [Retention Policies](/arc-enterprise/data-lifecycle/retention-policies/) for more details on how retention works in Arc.
## Continuous queries [#continuous-queries]
Continuous queries (CQs) define aggregation rules that transform data from one measurement to another. Use them to:
* Downsample high-resolution data to save storage
* Pre-compute aggregations for faster dashboard queries
* Create materialized views of your data
CQs define *what* to aggregate and *where* to store results. The `interval` parameter documents the intended frequency, but you must trigger execution manually or via an external scheduler. See [Scheduling with External Tools](#scheduling-with-external-tools).
### Create a continuous query [#create-a-continuous-query]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
cq = client.continuous_queries.create(
name="cpu-hourly-avg",
database="default",
source_measurement="cpu",
destination_measurement="cpu_1h",
query="""
SELECT
time_bucket('1 hour', time) as time,
host,
avg(usage_idle) as usage_idle,
avg(usage_system) as usage_system,
max(usage_user) as max_usage_user
FROM default.cpu
GROUP BY 1, 2
""",
interval="1h",
description="Hourly CPU averages per host",
)
print(f"Created CQ: {cq.name} (id={cq.id})")
```
### Parameters [#parameters-1]
| Parameter | Type | Required | Description |
| ------------------------- | ----- | -------- | -------------------------------------------- |
| `name` | `str` | Yes | Unique name for the CQ |
| `database` | `str` | Yes | Target database |
| `source_measurement` | `str` | Yes | Measurement to read from |
| `destination_measurement` | `str` | Yes | Measurement to write results to |
| `query` | `str` | Yes | SQL aggregation query |
| `interval` | `str` | Yes | Execution interval (e.g., `1h`, `15m`, `1d`) |
| `description` | `str` | No | Human-readable description |
### Query guidelines [#query-guidelines]
Your CQ query should:
* Use `time_bucket()` to aggregate time into intervals
* Include `time` as the first column in SELECT and GROUP BY
* Use aggregate functions (`avg`, `sum`, `count`, `min`, `max`, etc.)
* Reference the source measurement with `database.measurement` syntax
### List continuous queries [#list-continuous-queries]
```python
cqs = client.continuous_queries.list(database="default")
for cq in cqs:
status = "active" if cq.is_active else "inactive"
print(f"{cq.name}: {cq.source_measurement} → {cq.destination_measurement}")
print(f" Interval: {cq.interval} ({status})")
```
### Manual execution [#manual-execution]
Execute a CQ manually for a specific time range:
```python
# Dry run first
result = client.continuous_queries.execute(
cq.id,
start_time="2024-01-01T00:00:00Z",
end_time="2024-01-02T00:00:00Z",
dry_run=True,
)
print(f"Would process {result.records_read or 0} records")
print(f"Would write {result.records_written or 0} records")
# Execute for real
result = client.continuous_queries.execute(
cq.id,
start_time="2024-01-01T00:00:00Z",
end_time="2024-01-02T00:00:00Z",
dry_run=False,
)
```
### Update a CQ [#update-a-cq]
```python
client.continuous_queries.update(
cq.id,
interval="30m", # Change interval
is_active=False, # Pause the CQ
)
```
### Delete a CQ [#delete-a-cq]
```python
client.continuous_queries.delete(cq.id)
```
### Full example [#full-example-1]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
# Create hourly rollup
hourly_cq = client.continuous_queries.create(
name="cpu-hourly",
database="default",
source_measurement="cpu",
destination_measurement="cpu_1h",
query="""
SELECT
time_bucket('1 hour', time) as time,
host,
avg(usage_idle) as avg_idle,
min(usage_idle) as min_idle,
max(usage_idle) as max_idle
FROM default.cpu
GROUP BY 1, 2
""",
interval="1h",
)
# Create daily rollup from hourly data
daily_cq = client.continuous_queries.create(
name="cpu-daily",
database="default",
source_measurement="cpu_1h",
destination_measurement="cpu_1d",
query="""
SELECT
time_bucket('1 day', time) as time,
host,
avg(avg_idle) as avg_idle,
min(min_idle) as min_idle,
max(max_idle) as max_idle
FROM default.cpu_1h
GROUP BY 1, 2
""",
interval="1d",
)
print("Created CQ hierarchy: cpu → cpu_1h → cpu_1d")
```
See [Continuous Queries](/arc-enterprise/data-lifecycle/continuous-queries/) for more details.
## Delete operations [#delete-operations]
Delete data matching specific conditions. Use this for:
* Removing erroneous data
* Deleting data for specific hosts or time ranges
* GDPR/compliance data removal
### Delete with conditions [#delete-with-conditions]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
# ALWAYS dry_run first!
result = client.delete.delete(
database="default",
measurement="logs",
where="time < '2024-01-01' AND level = 'debug'",
dry_run=True,
)
print(f"Would delete {result.deleted_count} rows")
print(f"Affected files: {result.affected_files}")
# Execute deletion (requires confirm=True)
result = client.delete.delete(
database="default",
measurement="logs",
where="time < '2024-01-01' AND level = 'debug'",
dry_run=False,
confirm=True,
)
print(f"Deleted {result.deleted_count} rows")
```
### Parameters [#parameters-2]
| Parameter | Type | Required | Description |
| ------------- | ------ | -------- | -------------------------------------------- |
| `database` | `str` | Yes | Target database |
| `measurement` | `str` | Yes | Target measurement |
| `where` | `str` | Yes | SQL WHERE clause (without "WHERE" keyword) |
| `dry_run` | `bool` | No | Preview only, don't delete (default: `True`) |
| `confirm` | `bool` | No | Required for large deletes |
### Common delete patterns [#common-delete-patterns]
```python
# Delete old data
client.delete.delete(
database="default",
measurement="logs",
where="time < '2024-01-01'",
dry_run=False,
confirm=True,
)
# Delete data for a specific host
client.delete.delete(
database="default",
measurement="metrics",
where="host = 'decommissioned-server'",
dry_run=False,
confirm=True,
)
# Delete data in a time range
client.delete.delete(
database="default",
measurement="events",
where="time BETWEEN '2024-01-15' AND '2024-01-16'",
dry_run=False,
confirm=True,
)
```
See [Delete Operations](/arc-enterprise/data-lifecycle/delete-operations/) for more details.
## Authentication [#authentication]
Manage API tokens for accessing Arc.
### Verify current token [#verify-current-token]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
result = client.auth.verify()
if result.valid:
print(f"Token name: {result.token_info.name}")
print(f"Permissions: {result.permissions}")
print(f"Created: {result.token_info.created_at}")
else:
print("Token is invalid or expired")
```
### Create a new token [#create-a-new-token]
```python
result = client.auth.create_token(
name="my-app-token",
description="Token for my application",
permissions=["read", "write"],
)
# IMPORTANT: Save this token - it's only shown once!
print(f"New token: {result.token}")
print(f"Token ID: {result.token_id}")
```
### Available permissions [#available-permissions]
| Permission | Description |
| ---------- | -------------------------------------- |
| `read` | Query data |
| `write` | Write/ingest data |
| `admin` | Manage tokens, retention policies, CQs |
### List tokens [#list-tokens]
```python
tokens = client.auth.list_tokens()
for t in tokens:
print(f"{t.name} (id={t.id})")
print(f" Created: {t.created_at}")
print(f" Last used: {t.last_used_at or 'never'}")
```
### Rotate a token [#rotate-a-token]
Generate a new token value while keeping the same token ID and permissions:
```python
result = client.auth.rotate_token(token_id=123)
# IMPORTANT: Save the new token - the old one is now invalid!
print(f"New token: {result.new_token}")
```
### Revoke a token [#revoke-a-token]
```python
client.auth.revoke_token(token_id=123)
print("Token revoked")
```
## Error handling [#error-handling]
All data management operations can raise specific exceptions:
```python
from arc_client import ArcClient
from arc_client.exceptions import (
ArcError,
ArcNotFoundError,
ArcValidationError,
ArcAuthenticationError,
)
with ArcClient(host="localhost", token="your-token") as client:
try:
client.retention.delete(999) # Non-existent policy
except ArcNotFoundError:
print("Policy not found")
try:
client.retention.create(
name="", # Invalid name
database="default",
retention_days=-1, # Invalid days
)
except ArcValidationError as e:
print(f"Validation error: {e}")
try:
client.auth.create_token(name="test", permissions=["admin"])
except ArcAuthenticationError:
print("Current token doesn't have permission to create tokens")
```
## Async support [#async-support]
All data management operations have async equivalents:
```python
import asyncio
from arc_client import AsyncArcClient
async def main():
async with AsyncArcClient(host="localhost", token="your-token") as client:
# Retention
policy = await client.retention.create(
name="async-policy",
database="default",
retention_days=30,
)
# CQs
cq = await client.continuous_queries.create(
name="async-cq",
database="default",
source_measurement="cpu",
destination_measurement="cpu_1h",
query="SELECT time_bucket('1 hour', time) as time, avg(usage) as usage FROM default.cpu GROUP BY 1",
interval="1h",
)
# Delete
result = await client.delete.delete(
database="default",
measurement="logs",
where="time < '2024-01-01'",
dry_run=True,
)
# Auth
verify = await client.auth.verify()
asyncio.run(main())
```
## Best practices [#best-practices]
### 1. Always dry run first [#1-always-dry-run-first]
For any destructive operation (delete, retention execution), always preview first:
```python
# ✅ Good: Preview before executing
result = client.delete.delete(..., dry_run=True)
print(f"Would delete {result.deleted_count} rows")
# Review the count, then execute
# ❌ Bad: Delete without preview
result = client.delete.delete(..., dry_run=False, confirm=True)
```
### 2. Use descriptive names [#2-use-descriptive-names]
```python
# ✅ Good: Descriptive names
client.retention.create(name="logs-30d-cleanup", ...)
client.continuous_queries.create(name="cpu-hourly-avg-by-host", ...)
# ❌ Bad: Generic names
client.retention.create(name="policy1", ...)
```
### 3. Document your CQ queries [#3-document-your-cq-queries]
```python
# ✅ Good: Include description
cq = client.continuous_queries.create(
name="cpu-hourly",
description="Hourly CPU averages per host. Used by main dashboard.",
...
)
```
### 4. Secure token management [#4-secure-token-management]
```python
# ✅ Good: Use environment variables
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
token = os.getenv("ARC_TOKEN")
client = ArcClient(host="localhost", token=token)
# ❌ Bad: Hardcode tokens
client = ArcClient(host="localhost", token="arc_abc123...")
```
## Scheduling with external tools [#scheduling-with-external-tools]
Since Arc OSS doesn't include a built-in scheduler, you need to trigger retention policies and continuous queries externally. Here are several approaches:
### Simple Python script with cron [#simple-python-script-with-cron]
Create a script that executes your policies and CQs:
```python title="arc_scheduler.py"
#!/usr/bin/env python3
"""Execute Arc retention policies and continuous queries."""
import logging
from datetime import datetime, timedelta
from arc_client import ArcClient
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def run_retention_policies(client: ArcClient):
"""Execute all active retention policies."""
policies = client.retention.list()
for policy in policies:
if not policy.is_active:
continue
logger.info(f"Executing retention policy: {policy.name}")
try:
result = client.retention.execute(policy.id, dry_run=False, confirm=True)
logger.info(f" Deleted {result.deleted_count} rows")
except Exception as e:
logger.error(f" Failed: {e}")
def run_continuous_queries(client: ArcClient):
"""Execute all active continuous queries for the last interval."""
cqs = client.continuous_queries.list()
for cq in cqs:
if not cq.is_active:
continue
# Calculate time range based on CQ interval
end_time = datetime.utcnow()
# Parse interval (e.g., "1h" -> 1 hour)
interval_hours = parse_interval_hours(cq.interval)
start_time = end_time - timedelta(hours=interval_hours)
logger.info(f"Executing CQ: {cq.name} ({start_time} to {end_time})")
try:
result = client.continuous_queries.execute(
cq.id,
start_time=start_time.isoformat() + "Z",
end_time=end_time.isoformat() + "Z",
dry_run=False,
)
logger.info(f" Processed {result.records_read or 0} records")
except Exception as e:
logger.error(f" Failed: {e}")
def parse_interval_hours(interval: str) -> int:
"""Parse interval string to hours (e.g., '1h' -> 1, '1d' -> 24)."""
if interval.endswith("h"):
return int(interval[:-1])
elif interval.endswith("d"):
return int(interval[:-1]) * 24
elif interval.endswith("m"):
return max(1, int(interval[:-1]) // 60)
return 1
def main():
import os
with ArcClient(
host=os.getenv("ARC_HOST", "localhost"),
port=int(os.getenv("ARC_PORT", "8000")),
token=os.getenv("ARC_TOKEN"),
) as client:
logger.info("Starting scheduled Arc maintenance...")
run_retention_policies(client)
run_continuous_queries(client)
logger.info("Scheduled maintenance complete")
if __name__ == "__main__":
main()
```
Schedule with cron (run hourly):
```bash
# Edit crontab
crontab -e
# Add entry to run every hour
0 * * * * cd /path/to/project && ARC_TOKEN=your-token python arc_scheduler.py >> /var/log/arc_scheduler.log 2>&1
```
### Using APScheduler [#using-apscheduler]
For more control, use [APScheduler](https://apscheduler.readthedocs.io/) to run different tasks at different intervals:
```python title="arc_scheduler_advanced.py"
#!/usr/bin/env python3
"""Advanced Arc scheduler with APScheduler."""
import os
import logging
from apscheduler.schedulers.blocking import BlockingScheduler
from arc_client import ArcClient
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def get_client():
return ArcClient(
host=os.getenv("ARC_HOST", "localhost"),
token=os.getenv("ARC_TOKEN"),
)
def execute_hourly_cqs():
"""Run CQs with 1h interval."""
logger.info("Running hourly CQs...")
with get_client() as client:
for cq in client.continuous_queries.list():
if cq.is_active and cq.interval == "1h":
try:
client.continuous_queries.execute(cq.id, dry_run=False)
logger.info(f" Executed: {cq.name}")
except Exception as e:
logger.error(f" Failed {cq.name}: {e}")
def execute_daily_retention():
"""Run retention policies once per day."""
logger.info("Running daily retention...")
with get_client() as client:
for policy in client.retention.list():
if policy.is_active:
try:
result = client.retention.execute(
policy.id, dry_run=False, confirm=True
)
logger.info(f" {policy.name}: deleted {result.deleted_count} rows")
except Exception as e:
logger.error(f" Failed {policy.name}: {e}")
if __name__ == "__main__":
scheduler = BlockingScheduler()
# Run hourly CQs every hour at :05
scheduler.add_job(execute_hourly_cqs, "cron", minute=5)
# Run retention policies daily at 3:00 AM
scheduler.add_job(execute_daily_retention, "cron", hour=3, minute=0)
logger.info("Arc scheduler started. Press Ctrl+C to exit.")
scheduler.start()
```
Install and run:
```bash
pip install apscheduler
python arc_scheduler_advanced.py
```
### Docker deployment [#docker-deployment]
Run the scheduler as a Docker container alongside Arc:
```yaml title="docker-compose.yml"
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
ports:
- "8000:8000"
volumes:
- arc-data:/app/data
arc-scheduler:
build:
context: .
dockerfile: Dockerfile.scheduler
environment:
- ARC_HOST=arc
- ARC_PORT=8000
- ARC_TOKEN=${ARC_TOKEN}
depends_on:
- arc
restart: unless-stopped
volumes:
arc-data:
```
```dockerfile title="Dockerfile.scheduler"
FROM python:3.11-slim
WORKDIR /app
RUN pip install arc-tsdb-client apscheduler
COPY arc_scheduler_advanced.py .
CMD ["python", "arc_scheduler_advanced.py"]
```
### Kubernetes CronJob [#kubernetes-cronjob]
For Kubernetes deployments, use a CronJob:
```yaml title="arc-scheduler-cronjob.yaml"
apiVersion: batch/v1
kind: CronJob
metadata:
name: arc-maintenance
spec:
schedule: "0 * * * *" # Every hour
jobTemplate:
spec:
template:
spec:
containers:
- name: arc-scheduler
image: python:3.11-slim
command:
- /bin/sh
- -c
- |
pip install -q arc-tsdb-client
python -c "
from arc_client import ArcClient
import os
with ArcClient(
host=os.environ['ARC_HOST'],
token=os.environ['ARC_TOKEN']
) as client:
# Execute retention policies
for p in client.retention.list():
if p.is_active:
client.retention.execute(p.id, dry_run=False, confirm=True)
# Execute continuous queries
for cq in client.continuous_queries.list():
if cq.is_active:
client.continuous_queries.execute(cq.id, dry_run=False)
"
env:
- name: ARC_HOST
value: "arc-service"
- name: ARC_TOKEN
valueFrom:
secretKeyRef:
name: arc-secrets
key: token
restartPolicy: OnFailure
```
### Apache Airflow [#apache-airflow]
For complex workflows, use Airflow:
```python title="dags/arc_maintenance.py"
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
default_args = {
"owner": "data-team",
"retries": 3,
"retry_delay": timedelta(minutes=5),
}
def run_retention():
from arc_client import ArcClient
import os
with ArcClient(host=os.getenv("ARC_HOST"), token=os.getenv("ARC_TOKEN")) as client:
for policy in client.retention.list():
if policy.is_active:
client.retention.execute(policy.id, dry_run=False, confirm=True)
def run_hourly_cqs():
from arc_client import ArcClient
import os
with ArcClient(host=os.getenv("ARC_HOST"), token=os.getenv("ARC_TOKEN")) as client:
for cq in client.continuous_queries.list():
if cq.is_active and cq.interval == "1h":
client.continuous_queries.execute(cq.id, dry_run=False)
with DAG(
"arc_maintenance",
default_args=default_args,
schedule_interval="@hourly",
start_date=datetime(2024, 1, 1),
catchup=False,
) as dag:
retention_task = PythonOperator(
task_id="run_retention",
python_callable=run_retention,
)
cq_task = PythonOperator(
task_id="run_hourly_cqs",
python_callable=run_hourly_cqs,
)
retention_task >> cq_task
```
## Next steps [#next-steps]
* **[Data Ingestion](/arc-enterprise/sdks/python/ingestion/)** - Write data to Arc
* **[Querying](/arc-enterprise/sdks/python/querying/)** - Query data with DataFrames
* **[Retention Policies](/arc-enterprise/data-lifecycle/retention-policies/)** - Deep dive on retention
* **[Continuous Queries](/arc-enterprise/data-lifecycle/continuous-queries/)** - Deep dive on CQs
# Python SDK (/arc-enterprise/sdks/python)
Official Python SDK for the Arc Enterprise time-series database.
[](https://badge.fury.io/py/arc-tsdb-client)
[](https://www.python.org/downloads/)
## What is it? [#what-is-it]
The `arc-tsdb-client` package is the official Python client for [Arc](https://github.com/basekick-labs/arc), an open, SQL-native time-series database. It provides a high-level, Pythonic interface for:
* **Writing data** at scale
* **Querying with SQL** and getting results as DataFrames
* **Managing data lifecycle** (retention, aggregation, deletion)
* **Handling authentication** (tokens, permissions)
## Why use the SDK? [#why-use-the-sdk]
While you can interact with Arc's REST API directly, the SDK provides significant advantages:
| Feature | Raw API | Python SDK |
| --------------------- | -------------------- | ---------------------------- |
| Connection management | Manual | Automatic (context managers) |
| Data serialization | Manual MessagePack | Automatic |
| DataFrame support | Convert manually | Native pandas/polars/arrow |
| Buffered writes | Implement yourself | Built-in with auto-batching |
| Error handling | Parse HTTP responses | Typed exceptions |
| Async support | Manual httpx/aiohttp | Built-in `AsyncArcClient` |
| Compression | Configure headers | Automatic gzip |
## Quick example [#quick-example]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
# Write metrics
client.write.write_columnar(
measurement="cpu",
columns={
"time": [1704067200000000, 1704067260000000],
"host": ["server01", "server01"],
"usage_idle": [95.2, 94.8],
},
)
# Query to pandas
df = client.query.query_pandas(
"SELECT * FROM default.cpu WHERE host = 'server01'"
)
print(df)
```
## Architecture [#architecture]
The SDK is organized into specialized clients for different operations:
```text
ArcClient
├── .write # Data ingestion (WriteClient)
│ ├── write_columnar()
│ ├── write_dataframe()
│ ├── write_line_protocol()
│ └── buffered()
│
├── .query # Data querying (QueryClient)
│ ├── query()
│ ├── query_pandas()
│ ├── query_polars()
│ ├── query_arrow()
│ └── estimate()
│
├── .retention # Retention policies (RetentionClient)
├── .continuous_queries # CQs (ContinuousQueryClient)
├── .delete # Delete operations (DeleteClient)
├── .auth # Authentication (AuthClient)
└── .health() # Health check
```
## Documentation [#documentation]
## Source code [#source-code]
* **Repository**: [github.com/basekick-labs/arc-client-python](https://github.com/basekick-labs/arc-client-python)
* **PyPI**: [pypi.org/project/arc-tsdb-client](https://pypi.org/project/arc-tsdb-client/)
# Data Ingestion (/arc-enterprise/sdks/python/ingestion)
How to write data to Arc using the Python SDK.
## Overview [#overview]
The SDK provides multiple ways to ingest data, each optimized for different use cases:
| Method | Best For | Performance | Format |
| ----------------------- | ------------------------- | ------------ | -------------------- |
| `write_columnar()` | High-throughput ingestion | Highest | MessagePack columnar |
| `write_dataframe()` | pandas/polars workflows | High | MessagePack columnar |
| `buffered()` | Streaming data | Auto-batched | MessagePack columnar |
| `write_line_protocol()` | InfluxDB compatibility | Moderate | Line protocol text |
All write methods are available on `client.write`.
## Columnar format (recommended) [#columnar-format-recommended]
The fastest way to write data. Data is organized by columns (like a DataFrame) rather than rows, which enables efficient compression and fast serialization.
### Basic usage [#basic-usage]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
client.write.write_columnar(
measurement="cpu",
columns={
"time": [1704067200000000, 1704067260000000, 1704067320000000],
"host": ["server01", "server01", "server01"],
"region": ["us-east", "us-east", "us-east"],
"usage_idle": [95.2, 94.8, 93.1],
"usage_system": [2.1, 2.5, 3.2],
"usage_user": [2.7, 2.7, 3.7],
},
)
```
### Parameters [#parameters]
| Parameter | Type | Required | Description |
| ------------- | ------ | -------- | -------------------------------------------- |
| `measurement` | `str` | Yes | Name of the measurement (like a table) |
| `columns` | `dict` | Yes | Column name → list of values |
| `database` | `str` | No | Target database (default: client's database) |
### Column types [#column-types]
The SDK automatically handles type conversion:
| Python Type | Arc Type | Example |
| ----------- | ------------------------ | --------------------- |
| `int` | Integer | `[1, 2, 3]` |
| `float` | Float | `[1.5, 2.7, 3.9]` |
| `str` | String (tag or field) | `["a", "b", "c"]` |
| `bool` | Boolean | `[True, False, True]` |
| `datetime` | Timestamp (microseconds) | `[datetime.now()]` |
### Timestamps [#timestamps]
The `time` column must contain a **numeric Unix epoch**. Arc auto-detects the
unit by magnitude — seconds, milliseconds, microseconds, or nanoseconds — and
normalizes everything to microseconds internally. Microseconds is the
recommended, unambiguous form:
```python
import time
from datetime import datetime
# From current time
timestamp_us = int(time.time() * 1_000_000)
# From datetime
timestamp_us = int(datetime.now().timestamp() * 1_000_000)
# Multiple timestamps
timestamps = [
1704067200000000, # 2024-01-01 00:00:00 UTC
1704067260000000, # 2024-01-01 00:01:00 UTC
1704067320000000, # 2024-01-01 00:02:00 UTC
]
```
### Tags vs fields [#tags-vs-fields]
In Arc:
* **Tags** are indexed string columns used for filtering (e.g., `host`, `region`, `sensor_id`)
* **Fields** are the actual metric values (e.g., `temperature`, `cpu_usage`, `count`)
For `write_columnar()`, all columns are sent directly to Arc. Tag/field distinction is handled by the Arc server based on its schema detection.
For `write_dataframe()`, you can explicitly specify which columns are tags using the `tag_columns` parameter.
## DataFrame ingestion [#dataframe-ingestion]
Write directly from pandas or polars DataFrames. The SDK converts DataFrames to columnar format automatically.
### pandas example [#pandas-example]
```python
import pandas as pd
from arc_client import ArcClient
# Create a DataFrame
df = pd.DataFrame({
"time": pd.date_range("2024-01-01", periods=100, freq="1min"),
"host": ["server-01"] * 50 + ["server-02"] * 50,
"region": ["us-east"] * 100,
"cpu_usage": [50 + i * 0.1 for i in range(100)],
"memory_mb": [1024 + i for i in range(100)],
})
with ArcClient(host="localhost", token="your-token") as client:
client.write.write_dataframe(
df,
measurement="server_metrics",
time_column="time", # Column containing timestamps
tag_columns=["host", "region"], # Columns to treat as tags
)
print(f"Wrote {len(df)} rows")
```
### Parameters [#parameters-1]
| Parameter | Type | Required | Description |
| ------------- | ----------- | -------- | -------------------------------- |
| `df` | DataFrame | Yes | pandas or polars DataFrame |
| `measurement` | `str` | Yes | Target measurement name |
| `time_column` | `str` | Yes | Name of the timestamp column |
| `tag_columns` | `list[str]` | No | Columns to treat as indexed tags |
| `database` | `str` | No | Target database |
### Polars example [#polars-example]
```python
import polars as pl
from arc_client import ArcClient
df = pl.DataFrame({
"time": pl.datetime_range(
datetime(2024, 1, 1),
datetime(2024, 1, 1, 1),
interval="1m",
eager=True
),
"sensor_id": ["sensor-001"] * 61,
"temperature": [20.0 + i * 0.1 for i in range(61)],
})
with ArcClient(host="localhost", token="your-token") as client:
client.write.write_dataframe(
df,
measurement="temperatures",
time_column="time",
tag_columns=["sensor_id"],
)
```
## Buffered writes [#buffered-writes]
For streaming or high-throughput scenarios, use buffered writes. The buffer automatically batches records and flushes them efficiently.
### Basic usage [#basic-usage-1]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
with client.write.buffered(batch_size=5000, flush_interval=2.0) as buffer:
for i in range(50000):
buffer.write(
measurement="events",
tags={"source": "sensor-001", "type": "temperature"},
fields={"value": 22.5 + i * 0.01},
timestamp=1704067200000000 + (i * 1000),
)
# Auto-flushes on exit
```
### Parameters [#parameters-2]
| Parameter | Type | Default | Description |
| ---------------- | ------- | ------- | ---------------------------------------------- |
| `batch_size` | `int` | 5000 | Flush after N records |
| `flush_interval` | `float` | 5.0 | Flush after N seconds (even if batch not full) |
### How it works [#how-it-works]
1. Records are queued in memory
2. When `batch_size` is reached OR `flush_interval` expires, the buffer flushes
3. On context manager exit, any remaining records are flushed
4. Uses columnar format internally for best performance
### When to use buffered writes [#when-to-use-buffered-writes]
✅ **Use buffered writes when:**
* Processing streaming data (sensors, logs, events)
* Ingesting data in a loop
* You don't know the batch size ahead of time
❌ **Don't use buffered writes when:**
* You already have data in columnar format or DataFrame
* You're writing a single batch (use `write_columnar()` directly)
### Async buffered writes [#async-buffered-writes]
```python
import asyncio
from arc_client import AsyncArcClient
async def ingest_stream():
async with AsyncArcClient(host="localhost", token="your-token") as client:
async with client.write.buffered(batch_size=5000) as buffer:
async for event in event_stream():
await buffer.write(
measurement="events",
tags={"source": event.source},
fields={"value": event.value},
timestamp=event.timestamp,
)
asyncio.run(ingest_stream())
```
## Line Protocol [#line-protocol]
For compatibility with InfluxDB tooling (Telegraf, etc.), use line protocol format.
### Basic usage [#basic-usage-2]
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
# Single line
client.write.write_line_protocol(
"cpu,host=server01,region=us-east usage_idle=95.2 1704067200000000000"
)
# Multiple lines
lines = [
"cpu,host=server01 usage_idle=95.2,usage_system=2.1",
"cpu,host=server02 usage_idle=87.5,usage_system=4.3",
"mem,host=server01 used_percent=45.2",
]
client.write.write_line_protocol(lines)
```
### Line Protocol format [#line-protocol-format]
```text
,=,... =,... [timestamp]
```
Example breakdown:
```text
cpu,host=server01,region=us-east usage_idle=95.2,usage_system=2.1 1704067200000000000
│ │ │ │
│ │ │ └── timestamp (nanoseconds)
│ │ └── fields (space-separated from tags)
│ └── tags (comma-separated)
└── measurement name
```
### When to use Line Protocol [#when-to-use-line-protocol]
✅ **Use line protocol when:**
* Integrating with Telegraf or other InfluxDB tools
* Migrating from InfluxDB
* You already have data in line protocol format
❌ **Don't use line protocol when:**
* Building new applications (use columnar format)
* Performance is critical (columnar is considerably faster)
## Async ingestion [#async-ingestion]
All write methods have async equivalents:
```python
import asyncio
from arc_client import AsyncArcClient
async def main():
async with AsyncArcClient(host="localhost", token="your-token") as client:
# Columnar write
await client.write.write_columnar(
measurement="cpu",
columns={
"time": [1704067200000000],
"host": ["server01"],
"usage": [45.2],
},
)
# DataFrame write
await client.write.write_dataframe(
df, measurement="metrics", time_column="time"
)
# Line protocol
await client.write.write_line_protocol("cpu,host=server01 usage=45.2")
asyncio.run(main())
```
## Error handling [#error-handling]
```python
from arc_client import ArcClient
from arc_client.exceptions import (
ArcIngestionError,
ArcValidationError,
ArcConnectionError,
)
with ArcClient(host="localhost", token="your-token") as client:
try:
client.write.write_columnar(
measurement="cpu",
columns={"time": [1], "value": [1.0]},
)
except ArcValidationError as e:
print(f"Invalid data: {e}")
except ArcIngestionError as e:
print(f"Write failed: {e}")
except ArcConnectionError as e:
print(f"Connection error: {e}")
```
## Best practices [#best-practices]
### 1. Batch your data [#1-batch-your-data]
Send multiple rows per request rather than one at a time:
```python
# ✅ Good: Batch write
client.write.write_columnar(
measurement="cpu",
columns={
"time": [t1, t2, t3, ...], # 1000+ values
"host": [h1, h2, h3, ...],
"value": [v1, v2, v3, ...],
},
)
# ❌ Bad: Individual writes
for record in records:
client.write.write_columnar(
measurement="cpu",
columns={
"time": [record.time],
"host": [record.host],
"value": [record.value],
},
)
```
### 2. Use appropriate batch sizes [#2-use-appropriate-batch-sizes]
* **Small batches** (100-1000): Lower latency, more HTTP overhead
* **Medium batches** (1000-10000): Good balance for most use cases
* **Large batches** (10000+): Best throughput, higher memory usage
### 3. Handle backpressure [#3-handle-backpressure]
For high-throughput scenarios, implement backpressure handling:
```python
import time
def write_with_backoff(client, data, max_retries=3):
for attempt in range(max_retries):
try:
client.write.write_columnar(**data)
return
except ArcIngestionError as e:
if attempt == max_retries - 1:
raise
time.sleep(2 ** attempt) # Exponential backoff
```
## Next steps [#next-steps]
* **[Querying](/arc-enterprise/sdks/python/querying/)** - Query data and work with DataFrames
* **[Data Management](/arc-enterprise/sdks/python/data-management/)** - Retention, CQs, and deletion
* **[API Reference](/arc-enterprise/api-reference/overview/)** - Raw REST API documentation
# Install the Python SDK (/arc-enterprise/sdks/python/installation)
How to install the Arc Python SDK and its optional dependencies.
## Requirements [#requirements]
* **Python 3.9+** (3.10+ recommended for best performance)
* **Arc server** running and accessible
## Basic installation [#basic-installation]
Install the core SDK with pip:
```bash
pip install arc-tsdb-client
```
This installs the minimal dependencies needed to connect to Arc and write/query data using JSON responses.
**Core dependencies:**
* `httpx` - HTTP client with connection pooling
* `msgpack` - Binary serialization for high-performance writes
## Optional dependencies [#optional-dependencies]
The SDK supports optional extras for DataFrame integration:
### pandas support [#pandas-support]
For `query_pandas()` and `write_dataframe()` with pandas:
```bash
pip install arc-tsdb-client[pandas]
```
**What it adds:**
* `pandas` - DataFrame library
* `pyarrow` - Required for efficient Arrow-to-pandas conversion
**Use when:** You're working in Jupyter notebooks, data science workflows, or need pandas DataFrames.
### Polars support [#polars-support]
For `query_polars()` with the high-performance Polars library:
```bash
pip install arc-tsdb-client[polars]
```
**What it adds:**
* `polars` - Fast DataFrame library written in Rust
* `pyarrow` - Required for Arrow IPC parsing
**Use when:** You need maximum query performance, are processing large datasets, or prefer Polars' API.
### All dependencies [#all-dependencies]
Install everything for full functionality:
```bash
pip install arc-tsdb-client[all]
```
**What it adds:**
* `pandas`
* `polars`
* `pyarrow`
**Use when:** You want access to all features without worrying about which extras you need.
## Using uv (recommended) [#using-uv-recommended]
[uv](https://github.com/astral-sh/uv) is a fast Python package manager. Install the SDK with:
```bash
# Core only
uv add arc-tsdb-client
# With pandas
uv add arc-tsdb-client --extra pandas
# With all extras
uv add arc-tsdb-client --extra all
```
## Verifying installation [#verifying-installation]
Test that the SDK is installed correctly:
```python
from arc_client import ArcClient, AsyncArcClient
# Check version
import arc_client
print(f"arc-tsdb-client version: {arc_client.__version__}")
# Test connection
with ArcClient(host="localhost", token="your-token") as client:
health = client.health()
print(f"Server status: {health.status}")
```
## Configuration [#configuration]
### Basic client setup [#basic-client-setup]
```python
from arc_client import ArcClient
client = ArcClient(
host="localhost", # Arc server hostname
port=8000, # Arc server port (default: 8000)
token="your-token", # API token (required)
database="default", # Default database for queries
timeout=30.0, # Request timeout in seconds
compression=True, # Enable gzip compression for writes
ssl=False, # Use HTTPS instead of HTTP
verify_ssl=True, # Verify SSL certificates
)
```
### Environment variables [#environment-variables]
You can also configure the client using environment variables:
```bash
export ARC_HOST="localhost"
export ARC_PORT="8000"
export ARC_TOKEN="your-token"
export ARC_DATABASE="default"
```
```python
import os
ARC_TOKEN = os.environ["ARC_TOKEN"]
from arc_client import ArcClient
client = ArcClient(
host=os.getenv("ARC_HOST", "localhost"),
port=int(os.getenv("ARC_PORT", "8000")),
token=os.getenv("ARC_TOKEN"),
database=os.getenv("ARC_DATABASE", "default"),
)
```
### Context manager (recommended) [#context-manager-recommended]
Always use the client as a context manager to ensure proper connection cleanup:
```python
# Sync client
with ArcClient(host="localhost", token="your-token") as client:
# Use client...
pass
# Connection automatically closed
# Async client
async with AsyncArcClient(host="localhost", token="your-token") as client:
# Use client...
pass
# Connection automatically closed
```
### Manual connection management [#manual-connection-management]
If you can't use a context manager:
```python
client = ArcClient(host="localhost", token="your-token")
try:
# Use client...
pass
finally:
client.close()
```
## Troubleshooting [#troubleshooting]
### ImportError: No module named 'pandas' [#importerror-no-module-named-pandas]
You need to install the pandas extra:
```bash
pip install arc-tsdb-client[pandas]
```
### ImportError: No module named 'polars' [#importerror-no-module-named-polars]
You need to install the polars extra:
```bash
pip install arc-tsdb-client[polars]
```
### Connection refused [#connection-refused]
Make sure Arc is running and accessible:
```bash
curl http://localhost:8000/health
```
### Authentication failed [#authentication-failed]
Verify your token is correct:
```python
with ArcClient(host="localhost", token="your-token") as client:
result = client.auth.verify()
if result.valid:
print(f"Token is valid: {result.token_info.name}")
else:
print("Token is invalid")
```
## Next steps [#next-steps]
* **[Data Ingestion](/arc-enterprise/sdks/python/ingestion/)** - Learn how to write data to Arc
* **[Querying](/arc-enterprise/sdks/python/querying/)** - Query data and work with DataFrames
* **[Data Management](/arc-enterprise/sdks/python/data-management/)** - Manage retention, CQs, and more
# Querying with the Python SDK (/arc-enterprise/sdks/python/querying)
How to query data from Arc using the Python SDK.
## Overview [#overview]
The SDK provides multiple query methods, each returning data in a different format:
| Method | Returns | Best For | Performance |
| ---------------- | -------------------- | --------------------------- | ----------- |
| `query()` | `QueryResult` object | Simple queries, inspection | Good |
| `query_pandas()` | pandas DataFrame | Data science, notebooks | Good |
| `query_polars()` | polars DataFrame | Large datasets, performance | Better |
| `query_arrow()` | PyArrow Table | Zero-copy, interop | Best |
All query methods are available on `client.query`.
## SQL syntax [#sql-syntax]
Arc uses SQL with the table syntax `database.measurement`:
```sql
SELECT * FROM default.cpu WHERE host = 'server01'
```
Arc runs a full analytical SQL engine, so you have access to:
* Window functions
* CTEs (Common Table Expressions)
* `time_bucket()` for time-series aggregation
* JSON functions
* And more
## Basic query (JSON) [#basic-query-json]
The simplest way to query data. Returns a `QueryResult` object with columns and data.
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
result = client.query.query(
"SELECT * FROM default.cpu WHERE time > now() - INTERVAL '1 hour' LIMIT 100"
)
print(f"Columns: {result.columns}")
print(f"Row count: {result.row_count}")
for row in result.data:
print(row)
```
### QueryResult object [#queryresult-object]
| Property | Type | Description |
| ----------- | ------------ | ----------------------- |
| `columns` | `list[str]` | Column names |
| `data` | `list[list]` | Rows as nested lists |
| `row_count` | `int` | Number of rows returned |
### When to use [#when-to-use]
✅ **Use `query()` when:**
* You need to inspect results quickly
* Working with small result sets
* Don't need DataFrame functionality
❌ **Don't use when:**
* Processing large datasets (use Arrow)
* Need DataFrame operations (use pandas/polars)
## pandas DataFrame [#pandas-dataframe]
Returns query results as a pandas DataFrame. Requires `pip install arc-tsdb-client[pandas]`.
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
df = client.query.query_pandas(
"SELECT * FROM default.cpu WHERE host = 'server01' LIMIT 1000"
)
print(df.head())
print(df.dtypes)
# Use pandas operations
avg_by_host = df.groupby("host")["usage_idle"].mean()
print(avg_by_host)
```
### How it works [#how-it-works]
1. Query is sent to Arc
2. Results are returned as Arrow IPC stream
3. Arrow data is converted to pandas DataFrame (zero-copy where possible)
### When to use [#when-to-use-1]
✅ **Use `query_pandas()` when:**
* Working in Jupyter notebooks
* Need pandas-specific operations
* Integrating with pandas-based tools (matplotlib, seaborn, scikit-learn)
❌ **Don't use when:**
* Processing very large datasets (polars is faster)
* Memory is constrained (polars uses less memory)
## Polars DataFrame [#polars-dataframe]
Returns query results as a polars DataFrame. Requires `pip install arc-tsdb-client[polars]`.
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
df = client.query.query_polars(
"SELECT * FROM default.cpu LIMIT 100000"
)
print(df.head())
# Use polars operations (lazy evaluation, parallel execution)
result = (
df.lazy()
.filter(pl.col("usage_idle") > 90)
.group_by("host")
.agg(pl.col("usage_idle").mean().alias("avg_idle"))
.collect()
)
print(result)
```
### Why Polars? [#why-polars]
Polars is a DataFrame library written in Rust that offers:
* **Faster operations**: Especially for large datasets
* **Lower memory usage**: Efficient memory layout
* **Lazy evaluation**: Optimize query plans before execution
* **Parallel execution**: Uses all CPU cores automatically
### When to use [#when-to-use-2]
✅ **Use `query_polars()` when:**
* Processing large datasets (100K+ rows)
* Performance is critical
* You prefer polars' API
* Running in production pipelines
❌ **Don't use when:**
* Need pandas compatibility for downstream tools
* Working in environments that only support pandas
## PyArrow table (zero-copy) [#pyarrow-table-zero-copy]
Returns query results as a PyArrow Table. This is the most efficient option for large datasets.
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
table = client.query.query_arrow(
"SELECT * FROM default.cpu LIMIT 1000000"
)
print(f"Rows: {table.num_rows}")
print(f"Columns: {table.num_columns}")
print(f"Schema: {table.schema}")
# Convert to pandas (zero-copy where possible)
df = table.to_pandas()
# Convert to polars
import polars as pl
df = pl.from_arrow(table)
# Save to Parquet
import pyarrow.parquet as pq
pq.write_table(table, "output.parquet")
```
### Why Arrow? [#why-arrow]
Apache Arrow is a columnar memory format that enables:
* **Zero-copy reads**: Data stays in the same memory layout
* **Interoperability**: Share data between pandas, polars, DuckDB, Spark
* **Efficient serialization**: Arrow IPC format is compact and fast
### When to use [#when-to-use-3]
✅ **Use `query_arrow()` when:**
* Processing very large datasets
* Need to pass data to multiple tools
* Saving results to Parquet files
* Maximum performance is required
## Query estimation [#query-estimation]
Preview the cost of a query before executing it:
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
estimate = client.query.estimate(
"SELECT * FROM default.cpu WHERE time > now() - INTERVAL '30 days'"
)
print(f"Estimated rows: {estimate.estimated_rows}")
print(f"Warning level: {estimate.warning_level}") # none, low, medium, high
if estimate.warning_level == "high":
print("Consider adding filters or LIMIT clause")
```
### Estimate result [#estimate-result]
| Property | Type | Description |
| ---------------- | ----- | ------------------------------- |
| `estimated_rows` | `int` | Approximate row count |
| `warning_level` | `str` | `none`, `low`, `medium`, `high` |
## List measurements [#list-measurements]
Discover what measurements exist in a database:
```python
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
measurements = client.query.list_measurements(database="default")
for m in measurements:
print(f"{m.measurement}:")
print(f" Files: {m.file_count}")
print(f" Size: {m.total_size_mb:.1f} MB")
```
## Common query patterns [#common-query-patterns]
### Time-series aggregation [#time-series-aggregation]
Use `time_bucket()` to aggregate data into time intervals:
```python
df = client.query.query_pandas("""
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
host,
AVG(usage_idle) as avg_idle,
MAX(usage_system) as max_system,
COUNT(*) as samples
FROM default.cpu
WHERE time > now() - INTERVAL '1 hour'
GROUP BY bucket, host
ORDER BY bucket DESC
""")
```
### Latest value per host [#latest-value-per-host]
```python
df = client.query.query_pandas("""
SELECT DISTINCT ON (host)
time, host, usage_idle, usage_system
FROM default.cpu
ORDER BY host, time DESC
""")
```
### Percentiles [#percentiles]
```python
df = client.query.query_pandas("""
SELECT
host,
percentile_cont(0.50) WITHIN GROUP (ORDER BY usage_idle) as p50,
percentile_cont(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
percentile_cont(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
FROM default.cpu
WHERE time > now() - INTERVAL '24 hours'
GROUP BY host
""")
```
### Join measurements [#join-measurements]
```python
df = client.query.query_pandas("""
SELECT
c.time,
c.host,
c.usage_idle as cpu_idle,
m.used_percent as mem_used
FROM default.cpu c
JOIN default.mem m
ON c.time = m.time AND c.host = m.host
WHERE c.time > now() - INTERVAL '10 minutes'
ORDER BY c.time DESC
""")
```
## Async queries [#async-queries]
All query methods have async equivalents:
```python
import asyncio
from arc_client import AsyncArcClient
async def main():
async with AsyncArcClient(host="localhost", token="your-token") as client:
# JSON result
result = await client.query.query("SELECT * FROM default.cpu LIMIT 10")
# pandas DataFrame
df = await client.query.query_pandas("SELECT * FROM default.cpu LIMIT 1000")
# Polars DataFrame
pl_df = await client.query.query_polars("SELECT * FROM default.cpu LIMIT 1000")
# Arrow Table
table = await client.query.query_arrow("SELECT * FROM default.cpu LIMIT 10000")
asyncio.run(main())
```
### Concurrent queries [#concurrent-queries]
Run multiple queries in parallel:
```python
import asyncio
from arc_client import AsyncArcClient
async def main():
async with AsyncArcClient(host="localhost", token="your-token") as client:
# Run queries concurrently
cpu_task = client.query.query_pandas("SELECT * FROM default.cpu LIMIT 1000")
mem_task = client.query.query_pandas("SELECT * FROM default.mem LIMIT 1000")
disk_task = client.query.query_pandas("SELECT * FROM default.disk LIMIT 1000")
cpu_df, mem_df, disk_df = await asyncio.gather(cpu_task, mem_task, disk_task)
print(f"CPU: {len(cpu_df)} rows")
print(f"Memory: {len(mem_df)} rows")
print(f"Disk: {len(disk_df)} rows")
asyncio.run(main())
```
## Error handling [#error-handling]
```python
from arc_client import ArcClient
from arc_client.exceptions import (
ArcQueryError,
ArcConnectionError,
ArcAuthenticationError,
)
with ArcClient(host="localhost", token="your-token") as client:
try:
df = client.query.query_pandas("SELECT * FROM nonexistent.table")
except ArcQueryError as e:
print(f"Query failed: {e}") # Invalid SQL or table not found
except ArcAuthenticationError as e:
print(f"Auth failed: {e}") # Invalid token
except ArcConnectionError as e:
print(f"Connection error: {e}") # Server unreachable
```
## Performance tips [#performance-tips]
### 1. Filter early [#1-filter-early]
Push filters to Arc rather than filtering in Python:
```python
# ✅ Good: Filter in SQL
df = client.query.query_pandas("""
SELECT * FROM default.cpu
WHERE time > now() - INTERVAL '1 hour'
AND host = 'server01'
""")
# ❌ Bad: Fetch all, filter in Python
df = client.query.query_pandas("SELECT * FROM default.cpu")
df = df[df["host"] == "server01"]
```
### 2. Select only needed columns [#2-select-only-needed-columns]
```python
# ✅ Good: Select specific columns
df = client.query.query_pandas("""
SELECT time, host, usage_idle FROM default.cpu
""")
# ❌ Bad: Select all columns
df = client.query.query_pandas("SELECT * FROM default.cpu")
```
### 3. Use LIMIT for exploration [#3-use-limit-for-exploration]
```python
# ✅ Good: Limit during exploration
df = client.query.query_pandas("SELECT * FROM default.cpu LIMIT 100")
```
### 4. Use Arrow for large results [#4-use-arrow-for-large-results]
```python
# For 100K+ rows, Arrow is significantly faster
table = client.query.query_arrow("SELECT * FROM default.cpu")
df = table.to_pandas() # Zero-copy conversion
```
## Next steps [#next-steps]
* **[Data Management](/arc-enterprise/sdks/python/data-management/)** - Retention, CQs, and deletion
* **[Data Ingestion](/arc-enterprise/sdks/python/ingestion/)** - Write data to Arc
* **[API Reference](/arc-enterprise/api-reference/overview/)** - Raw REST API documentation