MQTT Integration
Subscribe an Arc Enterprise cluster to MQTT brokers through the API, start and stop subscriptions on a node without restarting it, and from v27.01.1 map topics to measurements and tags.
Ingest data directly from MQTT brokers into Arc. Connect to IoT devices, industrial sensors, and message brokers without middleware.
Overview
Arc provides native MQTT subscription with dynamic, API-driven configuration. Manage multiple MQTT brokers and subscriptions at runtime without server restarts.
Key features:
- API-driven subscription management - Create, update, delete, start/stop subscriptions via REST API
- Multiple simultaneous brokers - Connect to different MQTT brokers for different data sources
- Topic wildcards - Subscribe using
+(single level) and#(multi-level) wildcards - Auto-detection - Automatically detects JSON and MessagePack message formats
- High performance - MessagePack columnar format for high-throughput ingestion
- Topic mapping - Route a topic to another database; from v27.01.1, also set the measurement and extract tags from topic path segments
- TLS/SSL support - Client certificates and CA verification
- Encrypted credentials - Passwords encrypted at rest using AES-256-GCM
- Auto-reconnect - Exponential backoff on connection loss
- QoS support - QoS 0, 1, and 2
Prerequisites
- Arc server running (v26.02.1 or higher)
- Arc API token (if authentication is enabled)
- MQTT broker accessible from Arc server
Quick start
1. Enable MQTT in Arc
[mqtt]
enabled = trueOr via environment variable:
ARC_MQTT_ENABLED=trueRestart Arc to apply the configuration.
2. Create a subscription
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#"],
"database": "iot",
"auto_start": true
}'Response:
{
"success": true,
"subscription": {
"id": "056fbfda-eab0-4358-b139-42c539896a1f",
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"client_id": "arc-e5ccee1d",
"topics": ["sensors/#"],
"qos": 1,
"database": "iot",
"has_password": false,
"tls_enabled": false,
"tls_insecure_skip_verify": false,
"auto_start": true,
"status": "stopped",
"keep_alive_seconds": 60,
"connect_timeout_seconds": 30,
"reconnect_max_seconds": 60,
"clean_session": false,
"created_at": "2026-02-13T10:00:00Z",
"updated_at": "2026-02-13T10:00:00Z"
}
}3. Start the subscription
Up to Arc 26.09.x, a new subscription is created stopped, even with "auto_start": true: auto_start only makes Arc start the subscription each time the server starts. From 27.01.1, "auto_start": true also starts it on creation, and the response shows "status": "running", or "error" with the reason in error_message if it cannot connect (the subscription is still created). Start it, replacing <id> with the id from the response. Subscriptions are stored on the node that created them and MQTT API calls are not forwarded between nodes, so send this request to that node directly, not through a load balancer:
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions/<id>/start \
-H "Authorization: Bearer $ARC_TOKEN"Response:
{"success": true, "message": "Subscription started"}GET /api/v1/mqtt/subscriptions/{id} now reports "status": "running".
4. Send test data
Publish a message to your MQTT broker. The m key names the measurement; a payload without m or measurement is written to the mqtt measurement (see Measurement, tags, and fields):
mosquitto_pub -h localhost -t "sensors/temperature" \
-m '{"m": "temperature", "time": 1706745600000000, "value": 23.5, "device_id": "sensor-001"}'5. Query the data
curl -X POST http://localhost:8000/api/v1/query \
-H "Authorization: Bearer $ARC_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"sql": "SELECT * FROM iot.temperature ORDER BY time DESC LIMIT 10",
"format": "json"
}'API reference
Subscription management
| Method | Endpoint | Description |
|---|---|---|
POST | /api/v1/mqtt/subscriptions | Create a new subscription |
GET | /api/v1/mqtt/subscriptions | List all subscriptions |
GET | /api/v1/mqtt/subscriptions/{id} | Get subscription details |
PUT | /api/v1/mqtt/subscriptions/{id} | Update subscription |
DELETE | /api/v1/mqtt/subscriptions/{id} | Delete subscription |
Lifecycle control
| Method | Endpoint | Description |
|---|---|---|
POST | /api/v1/mqtt/subscriptions/{id}/start | Start subscription |
POST | /api/v1/mqtt/subscriptions/{id}/stop | Stop subscription |
POST | /api/v1/mqtt/subscriptions/{id}/restart | Restart subscription |
Monitoring
| Method | Endpoint | Description |
|---|---|---|
GET | /api/v1/mqtt/subscriptions/{id}/stats | Get subscription stats |
GET | /api/v1/mqtt/stats | Aggregate stats (all subscriptions) |
GET | /api/v1/mqtt/health | Health check |
Subscription options
Create subscription request
{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["sensors/#", "factory/+/metrics"],
"database": "iot",
"qos": 1,
"client_id": "arc-factory",
"username": "mqtt_user",
"password": "mqtt_pass",
"tls_enabled": false,
"tls_cert_path": "/path/to/client.crt",
"tls_key_path": "/path/to/client.key",
"tls_ca_path": "/path/to/ca.crt",
"topic_mapping": {},
"keep_alive_seconds": 60,
"connect_timeout_seconds": 30,
"reconnect_max_seconds": 60,
"auto_start": true
}Field reference
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string | Yes | - | Unique subscription name |
broker | string | Yes | - | MQTT broker URL (tcp://, ssl://, ws://, wss://) |
topics | array | Yes | - | List of topics to subscribe |
database | string | Yes | - | Target Arc database |
qos | int | No | 1 | QoS level: 0, 1, or 2 |
client_id | string | No | auto | MQTT client ID |
username | string | No | - | MQTT username |
password | string | No | - | MQTT password (encrypted at rest) |
tls_enabled | bool | No | false | Enable TLS/SSL |
tls_cert_path | string | No | - | Client certificate path |
tls_key_path | string | No | - | Client key path |
tls_ca_path | string | No | - | CA certificate path |
topic_mapping | object | No | In 26.09.x and earlier: exact topic to database name. From v27.01.1: topic filter (+, #) to a database name or a rule setting database, measurement and tags_from_topic. See Topic mapping. | |
keep_alive_seconds | int | No | 60 | MQTT keep-alive interval |
connect_timeout_seconds | int | No | 30 | Connection timeout |
reconnect_max_seconds | int | No | 60 | Maximum reconnect backoff delay. The reconnect delay starts at 1 second and doubles up to this cap (the 1-second minimum is fixed by the MQTT client library and is not configurable). |
auto_start | bool | No | false | Start the subscription whenever Arc starts. Up to 26.09.x, creating a subscription does not start it: call POST /api/v1/mqtt/subscriptions/{id}/start. From 27.01.1 it also starts on creation. |
Message formats
Arc automatically detects the message format based on content.
JSON single record
{
"time": 1706745600000000,
"temperature": 23.5,
"humidity": 65.2,
"device_id": "sensor-001"
}JSON batch
[
{"time": 1706745600000000, "temperature": 23.5},
{"time": 1706745601000000, "temperature": 23.6},
{"time": 1706745602000000, "temperature": 23.4}
]MessagePack row-based
Same structure as JSON, but MessagePack encoded. Detected via magic bytes.
MessagePack columnar (fastest)
{
"m": "temperature",
"columns": {
"time": [1706745600000000, 1706745601000000],
"value": [23.5, 23.6],
"device_id": ["sensor-001", "sensor-001"]
}
}Performance: MessagePack columnar format sustains high ingest throughput.
Timestamp handling
- If
timefield is present: used as-is (auto-detects milliseconds/microseconds/nanoseconds) - If
timefield is missing: current UTC time is used
Measurement, tags, and fields
Arc derives the measurement, tags, and fields from the message payload, unless a topic mapping rule (v27.01.1+) for the topic sets the measurement or takes tags from topic levels.
For each decoded message:
- Measurement — taken from the payload's
mfield, ormeasurementfield. If neither is present, it defaults tomqtt. - Tags — taken from a
tagsobject in the payload (string values). - Fields — taken from a
fieldsobject if present; otherwise every remaining top-level key (anything other thanm/measurement,t/time/timestamp,tags,fields) is treated as a field. - Timestamp — from
t,time, ortimestamp(auto-detects ms/µs/ns); current UTC time if absent.
So to land in measurement machine_metrics with tags line and machine_id, publish a payload like:
{
"m": "machine_metrics",
"time": 1706745600000000,
"tags": { "line": "A", "machine_id": "42" },
"fields": { "temperature": 71.5, "rpm": 1480 }
}A flat payload with no m/tags/fields (e.g. {"time": ..., "temperature": 23.5}) is also accepted: it lands in the default mqtt measurement with the remaining keys as fields and no tags.
Without a rule, publishers choose the measurement
Unless a topic_mapping rule sets measurement for the topic, the measurement comes from the payload, so anyone who can publish to a subscribed topic can write to any measurement in the subscription's database. See Trust model.
Topic mapping
topic_mapping sets, per topic, where a message is written and how it is
named. Each key is an MQTT topic filter; each value is either a database name
(the original form) or, from v27.01.1, a rule object
(arc#479).
26.09.x and earlier: exact topics, database names only
In 26.09.x and earlier, each key is matched against the published topic as an
exact string and each value is a database name. Wildcard keys, rule objects,
measurement and tags_from_topic, and the matching and tag-position rules
in this section need v27.01.1 or later. The database-name rule under
Validation already applies.
{
"name": "factory-sensors",
"broker": "tcp://localhost:1883",
"topics": ["factory/#"],
"database": "iot",
"topic_mapping": {
"factory/+/+/metrics": {
"measurement": "machine_metrics",
"tags_from_topic": [
{"position": 1, "tag_name": "line"},
{"position": 2, "tag_name": "machine_id"}
]
},
"factory/line9/status": "iot_line9"
}
}A message on factory/line1/m7/metrics is written to iot.machine_metrics
with the tags line=line1 and machine_id=m7. A message on
factory/line9/status goes to the iot_line9 database, with its measurement
and tags taken from the payload. Every other message uses the subscription's
database and the payload.
Rule fields
| Field | Type | Description |
|---|---|---|
database | string | Database for matching messages, instead of the subscription's database. A string value for the key is shorthand for a rule with only this field. |
measurement | string | Measurement for matching messages, instead of the payload's m / measurement. |
tags_from_topic | array | Tags taken from topic levels: each item needs both position and tag_name. |
A rule sets at least one field. Whatever it leaves out still comes from the payload. A rule with an unknown key is refused, so a misspelt field cannot leave a rule silently partial.
Matching
+matches exactly one topic level and#, only as the last level, any number of levels, including none (factory/#matchesfactory).- An exact key wins. Among filters that match, the one with more literal
levels wins, then the one with more
+, then the one with fewer#. So forfactory/line9/m1/metrics,factory/+/+/metricswins overfactory/line9/#: both have two literal levels and the first has more+. - A filter starting with
+or#does not match topics starting with$(such as$SYS/...), as in MQTT. Name them explicitly:$SYS/#. - Filters match the topic a message was published to. A shared subscription
in
topics($share/group/sensors/#) still delivers topics likesensors/a, and that is what the mapping sees.
Tag positions
position counts the levels of the full topic from 0: in
factory/line1/m7/metrics, factory is 0 and m7 is 2.
- On a filter without a trailing
#, a matching topic has exactly the filter's levels, so a position past the last one is refused when the subscription is created or updated. - Under a trailing
#a topic can be shorter: a message without that level is written without the tag. - A topic tag replaces a payload tag of the same name, in any letter case. A
payload field of exactly that name is kept, stored as
<name>_value. A field that differs only in letter case, or a column already stored under another spelling, clashes as any two such names do: one is dropped, with a warning.
Validation
Every target follows the same rule as database: it starts with a letter and
contains only letters, digits, underscores and hyphens, up to 64 characters.
A rule's measurement follows the HTTP write path's rule (letter first, then
letters, digits, underscore or hyphen, up to 128 characters), and a
tag_name cannot be time in any letter case or repeat another tag name of
the rule. Create and update requests that break any of these, or whose key is
not a valid MQTT filter, are refused with 400 and a message naming the
problem.
Upgrading from releases before v27.01.1
Keys used to be matched as exact strings, and a published topic never
contains + or #. A stored key with a wildcard, such as
"sensors/+/temperature": "temperature", never matched before and now routes
every topic it matches. Check stored mappings before upgrading.
A stored key that is not a valid filter (for example sensors/temp+x, or one
longer than 1024 characters) matches nothing: the subscription keeps running
and logs a warning naming the key. An update that sets topic_mapping must
use valid filters, so the first edit of the mapping removes it ({} clears
it). Launchpad's subscription editor currently sends the whole mapping on
every edit, so edit such a subscription through the API.
An earlier release cannot read a rule object, and with one stored it fails to list or auto-start any MQTT subscription. Before downgrading, change every rule object back to a database string or remove it.
Trust model
Arc does not authenticate MQTT publishers. It connects to the broker as one client, with the credentials of the subscription, and writes what arrives on the subscribed topics. Three things decide what a message can write:
- The subscription is the authorization. Only an admin token can create, change or delete a subscription. A subscription carries no token, team or tenant: Arc's RBAC applies to the people and tokens that manage subscriptions and query data, not to the devices that publish.
- The subscription's target decides where messages land. The
subscription's
database, or atopic_mappingrule'sdatabase, is the only database a message can reach. Within it, the measurement and tags come from the payload unless a rule setsmeasurementortags_from_topic. - The broker decides who may publish. Anyone the broker lets publish to a subscribed topic can write through Arc. Restrict publishing with the broker's ACLs.
So, to keep one measurement per topic, pin it with a topic_mapping rule (v27.01.1+) and
limit who can publish to that topic at the broker. To keep tenants apart, give
each tenant its own subscription and database, and its own topics with broker
ACLs.
Authentication
Basic authentication
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{
"name": "authenticated-broker",
"broker": "tcp://broker.example.com:1883",
"topics": ["data/#"],
"database": "production",
"username": "mqtt_user",
"password": "mqtt_password"
}'Password encryption
Passwords are encrypted at rest using AES-256-GCM. Set the encryption key:
# Generate a 32-byte key
openssl rand -base64 32
# Set environment variable before starting Arc
export ARC_ENCRYPTION_KEY="your-base64-encoded-32-byte-key"When the encryption key is required
The encryption key is only required when subscriptions have passwords. Subscriptions without credentials work without the key.
TLS/SSL configuration
Server certificate verification
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{
"name": "secure-broker",
"broker": "ssl://broker.example.com:8883",
"topics": ["secure/#"],
"database": "production",
"tls_enabled": true,
"tls_ca_path": "/etc/arc/certs/ca.crt"
}'Client certificate authentication (mTLS)
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{
"name": "mtls-broker",
"broker": "ssl://broker.example.com:8883",
"topics": ["secure/#"],
"database": "production",
"tls_enabled": true,
"tls_cert_path": "/etc/arc/certs/client.crt",
"tls_key_path": "/etc/arc/certs/client.key",
"tls_ca_path": "/etc/arc/certs/ca.crt"
}'Configuration examples
Each subscription below is created stopped. Start it with POST /api/v1/mqtt/subscriptions/<id>/start after creating it.
IoT sensor network
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{
"name": "iot-sensors",
"broker": "tcp://mosquitto:1883",
"topics": [
"sensors/+/temperature",
"sensors/+/humidity",
"sensors/+/pressure"
],
"database": "iot",
"qos": 1
}'Devices set the measurement and tags in the payload — e.g. a temperature sensor publishes:
{ "m": "temperature", "tags": { "sensor_id": "temp-001" }, "fields": { "value": 23.5 } }Industrial factory
curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ARC_TOKEN" \
-d '{
"name": "factory-floor",
"broker": "tcp://factory-mqtt:1883",
"topics": ["factory/+/+/metrics"],
"database": "manufacturing",
"qos": 2,
"topic_mapping": {
"factory/+/+/metrics": {
"measurement": "machine_metrics",
"tags_from_topic": [
{"position": 1, "tag_name": "line"},
{"position": 2, "tag_name": "machine_id"}
]
}
}
}'Machines publish only their readings; the measurement and the line and
machine_id tags come from the topic. The rule object in topic_mapping
needs v27.01.1 or later; 26.09.x refuses it with 400 Invalid request body:
{ "fields": { "temperature": 71.5, "rpm": 1480 } }Monitoring
Subscription stats
# Stats for a specific subscription
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/mqtt/subscriptions/{id}/stats
# Aggregate stats for all subscriptions
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/mqtt/statsResponse:
{
"success": true,
"stats": {
"subscriptions": [
{
"id": "056fbfda-eab0-4358-b139-42c539896a1f",
"name": "factory-sensors",
"status": "running",
"messages_received": 15420,
"messages_failed": 0,
"bytes_received": 2458320,
"last_message_at": "2026-02-01T10:30:15Z",
"connected_since": "2026-02-01T10:00:05Z",
"reconnects": 0
}
],
"summary": {
"total_subscriptions": 1,
"running": 1,
"stopped": 0,
"error": 0,
"total_messages": 15420,
"total_failed": 0,
"total_bytes": 2458320
}
}
}GET /api/v1/mqtt/subscriptions/{id}/stats returns one of these objects as stats, without the summary.
Health check
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/mqtt/healthResponse:
{
"status": "healthy",
"healthy": true,
"subscriptions": {
"total": 1,
"running": 1,
"stopped": 0,
"errors": 0
}
}status is degraded when any subscription is in error, and unhealthy when subscriptions exist but none is running, for example right after you create one and before you start it.
Prometheus metrics
Arc exposes MQTT metrics for Prometheus:
| Metric | Type | Description |
|---|---|---|
arc_mqtt_messages_received_total | Counter | Total messages received |
arc_mqtt_bytes_received_total | Counter | Total bytes received |
arc_mqtt_decode_errors_total | Counter | Message decode errors |
arc_mqtt_connection_status | Gauge | Connection status (1=connected) |
Querying MQTT data
Basic query
SELECT * FROM iot.temperature
ORDER BY time DESC
LIMIT 10;Time-based aggregation
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
AVG(value) as avg_temp,
MIN(value) as min_temp,
MAX(value) as max_temp
FROM iot.temperature
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;Filter by tag
SELECT * FROM iot.sensor_data
WHERE sensor_id = 'temp-001'
AND time > NOW() - INTERVAL '24 hours'
ORDER BY time DESC;Troubleshooting
Connection failed
Check subscription status:
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/mqtt/subscriptions/{id}If status is error, verify:
- Broker URL is correct (tcp://, ssl://, ws://)
- Broker is reachable from Arc server
- Credentials are correct
- TLS certificates are valid
No data appearing
- Verify subscription is running (status should be
"running"). A new subscription staysstoppeduntil you callPOST /api/v1/mqtt/subscriptions/{id}/start - If you are querying a measurement named after the topic, check the
mqttmeasurement: without anmormeasurementkey in the payload, messages land there - Check stats for received messages
- Verify topic pattern matches published topics
- Check Arc logs for decode errors
Messages not decoding
Ensure messages are valid JSON or MessagePack:
# Test with simple JSON
mosquitto_pub -h localhost -t "test/data" \
-m '{"time": 1706745600000000, "value": 42}'Check for decode errors in stats:
curl -H "Authorization: Bearer $ARC_TOKEN" \
http://localhost:8000/api/v1/mqtt/subscriptions/{id}/statsBest practices
-
Use descriptive subscription names — Names like
prod-factory-floor-sensorsare easier to manage thansub1. -
Separate databases by environment — Use different target databases for production, staging, and development data.
-
Use QoS appropriately — QoS 0 for highest throughput, QoS 1 for reliable delivery (recommended), QoS 2 for exactly-once (highest overhead).
-
Set reasonable reconnect intervals — Default min 1s / max 60s works well. Avoid setting min too low to prevent broker overload.
-
Use topic wildcards efficiently — Subscribe to specific patterns (
sensors/+/temperature) rather than overly broad ones (#). -
Monitor subscription health — Set up alerts on
arc_mqtt_connection_status == 0andrate(arc_mqtt_decode_errors_total[5m]) > 0.
Docker Compose example
version: '3.8'
services:
arc:
image: ghcr.io/basekick-labs/arc:latest
ports:
- "8000:8000"
volumes:
- arc-data:/app/data
environment:
ARC_MQTT_ENABLED: "true"
ARC_AUTH_ENABLED: "true"
ARC_ENCRYPTION_KEY: "${ARC_ENCRYPTION_KEY}"
depends_on:
- mosquitto
mosquitto:
image: eclipse-mosquitto:2
ports:
- "1883:1883"
volumes:
- ./mosquitto.conf:/mosquitto/config/mosquitto.conf
volumes:
arc-data:Next steps
- Tiered Storage — Manage MQTT data lifecycle with hot/cold tiering
- Automated Scheduling — Downsample MQTT data automatically
- Audit Logging — Track MQTT subscription changes