BasekickLabs

MQTT Integration

Subscribe an Arc Enterprise cluster to MQTT brokers through the API, start and stop subscriptions on a node without restarting it, and from v27.01.1 map topics to measurements and tags.

Ingest data directly from MQTT brokers into Arc. Connect to IoT devices, industrial sensors, and message brokers without middleware.

Overview

Arc provides native MQTT subscription with dynamic, API-driven configuration. Manage multiple MQTT brokers and subscriptions at runtime without server restarts.

Key features:

  • API-driven subscription management - Create, update, delete, start/stop subscriptions via REST API
  • Multiple simultaneous brokers - Connect to different MQTT brokers for different data sources
  • Topic wildcards - Subscribe using + (single level) and # (multi-level) wildcards
  • Auto-detection - Automatically detects JSON and MessagePack message formats
  • High performance - MessagePack columnar format for high-throughput ingestion
  • Topic mapping - Route a topic to another database; from v27.01.1, also set the measurement and extract tags from topic path segments
  • TLS/SSL support - Client certificates and CA verification
  • Encrypted credentials - Passwords encrypted at rest using AES-256-GCM
  • Auto-reconnect - Exponential backoff on connection loss
  • QoS support - QoS 0, 1, and 2

Prerequisites

  • Arc server running (v26.02.1 or higher)
  • Arc API token (if authentication is enabled)
  • MQTT broker accessible from Arc server

Quick start

1. Enable MQTT in Arc

[mqtt]
enabled = true

Or via environment variable:

ARC_MQTT_ENABLED=true

Restart Arc to apply the configuration.

2. Create a subscription

curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -d '{
    "name": "factory-sensors",
    "broker": "tcp://localhost:1883",
    "topics": ["sensors/#"],
    "database": "iot",
    "auto_start": true
  }'

Response:

{
  "success": true,
  "subscription": {
    "id": "056fbfda-eab0-4358-b139-42c539896a1f",
    "name": "factory-sensors",
    "broker": "tcp://localhost:1883",
    "client_id": "arc-e5ccee1d",
    "topics": ["sensors/#"],
    "qos": 1,
    "database": "iot",
    "has_password": false,
    "tls_enabled": false,
    "tls_insecure_skip_verify": false,
    "auto_start": true,
    "status": "stopped",
    "keep_alive_seconds": 60,
    "connect_timeout_seconds": 30,
    "reconnect_max_seconds": 60,
    "clean_session": false,
    "created_at": "2026-02-13T10:00:00Z",
    "updated_at": "2026-02-13T10:00:00Z"
  }
}

3. Start the subscription

Up to Arc 26.09.x, a new subscription is created stopped, even with "auto_start": true: auto_start only makes Arc start the subscription each time the server starts. From 27.01.1, "auto_start": true also starts it on creation, and the response shows "status": "running", or "error" with the reason in error_message if it cannot connect (the subscription is still created). Start it, replacing <id> with the id from the response. Subscriptions are stored on the node that created them and MQTT API calls are not forwarded between nodes, so send this request to that node directly, not through a load balancer:

curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions/<id>/start \
  -H "Authorization: Bearer $ARC_TOKEN"

Response:

{"success": true, "message": "Subscription started"}

GET /api/v1/mqtt/subscriptions/{id} now reports "status": "running".

4. Send test data

Publish a message to your MQTT broker. The m key names the measurement; a payload without m or measurement is written to the mqtt measurement (see Measurement, tags, and fields):

mosquitto_pub -h localhost -t "sensors/temperature" \
  -m '{"m": "temperature", "time": 1706745600000000, "value": 23.5, "device_id": "sensor-001"}'

5. Query the data

curl -X POST http://localhost:8000/api/v1/query \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "sql": "SELECT * FROM iot.temperature ORDER BY time DESC LIMIT 10",
    "format": "json"
  }'

API reference

Subscription management

MethodEndpointDescription
POST/api/v1/mqtt/subscriptionsCreate a new subscription
GET/api/v1/mqtt/subscriptionsList all subscriptions
GET/api/v1/mqtt/subscriptions/{id}Get subscription details
PUT/api/v1/mqtt/subscriptions/{id}Update subscription
DELETE/api/v1/mqtt/subscriptions/{id}Delete subscription

Lifecycle control

MethodEndpointDescription
POST/api/v1/mqtt/subscriptions/{id}/startStart subscription
POST/api/v1/mqtt/subscriptions/{id}/stopStop subscription
POST/api/v1/mqtt/subscriptions/{id}/restartRestart subscription

Monitoring

MethodEndpointDescription
GET/api/v1/mqtt/subscriptions/{id}/statsGet subscription stats
GET/api/v1/mqtt/statsAggregate stats (all subscriptions)
GET/api/v1/mqtt/healthHealth check

Subscription options

Create subscription request

{
  "name": "factory-sensors",
  "broker": "tcp://localhost:1883",
  "topics": ["sensors/#", "factory/+/metrics"],
  "database": "iot",
  "qos": 1,
  "client_id": "arc-factory",
  "username": "mqtt_user",
  "password": "mqtt_pass",
  "tls_enabled": false,
  "tls_cert_path": "/path/to/client.crt",
  "tls_key_path": "/path/to/client.key",
  "tls_ca_path": "/path/to/ca.crt",
  "topic_mapping": {},
  "keep_alive_seconds": 60,
  "connect_timeout_seconds": 30,
  "reconnect_max_seconds": 60,
  "auto_start": true
}

Field reference

FieldTypeRequiredDefaultDescription
namestringYes-Unique subscription name
brokerstringYes-MQTT broker URL (tcp://, ssl://, ws://, wss://)
topicsarrayYes-List of topics to subscribe
databasestringYes-Target Arc database
qosintNo1QoS level: 0, 1, or 2
client_idstringNoautoMQTT client ID
usernamestringNo-MQTT username
passwordstringNo-MQTT password (encrypted at rest)
tls_enabledboolNofalseEnable TLS/SSL
tls_cert_pathstringNo-Client certificate path
tls_key_pathstringNo-Client key path
tls_ca_pathstringNo-CA certificate path
topic_mappingobjectNoIn 26.09.x and earlier: exact topic to database name. From v27.01.1: topic filter (+, #) to a database name or a rule setting database, measurement and tags_from_topic. See Topic mapping.
keep_alive_secondsintNo60MQTT keep-alive interval
connect_timeout_secondsintNo30Connection timeout
reconnect_max_secondsintNo60Maximum reconnect backoff delay. The reconnect delay starts at 1 second and doubles up to this cap (the 1-second minimum is fixed by the MQTT client library and is not configurable).
auto_startboolNofalseStart the subscription whenever Arc starts. Up to 26.09.x, creating a subscription does not start it: call POST /api/v1/mqtt/subscriptions/{id}/start. From 27.01.1 it also starts on creation.

Message formats

Arc automatically detects the message format based on content.

JSON single record

{
  "time": 1706745600000000,
  "temperature": 23.5,
  "humidity": 65.2,
  "device_id": "sensor-001"
}

JSON batch

[
  {"time": 1706745600000000, "temperature": 23.5},
  {"time": 1706745601000000, "temperature": 23.6},
  {"time": 1706745602000000, "temperature": 23.4}
]

MessagePack row-based

Same structure as JSON, but MessagePack encoded. Detected via magic bytes.

MessagePack columnar (fastest)

{
  "m": "temperature",
  "columns": {
    "time": [1706745600000000, 1706745601000000],
    "value": [23.5, 23.6],
    "device_id": ["sensor-001", "sensor-001"]
  }
}

Performance: MessagePack columnar format sustains high ingest throughput.

Timestamp handling

  • If time field is present: used as-is (auto-detects milliseconds/microseconds/nanoseconds)
  • If time field is missing: current UTC time is used

Measurement, tags, and fields

Arc derives the measurement, tags, and fields from the message payload, unless a topic mapping rule (v27.01.1+) for the topic sets the measurement or takes tags from topic levels.

For each decoded message:

  • Measurement — taken from the payload's m field, or measurement field. If neither is present, it defaults to mqtt.
  • Tags — taken from a tags object in the payload (string values).
  • Fields — taken from a fields object if present; otherwise every remaining top-level key (anything other than m/measurement, t/time/timestamp, tags, fields) is treated as a field.
  • Timestamp — from t, time, or timestamp (auto-detects ms/µs/ns); current UTC time if absent.

So to land in measurement machine_metrics with tags line and machine_id, publish a payload like:

{
  "m": "machine_metrics",
  "time": 1706745600000000,
  "tags": { "line": "A", "machine_id": "42" },
  "fields": { "temperature": 71.5, "rpm": 1480 }
}

A flat payload with no m/tags/fields (e.g. {"time": ..., "temperature": 23.5}) is also accepted: it lands in the default mqtt measurement with the remaining keys as fields and no tags.

Without a rule, publishers choose the measurement

Unless a topic_mapping rule sets measurement for the topic, the measurement comes from the payload, so anyone who can publish to a subscribed topic can write to any measurement in the subscription's database. See Trust model.

Topic mapping

topic_mapping sets, per topic, where a message is written and how it is named. Each key is an MQTT topic filter; each value is either a database name (the original form) or, from v27.01.1, a rule object (arc#479).

26.09.x and earlier: exact topics, database names only

In 26.09.x and earlier, each key is matched against the published topic as an exact string and each value is a database name. Wildcard keys, rule objects, measurement and tags_from_topic, and the matching and tag-position rules in this section need v27.01.1 or later. The database-name rule under Validation already applies.

{
  "name": "factory-sensors",
  "broker": "tcp://localhost:1883",
  "topics": ["factory/#"],
  "database": "iot",
  "topic_mapping": {
    "factory/+/+/metrics": {
      "measurement": "machine_metrics",
      "tags_from_topic": [
        {"position": 1, "tag_name": "line"},
        {"position": 2, "tag_name": "machine_id"}
      ]
    },
    "factory/line9/status": "iot_line9"
  }
}

A message on factory/line1/m7/metrics is written to iot.machine_metrics with the tags line=line1 and machine_id=m7. A message on factory/line9/status goes to the iot_line9 database, with its measurement and tags taken from the payload. Every other message uses the subscription's database and the payload.

Rule fields

FieldTypeDescription
databasestringDatabase for matching messages, instead of the subscription's database. A string value for the key is shorthand for a rule with only this field.
measurementstringMeasurement for matching messages, instead of the payload's m / measurement.
tags_from_topicarrayTags taken from topic levels: each item needs both position and tag_name.

A rule sets at least one field. Whatever it leaves out still comes from the payload. A rule with an unknown key is refused, so a misspelt field cannot leave a rule silently partial.

Matching

  • + matches exactly one topic level and #, only as the last level, any number of levels, including none (factory/# matches factory).
  • An exact key wins. Among filters that match, the one with more literal levels wins, then the one with more +, then the one with fewer #. So for factory/line9/m1/metrics, factory/+/+/metrics wins over factory/line9/#: both have two literal levels and the first has more +.
  • A filter starting with + or # does not match topics starting with $ (such as $SYS/...), as in MQTT. Name them explicitly: $SYS/#.
  • Filters match the topic a message was published to. A shared subscription in topics ($share/group/sensors/#) still delivers topics like sensors/a, and that is what the mapping sees.

Tag positions

position counts the levels of the full topic from 0: in factory/line1/m7/metrics, factory is 0 and m7 is 2.

  • On a filter without a trailing #, a matching topic has exactly the filter's levels, so a position past the last one is refused when the subscription is created or updated.
  • Under a trailing # a topic can be shorter: a message without that level is written without the tag.
  • A topic tag replaces a payload tag of the same name, in any letter case. A payload field of exactly that name is kept, stored as <name>_value. A field that differs only in letter case, or a column already stored under another spelling, clashes as any two such names do: one is dropped, with a warning.

Validation

Every target follows the same rule as database: it starts with a letter and contains only letters, digits, underscores and hyphens, up to 64 characters. A rule's measurement follows the HTTP write path's rule (letter first, then letters, digits, underscore or hyphen, up to 128 characters), and a tag_name cannot be time in any letter case or repeat another tag name of the rule. Create and update requests that break any of these, or whose key is not a valid MQTT filter, are refused with 400 and a message naming the problem.

Upgrading from releases before v27.01.1

Keys used to be matched as exact strings, and a published topic never contains + or #. A stored key with a wildcard, such as "sensors/+/temperature": "temperature", never matched before and now routes every topic it matches. Check stored mappings before upgrading.

A stored key that is not a valid filter (for example sensors/temp+x, or one longer than 1024 characters) matches nothing: the subscription keeps running and logs a warning naming the key. An update that sets topic_mapping must use valid filters, so the first edit of the mapping removes it ({} clears it). Launchpad's subscription editor currently sends the whole mapping on every edit, so edit such a subscription through the API.

An earlier release cannot read a rule object, and with one stored it fails to list or auto-start any MQTT subscription. Before downgrading, change every rule object back to a database string or remove it.

Trust model

Arc does not authenticate MQTT publishers. It connects to the broker as one client, with the credentials of the subscription, and writes what arrives on the subscribed topics. Three things decide what a message can write:

  • The subscription is the authorization. Only an admin token can create, change or delete a subscription. A subscription carries no token, team or tenant: Arc's RBAC applies to the people and tokens that manage subscriptions and query data, not to the devices that publish.
  • The subscription's target decides where messages land. The subscription's database, or a topic_mapping rule's database, is the only database a message can reach. Within it, the measurement and tags come from the payload unless a rule sets measurement or tags_from_topic.
  • The broker decides who may publish. Anyone the broker lets publish to a subscribed topic can write through Arc. Restrict publishing with the broker's ACLs.

So, to keep one measurement per topic, pin it with a topic_mapping rule (v27.01.1+) and limit who can publish to that topic at the broker. To keep tenants apart, give each tenant its own subscription and database, and its own topics with broker ACLs.

Authentication

Basic authentication

curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -d '{
    "name": "authenticated-broker",
    "broker": "tcp://broker.example.com:1883",
    "topics": ["data/#"],
    "database": "production",
    "username": "mqtt_user",
    "password": "mqtt_password"
  }'

Password encryption

Passwords are encrypted at rest using AES-256-GCM. Set the encryption key:

# Generate a 32-byte key
openssl rand -base64 32

# Set environment variable before starting Arc
export ARC_ENCRYPTION_KEY="your-base64-encoded-32-byte-key"

When the encryption key is required

The encryption key is only required when subscriptions have passwords. Subscriptions without credentials work without the key.

TLS/SSL configuration

Server certificate verification

curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -d '{
    "name": "secure-broker",
    "broker": "ssl://broker.example.com:8883",
    "topics": ["secure/#"],
    "database": "production",
    "tls_enabled": true,
    "tls_ca_path": "/etc/arc/certs/ca.crt"
  }'

Client certificate authentication (mTLS)

curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -d '{
    "name": "mtls-broker",
    "broker": "ssl://broker.example.com:8883",
    "topics": ["secure/#"],
    "database": "production",
    "tls_enabled": true,
    "tls_cert_path": "/etc/arc/certs/client.crt",
    "tls_key_path": "/etc/arc/certs/client.key",
    "tls_ca_path": "/etc/arc/certs/ca.crt"
  }'

Configuration examples

Each subscription below is created stopped. Start it with POST /api/v1/mqtt/subscriptions/<id>/start after creating it.

IoT sensor network

curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -d '{
    "name": "iot-sensors",
    "broker": "tcp://mosquitto:1883",
    "topics": [
      "sensors/+/temperature",
      "sensors/+/humidity",
      "sensors/+/pressure"
    ],
    "database": "iot",
    "qos": 1
  }'

Devices set the measurement and tags in the payload — e.g. a temperature sensor publishes:

{ "m": "temperature", "tags": { "sensor_id": "temp-001" }, "fields": { "value": 23.5 } }

Industrial factory

curl -X POST http://localhost:8000/api/v1/mqtt/subscriptions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -d '{
    "name": "factory-floor",
    "broker": "tcp://factory-mqtt:1883",
    "topics": ["factory/+/+/metrics"],
    "database": "manufacturing",
    "qos": 2,
    "topic_mapping": {
      "factory/+/+/metrics": {
        "measurement": "machine_metrics",
        "tags_from_topic": [
          {"position": 1, "tag_name": "line"},
          {"position": 2, "tag_name": "machine_id"}
        ]
      }
    }
  }'

Machines publish only their readings; the measurement and the line and machine_id tags come from the topic. The rule object in topic_mapping needs v27.01.1 or later; 26.09.x refuses it with 400 Invalid request body:

{ "fields": { "temperature": 71.5, "rpm": 1480 } }

Monitoring

Subscription stats

# Stats for a specific subscription
curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/mqtt/subscriptions/{id}/stats

# Aggregate stats for all subscriptions
curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/mqtt/stats

Response:

{
  "success": true,
  "stats": {
    "subscriptions": [
      {
        "id": "056fbfda-eab0-4358-b139-42c539896a1f",
        "name": "factory-sensors",
        "status": "running",
        "messages_received": 15420,
        "messages_failed": 0,
        "bytes_received": 2458320,
        "last_message_at": "2026-02-01T10:30:15Z",
        "connected_since": "2026-02-01T10:00:05Z",
        "reconnects": 0
      }
    ],
    "summary": {
      "total_subscriptions": 1,
      "running": 1,
      "stopped": 0,
      "error": 0,
      "total_messages": 15420,
      "total_failed": 0,
      "total_bytes": 2458320
    }
  }
}

GET /api/v1/mqtt/subscriptions/{id}/stats returns one of these objects as stats, without the summary.

Health check

curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/mqtt/health

Response:

{
  "status": "healthy",
  "healthy": true,
  "subscriptions": {
    "total": 1,
    "running": 1,
    "stopped": 0,
    "errors": 0
  }
}

status is degraded when any subscription is in error, and unhealthy when subscriptions exist but none is running, for example right after you create one and before you start it.

Prometheus metrics

Arc exposes MQTT metrics for Prometheus:

MetricTypeDescription
arc_mqtt_messages_received_totalCounterTotal messages received
arc_mqtt_bytes_received_totalCounterTotal bytes received
arc_mqtt_decode_errors_totalCounterMessage decode errors
arc_mqtt_connection_statusGaugeConnection status (1=connected)

Querying MQTT data

Basic query

SELECT * FROM iot.temperature
ORDER BY time DESC
LIMIT 10;

Time-based aggregation

SELECT
  time_bucket(INTERVAL '5 minutes', time) as bucket,
  AVG(value) as avg_temp,
  MIN(value) as min_temp,
  MAX(value) as max_temp
FROM iot.temperature
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket DESC;

Filter by tag

SELECT * FROM iot.sensor_data
WHERE sensor_id = 'temp-001'
  AND time > NOW() - INTERVAL '24 hours'
ORDER BY time DESC;

Troubleshooting

Connection failed

Check subscription status:

curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/mqtt/subscriptions/{id}

If status is error, verify:

  • Broker URL is correct (tcp://, ssl://, ws://)
  • Broker is reachable from Arc server
  • Credentials are correct
  • TLS certificates are valid

No data appearing

  1. Verify subscription is running (status should be "running"). A new subscription stays stopped until you call POST /api/v1/mqtt/subscriptions/{id}/start
  2. If you are querying a measurement named after the topic, check the mqtt measurement: without an m or measurement key in the payload, messages land there
  3. Check stats for received messages
  4. Verify topic pattern matches published topics
  5. Check Arc logs for decode errors

Messages not decoding

Ensure messages are valid JSON or MessagePack:

# Test with simple JSON
mosquitto_pub -h localhost -t "test/data" \
  -m '{"time": 1706745600000000, "value": 42}'

Check for decode errors in stats:

curl -H "Authorization: Bearer $ARC_TOKEN" \
  http://localhost:8000/api/v1/mqtt/subscriptions/{id}/stats

Best practices

  1. Use descriptive subscription names — Names like prod-factory-floor-sensors are easier to manage than sub1.

  2. Separate databases by environment — Use different target databases for production, staging, and development data.

  3. Use QoS appropriately — QoS 0 for highest throughput, QoS 1 for reliable delivery (recommended), QoS 2 for exactly-once (highest overhead).

  4. Set reasonable reconnect intervals — Default min 1s / max 60s works well. Avoid setting min too low to prevent broker overload.

  5. Use topic wildcards efficiently — Subscribe to specific patterns (sensors/+/temperature) rather than overly broad ones (#).

  6. Monitor subscription health — Set up alerts on arc_mqtt_connection_status == 0 and rate(arc_mqtt_decode_errors_total[5m]) > 0.

Docker Compose example

version: '3.8'
services:
  arc:
    image: ghcr.io/basekick-labs/arc:latest
    ports:
      - "8000:8000"
    volumes:
      - arc-data:/app/data
    environment:
      ARC_MQTT_ENABLED: "true"
      ARC_AUTH_ENABLED: "true"
      ARC_ENCRYPTION_KEY: "${ARC_ENCRYPTION_KEY}"
    depends_on:
      - mosquitto

  mosquitto:
    image: eclipse-mosquitto:2
    ports:
      - "1883:1883"
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf

volumes:
  arc-data:

Next steps

On this page