Node-RED

Write Node-RED messages to Arc as MessagePack with the arc-write node, query them back with arc-query, and set column types, batching, buffering and retry behaviour.

Write sensor readings, PLC values, MQTT messages or anything else in a Node-RED flow to Arc, and read them back with SQL. The @basekick-labs/node-red-arc package adds three nodes:

Node	What it does
arc-server	The connection: URL, token, default database, optional TLS. Shared by the other two.
arc write	Buffers rows per measurement and sends them to `/api/v1/write/msgpack` in MessagePack columnar batches.
arc query	Runs SQL against `/api/v1/query` and returns the rows.

Every value in a message is a column. There are no tags or fields to declare.

Versions

The package needs Node-RED 4.0 or later (Node.js 18.5 or later). It is tested on Node-RED 4.1.8 and 5.0.9 against Arc 26.09.4 and the 27.01.1 pre-release. Resending after a refused write needs Arc 27.01.1 or later; see Delivery and failures.

Install

In Node-RED, open Menu → Manage palette → Install, search for @basekick-labs/node-red-arc and click Install.

Or install it from the command line in your Node-RED user directory, then restart Node-RED:

cd ~/.node-red
npm install @basekick-labs/node-red-arc

The nodes appear in the storage section of the palette.

Quick start

Create a token

The flow writes and, for the query example, reads. With arcli connected as an admin (see First connection):

arcli auth token create --name node-red --permission read,write \
  --description "Node-RED flows"

arcli prints the secret once. A flow that only writes needs only write.

Configure the server

Drag an arc write node onto the flow, open it, and add a new Server:

URL: http://localhost:8000, or wherever Arc listens. A path prefix is kept, for Arc behind a reverse proxy.
Token: the token from the previous step. ${ARC_TOKEN} reads it from Node-RED's environment instead.
Database: the default database, for example sensors. Arc creates it on the first write.

Click Test connection. It checks /health, then the token, and lists the token's permissions.

Send rows

Wire an inject node to a function node to the arc write node:

msg.measurement = "boiler";
msg.payload = { sensor: "a1", line: 3, temperature: 21.5, ok: true };
return msg;

Set Integer columns on the arc write node to line (see Column types), deploy, and click the inject button. The node's status shows how many rows are buffered; it sends them within one Flush every interval (1 s by default).

Check the data

Arc acknowledges the write before the rows reach Parquet, so wait a few seconds, then query:

curl -s http://localhost:8000/api/v1/query \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -H "x-arc-database: sensors" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT * FROM boiler ORDER BY time DESC LIMIT 5"}'

Or add an arc query node; see Querying.

The package ships an example flow with a simulated sensor, a write, a Catch node for failures and a query. In Node-RED, open Menu → Import → Examples → @basekick-labs/node-red-arc → mqtt-to-arc.

Message formats

msg.payload can be:

Shape	Example
One row	`{ sensor: "a1", temperature: 21.5 }`
An array of rows	`[{ sensor: "a1", temperature: 21.5 }, { sensor: "a2", temperature: 19.0 }]`
`{rows: [...]}`	`{ rows: [{ sensor: "a1", temperature: 21.5 }] }`
`{columns: {...}}`	`{ columns: { sensor: ["a1", "a2"], temperature: [21.5, 19.0] } }`: arrays of equal length
A number	`21.5`: stored in a column named `value`

Other message properties:

Property	Meaning
`msg.measurement`	Overrides the node's measurement. One of the two is required.
`msg.database`	Overrides the node's and the server's database.
`msg.time`	Time for every row of the message that has no `time` of its own.
`msg.precision`	Time unit for this message: `s`, `ms`, `us` or `ns`.
`msg.flush`	`true` sends everything buffered now. A message with `flush` and no payload only flushes.

Time

A row takes its time from its own time key, then msg.time, then the moment it arrived:

Date: used as is.
Number, or a string of digits: in the node's Time unit (milliseconds by default) or msg.precision. In us and ns, plain JavaScript numbers lose precision past 2^53, so they must be safe integers; send nanosecond epochs as a BigInt or a digit string.
ISO 8601 string: 2026-11-10T08:30:00.123456Z. An offset is honoured; a string without one is read as UTC.
Missing: the arrival time. Rows that arrive in the same microsecond get distinct, increasing timestamps.

Arc stores time in microseconds, so nanoseconds are truncated. A time more than 50 years from now in either direction rejects the message, because it is almost always a wrong unit.

Column types

Arc fixes a column's type within a request from its first value, so the node decides types before sending:

Numbers are written as DOUBLE, even 21. A sensor that sends 21 and then 21.7 keeps one numeric column instead of an integer column that truncates the decimals.
Integer columns: list them in Integer columns (count, errors) to store them as BIGINT, or send a BigInt. A fractional or unsafe value for an integer column rejects the whole message rather than being truncated.
Strings and booleans are stored as they are.
null, undefined, NaN and Infinity are an empty cell.
Nested objects and arrays reject the message, unless Store nested objects as JSON strings is checked.
A column that is empty in every row of a batch is left out of that batch, so it does not create a text column.

Each arc write node remembers the type of every column it has sent. A later value of another type (say "N/A" in a numeric column) leaves that one cell empty and keeps the rest of the row, or rejects the message if Type mismatch is set to reject the message. Either way the value is listed in msg.arc.rejectedCells.

Set integer columns before the first write

The node keeps these types in memory, separately for each arc write node, and forgets them on redeploy. Changing Integer columns after data exists leaves the same column stored as DOUBLE in some files and BIGINT in others.

Names

Database: starts with a letter, then letters, digits, _ and -, up to 64 characters. system, internal and _internal are reserved.
Measurement: starts with a letter, then letters, digits, _ and -, up to 128 characters.
Column: must not start with _. Only the time column may be called time, in any letter case.

Arc treats column names that differ only in A–Z letter case as one column. The node keeps the first spelling it sees for a measurement and rejects messages that use another one.

A message with an invalid name fails at once and is never sent, so it cannot hold up other rows. Replace invalid characters in names with _ turns plant/line 3 into plant_line_3 instead.

Delivery and failures

The node calls done() for a message once Arc has answered 204 for every one of its rows, so a Complete node fires when Arc has acknowledged the data, up to one flush interval later. How durable an acknowledged row is depends on Arc: with the write-ahead log off, which is the default, Arc holds rows in memory for up to ingest.max_buffer_age_ms before writing Parquet.

A message that could not be written goes to Catch nodes with its payload untouched and msg.arc describing why: reason, status, database, measurement and written, the number of the message's rows Arc already has.

The node resends a batch only when it knows Arc stored nothing:

Arc's answer	What the node does
`204`	Finishes the messages.
`503` with `X-Arc-Ingest-Fenced`, or `X-Arc-Ingest-Refused` with any reason except `wal_not_durable` (Arc 27.01.1 and later)	Waits `Retry-After`, then resends.
Connection refused, DNS failure	Backs off from 1 s to 30 s and resends.
Connection reset or timeout before the request body was sent	Resends.
`401`	Pauses the node for every measurement, keeps buffering, and checks the token every 60 s. Resumes once it works.
`413`	Splits the batch in two and sends the halves. A single row that is still too large fails.
`400 partial write` (Arc 27.01.1 and later)	Fails the batch. Most of its rows were stored: `msg.arc.partial` is `true` and `msg.arc.dropped` says how many were not.
Anything else: other `400`s, `403`, `500`, `502`, `504`, a reset or timeout after the body was sent, `503` with `wal_not_durable`, any `503` from Arc before 27.01.1	Fails the batch's messages. Never resends.

Arc releases before 27.01.1 do not mark a refused write, so a server-side failure cannot be told apart from one that was stored. On those releases every server-side failure goes to Catch.

Rows are never written twice by the node

The node never sends the same batch twice. A flow that resends failed messages from a Catch node can duplicate rows; use msg.arc.written to skip the rows Arc already has.

With Output one message per batch sent checked, the node gets an output that emits {database, measurement, rows, status, reason, durationMs, action} for every batch, for monitoring.

Batching and buffering

Setting	Default	Meaning
Batch size	5000 rows	Send as soon as a measurement has this many rows.
Flush every	1000 ms	Send whatever is buffered at least this often.
Buffer cap	200000 rows, 64 MB	Total rows held in memory, across measurements.
Pending msgs	100000	Messages waiting for Arc's answer.
In flight	4	Requests sent at once; at most one per measurement.
When full	drop the oldest rows	Or reject new messages. Either way the affected messages fail.

Rows wait in memory while Arc is unreachable or refusing. Size the buffer as rows per second × the longest outage you want to ride out.

When a flow is stopped or redeployed, the node flushes what is buffered within the Node-RED node close timeout. Rows still left after that fail their messages with node closed with N rows still buffered.

Write bodies above 1 KiB are compressed: zstd when the Node.js running Node-RED supports it, gzip otherwise. Set Compression on the server node to force gzip, zstd or none. Arc detects the format from the body.

Querying

Put SQL in an arc query node and trigger it with any message:

SELECT time_bucket(INTERVAL '1 minute', time) AS minute, avg(temperature) AS temp
FROM boiler
WHERE sensor = {{sensor}} AND time > now() - INTERVAL '1 hour'
GROUP BY 1 ORDER BY 1

Arc has no query parameters, so the node fills placeholders itself:

{{name}} takes msg.params.name and writes it as an SQL literal: strings are quoted and escaped, numbers, booleans and null are written as such, a Date becomes a TIMESTAMP, and an array becomes a list for IN {{ids}}.
{{#name}} takes msg.params.name as an identifier, such as a column name. It must match [A-Za-z_][A-Za-z0-9_]*.

A placeholder inside a quoted string or a comment is refused. Do not build SQL by concatenating message values. msg.sql replaces the node's SQL only when Allow msg.sql is checked, because SQL from upstream data is an injection risk.

The database comes from msg.database, then the node, then the server's default. Refer to tables by name (FROM boiler); Arc rejects database.table syntax while a database is selected.

Output	`msg.payload`
one message, array of row objects	`[{time, minute, temp}, ...]`
one message per row	One row object per message, with `msg.parts` for a Join node. No rows sends one message with `payload: null`.
messages of N rows	Arrays of Rows per msg (1000) rows, with `msg.parts`.
one message, `{columns, data}`	Arc's response as is.

time column sets how the time column comes back: an ISO string with microseconds (exact), a Date (milliseconds), or epoch microseconds as a string. Only the column named time is converted. Integers larger than 2^53 come back as strings on Node.js 21 and later, and lose precision on older versions.

msg.arc carries rowCount, executionTimeMs, truncated, rowsCapped, rowCap and columns. A response larger than Max response (64 MB) fails. A 503 is retried once; a 429 from Arc's query limits is not retried.

TLS

Use an https:// URL on the server node. The TLS field then appears: add a tls-config node with the CA that signed Arc's certificate, and a client certificate and key if Arc requires one. Verify server certificate stays on unless you are testing.

Troubleshooting

Test connection fails

Message	Cause
`Cannot reach <url>: ECONNREFUSED`	Arc is not listening at that URL, or a firewall is in the way.
`Connected, but the token was rejected`	The token is wrong, revoked or expired.
`Connected, but the server requires a token`	The token field is empty and Arc has authentication enabled.
`The URL changed since the token was saved`	The stored token is only tested against the URL it was saved with. Type the token again.
`Token permissions: write (missing read)`	Writing works; an arc query node using this server will get `403`.

MQTT: let Arc subscribe to the broker directly when the flow does nothing but forward messages
Querying: the SQL Arc accepts
Grafana: chart what the flow writes
Authentication: token management
Source and issues