tsm2arc

Migrate InfluxDB 1.x, 2.x, and 3.x data into Arc by reading the storage directly: no running server, resumable checkpoints, multi-field point reconstruction, S3 sources.

tsm2arc (Apache-2.0) migrates InfluxDB 1.x (1.7/1.8), 2.x (2.0–2.7), and 3.x (3.0–3.11, Parquet engine) data into Arc by reading the storage directly — TSM and WAL files for 1.x/2.x, the parquet object store (including its WAL and catalog) for 3.x. No running influxd/influxdb3 is required, which makes it the right tool when the data sits on cold or unmounted volumes (an EBS snapshot, a decommissioned server's disk) or in an S3 bucket no server is serving.

The layout is auto-detected for every version. For 2.x it resolves bucket IDs to readable names from influxd.bolt and skips InfluxDB's internal system buckets (_monitoring, _tasks). For 3.x it reads all catalog formats natively (including the binary 3.10+/3.11 catalog), recovers un-snapshotted WAL so nothing is left behind, and can read the store in place on S3 — see InfluxDB 3 sources.

When to use which migration path

Live ingestion moves with a URL change (Arc speaks Line Protocol natively; dual-write from Telegraf and cut over). For history: tsm2arc for 1.x/2.x/3.x datasets, especially large ones on cold volumes or in object stores; Line Protocol export/import remains an option for small datasets or the out-of-scope 3.x cases. The InfluxDB migration guide covers the whole journey; these pages are the full tsm2arc reference.

What it does

InfluxDB stores each field of a point as a separate TSM key, each with its own timestamp and value stream. tsm2arc:

Parses the TSM index (header/index/footer) of each file.
Decodes every block with native Go implementations of the TSM codecs (timestamp, float, integer, unsigned, boolean, string), validated against the real InfluxDB encoder and cross-checked against real 2.7 data.
Rejoins fields by (series, timestamp) so multi-field points are reconstructed as single Line Protocol lines. (This is the correctness gap in influx_inspect export, which emits one line per field.)
Streams gzipped, size-bounded chunks into Arc's /api/v1/import/lp endpoint, in parallel across shards, with a SQLite checkpoint for crash-safe resume.

Capabilities at a glance:

Native TSM reader, field rejoin, LP encode, and --dry-run
Chunked gzip POST to Arc /api/v1/import/lp with per-database routing
SQLite checkpoint with crash-safe, cursor-seeking resume
WAL (.wal) reader, merged with TSM per shard
Parallel workers (--workers) with live progress reporting
InfluxDB 2.x layout auto-detection and bucket-name resolution
InfluxDB 3.x object stores, local or in place on S3 — all catalog eras, native WAL recovery, deterministic last-write-wins dedup
Measurement rename map and invalid-name policy with a checkpoint audit trail
Index-only shard profiling (--analyze), shareable with --redact

Install

Download a prebuilt binary from Releases (Linux and macOS on amd64/arm64, Windows on amd64), or:

# from source (Go 1.25+)
go install github.com/basekick-labs/tsm2arc/cmd/tsm2arc@latest

# container (multi-arch, linux amd64/arm64)
docker run --rm ghcr.io/basekick-labs/tsm2arc:latest --version

Each release ships an SBOM (SPDX) and checksums.txt.

Quick start: Dry run first

A dry run discovers shards, decodes every block, reconstructs points, and prints per-database counts plus sample Line Protocol, without writing to Arc. It is the safe first contact with the source data.

# InfluxDB 1.x: point at the data dir or its parent (layout auto-detected)
tsm2arc --datadir /var/lib/influxdb --dry-run --sample 10

# InfluxDB 2.x: point at the v2 root; engine/data, engine/wal, and
# influxd.bolt (for bucket names) are auto-detected
tsm2arc --datadir /var/lib/influxdb2 --dry-run --sample 10

# InfluxDB 3.x: point at the object store root — a local directory or an
# s3:// URL (read in place; credentials from the standard AWS chain)
tsm2arc --datadir /var/lib/influxdb3 --dry-run --sample 10
tsm2arc --datadir s3://my-bucket/influxdb3 --dry-run --sample 10

Then load:

tsm2arc \
  --datadir /mnt/influxdb/data \
  --waldir  /mnt/influxdb/wal \
  --arc-url https://arc.example.net \
  --token   "$ARC_TOKEN" \
  --verbose

Always pass --waldir

InfluxDB does not flush the write-ahead log to TSM on shutdown, so small or recently written shards can live entirely in .wal files. Without --waldir, that data is silently skipped. For 2.x the WAL directory is auto-detected from engine/wal; for 1.x pass it explicitly. When a point exists in both TSM and WAL, the WAL value wins (last-write-wins, matching InfluxDB and Arc compaction).

Where to go next

Migration runbook

The operator walkthrough: layout, dry run, measurement renames, resume, and count verification.

Performance and scaling

Sizing workers against the Arc node, memory math, shard-level parallelism, and multi-writer clusters.

The repo's DESIGN.md covers the internals: the verified Arc ingest constraints, the resume protocol, and the duplicate-bounding argument.

Reusable beyond Arc

The extraction side produces standard InfluxDB Line Protocol; the Arc sink is just the first sink. The TSM/WAL decoder is Apache-2.0 and contributions of new sinks (ClickHouse, QuestDB, TimescaleDB) are welcome. See CONTRIBUTING.md.