Migration runbook

This is the operator guide for migrating InfluxDB 1.x/2.x data into Arc with tsm2arc, covering the common case: terabytes of InfluxDB data on a cold or unmounted volume that no running influxd serves. For flag reference and install, see the tsm2arc overview; for sizing and throughput, see Performance and scaling.

Before you start

• The InfluxDB data volume is mounted read-only on the migration host (mount -o ro /dev/xvdf1 /mnt/influx). tsm2arc only ever reads the source files, but a read-only mount means a mistake cannot damage them.
• You know the InfluxDB root or data path (1.x: .../influxdb or .../data; 2.x: the v2 root containing engine/ and influxd.bolt).
• You have an admin-tier Arc API token (the import endpoint requires admin) and the network path to the Arc base URL is open.
• You have somewhere durable for the SQLite checkpoint file (tiny, KB to MB).

Understand the source layout

tsm2arc auto-detects 1.x vs 2.x; the TSM/WAL file format is identical between them, only the directory layout differs.

# InfluxDB 1.x
<root>/data/<database>/<retention-policy>/<shard-id>/*.tsm   # compacted data
<root>/wal/<database>/<retention-policy>/<shard-id>/*.wal    # un-flushed data

# InfluxDB 2.x (default root ~/.influxdbv2)
<root>/engine/data/<bucket-id>/autogen/<shard-id>/*.tsm
<root>/engine/wal/<bucket-id>/autogen/<shard-id>/*.wal
<root>/influxd.bolt                                          # bucket id-to-name map

Confirm what you have before running anything:

ls /mnt/influx/data                                  # databases present
find /mnt/influx/data/<db> -name '*.tsm' | head      # shards and TSM files
find /mnt/influx/wal/<db>  -name '*.wal' -size +0c   # non-empty WAL segments

Dry run: The safe first contact

tsm2arc --datadir /mnt/influx/data --waldir /mnt/influx/wal --dry-run --sample 10

Check the output:

• Databases match what you expect. _internal (InfluxDB's own monitoring DB) is skipped by default; pass --include-internal only if you truly want it.
• points / fields / keys are non-zero and roughly the expected magnitude; skipped-keys should be 0 or explainable.
• Time range looks sane. Pre-1970 timestamps are supported and show as negative epoch dates.
• Sample Line Protocol lines look right: measurement names, tags, field types (i integer, u unsigned, quoted strings, booleans).
• INVALID: lines list measurement names Arc would reject, with point counts, so you can author renames before the load (next section).

If an expected database is missing, check whether its data is WAL-only and whether you passed --waldir.

Profile the shards (optional)

--analyze reads only the TSM indexes and finishes in seconds: nothing is decoded, nothing is sent. It prints per-shard series/file/key counts and, for the largest merge runs, whether the files' time ranges partition into windows or overlap fully. Run it when discussing throughput, since the profile determines which optimizations help your data shape (see shard-level parallelism).

tsm2arc --datadir /mnt/influx/data --analyze

Scope the migration (optional)

Migrate a subset first to validate the round-trip end to end:

# one database
tsm2arc --datadir /mnt/influx/data --waldir /mnt/influx/wal --database-filter telemetry --dry-run

# a time window (RFC3339, UTC)
tsm2arc --datadir /mnt/influx/data --waldir /mnt/influx/wal \
        --start 2024-01-01T00:00:00Z --end 2024-02-01T00:00:00Z --dry-run

--start/--end skip out-of-window TSM blocks straight from the index, so a window bounds read work and time, not just output. A large source can be migrated in sequential windows; use a separate --checkpoint file per window (the tool enforces this, refusing to resume a checkpoint whose window changed).

Each source database maps to an Arc database of the same name; rename with --db-map old=new (repeatable).

Handle measurement names Arc rejects

Arc accepts only measurement names matching ^[a-zA-Z][a-zA-Z0-9_-]*$ (the dot is Arc's database.measurement separator in queries and RBAC grant keys). InfluxDB is far more permissive, and dotted <env>.<service> names are common. tsm2arc validates client-side, before sending, so a bad name cannot end a multi-hour load with a mid-flight Arc 400.

1. Dry-run first. Every name Arc would reject is listed with its point count.

2. Author a rename map with deterministic targets you choose:

   # renames.map: one old=new per line; # comments allowed
   edge-prod.gateway_services=edge_prod_gateway_services
   qa.node-b=qa_node_b

   Pass it with --measurement-map-file renames.map (or inline, repeatable, with --measurement-map). Targets are validated at startup, so a typo fails immediately rather than mid-load.

3. Pick the policy for anything still invalid with --on-invalid-measurement:

   • fail (default): abort with an actionable error before sending. Keep this with a map; it guarantees nothing unmapped slips through.
   • skip: drop those points, keep loading, report names and point counts at the end. Use it to land the good data now and deal with stragglers later.
   • map: deterministic auto-rename (disallowed characters become _, an m_ prefix if the name does not start with a letter). Beware that distinct names can collide after sanitizing (a.b and a_b both become a_b) and would merge; prefer an explicit map when names sit close together.

Every rename and skip is recorded in the checkpoint (table measurement_actions: source db, shard, source name, final name, origin, point count) and summarized when the run finishes, so renames are auditable and skipped data is on record rather than quietly missing:

sqlite3 /var/lib/tsm2arc/migration.checkpoint.db \
  'SELECT source_db, measurement, action, renamed_to, origin, SUM(points)
   FROM measurement_actions GROUP BY source_db, measurement'

Run the migration

export ARC_TOKEN='<admin-tier-token>'

tsm2arc \
  --datadir   /mnt/influx/data \
  --waldir    /mnt/influx/wal \
  --arc-url   https://arc.example.net \
  --token     "$ARC_TOKEN" \
  --workers   4 \
  --checkpoint /var/lib/tsm2arc/migration.checkpoint.db \
  --verbose

A heartbeat line reports progress while it runs:

[12/40 shards] 3821 chunks, 18402991 rows, 4210.5 MB raw — 38211 rows/s, 9.4 MB/s (480s)

• Put the --checkpoint file somewhere durable and keep it: it is how resume works, and its audit tables are your record of exactly what was sent.
• Run one process; scale with --workers (size it against the Arc node, see Performance and scaling). Never run two processes against the same checkpoint file.
• Transient failures (429, 5xx, network) retry with exponential backoff; 4xx errors are permanent and abort the run.

If it stops: Just resume

Re-run the exact same command. tsm2arc skips shards already fully migrated, and for a partially migrated shard it seeks to the stored cursor: series before it are never read, already-sent TSM blocks are skipped at the index without being decoded, and sending resumes from the first un-acknowledged chunk within seconds to minutes. Chunk progress commits only after Arc returns 2xx, and Arc's import handler flushes to storage before returning, so 2xx means durably persisted.

Resume requires the same shaping flags: the checkpoint fingerprints --chunk-bytes, --start, --end, --db-map, --precision, the measurement-map flags, and --on-invalid-measurement. Changing any of them would misalign chunk boundaries, so tsm2arc refuses with checkpoint was created with different settings rather than corrupting the migration. To change a shaping flag, start a fresh --checkpoint (a full re-migration).

Duplicates: a clean, uninterrupted run produces zero duplicates. The only duplication window is a crash between Arc persisting a chunk and the tool recording it; on resume that single chunk is re-sent. Arc compaction collapses the duplicate for tag-bearing series automatically; tagless series can retain at most one chunk of duplicate rows per shard per crash (bounded and attributable, the --verbose log names the re-sent chunk).

Verify counts before declaring success

Count reconciliation is how you know the migration is complete and correct.

# Tool side: re-run --dry-run with the SAME flags (including --waldir/--start/--end)
tsm2arc --datadir <same> --waldir <same> --dry-run --sample 0

# Arc side: count rows per measurement, with the same time bounds if you used a window
curl -s -X POST "https://arc.example.net/api/v1/query" \
  -H "Authorization: Bearer $ARC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT count(*) FROM <database>.<measurement>"}'

• Tag-bearing data: after Arc compaction runs, counts should match exactly.
• Tagless data: a small positive delta on Arc's side is resume-overlap duplicates (bounded per the previous section); a clean run shows no delta.
• A short count on Arc's side is the signal to investigate: check the --verbose log for WARN lines, non-zero skipped-keys, or an errored shard, then resume.
• If you loaded with --on-invalid-measurement=skip, subtract the skipped point counts (printed at run end and stored in measurement_actions) before comparing. If you used a rename map, query Arc under the renamed names.

Also confirm WAL coverage if you did not pass --waldir (find <waldir> -name '*.wal' -size +0c; non-empty segments mean un-migrated data, so re-run with --waldir and the same checkpoint), and spot-check a few series' min/max time and sampled values against the dry-run output.

Cleanup

Keep the checkpoint until the migration is verified; delete it only to force a full re-migration. Unmount the read-only source volume, and rotate the Arc admin token if it was placed on a shared host.

Troubleshooting