Data Import

Bulk import is for loading data you already have. For continuous writes, use the ingestion endpoints in the API reference instead.

Imports in a cluster

An import can be sent to any node. A reader, a compactor or a standby writer forwards it to a writer, once, and returns the writer's answer (v27.01.1+). A large import runs as long as it would on the writer directly, up to cluster.import_forward_timeout_ms (default 1 hour). Set your client's timeout for the whole import; for arcli, raise --timeout (default 60 s) for large files.

A client timeout or a 504 does not stop the import on the writer. Check the row count before you run the same file again: a second run writes its rows again. Compaction collapses rows that are exactly equal, timestamp included, so rows without an explicit timestamp in the file stay duplicated.

Memory and large files

An import reads the uploaded file in place in the request and processes it in chunks: about 4 MB of line protocol, or 100,000 CSV or Parquet rows (fewer for wide files). Each chunk is written before the next is read, and the import waits for the flushes to keep up, so it needs little memory beyond the upload itself (v27.01.1+). Before, a 164 MB line protocol file took over 3 GB.

• Validation first. The whole file is read once before anything is written, so a bad file (bad time value, invalid measurement name, missing permission, a CSV parse error, mixed types in a line protocol field) is refused with nothing written. CSV column types are decided by the whole column, as before.
• One import writes at a time on a node. Imports sent together are validated together and written one after the other. Every request body is held in memory while it waits, so send large imports one at a time.
• Failures part-way. If an import stops after part of it was written (the node is fenced, flushes are not keeping up, or a later part of the file cannot be read), it answers 500 with rows_written. Running the whole file again writes those rows again. A failure before the first rows are written (for example a corrupt Parquet page found while writing) answers the plain error, with nothing written.
• Rows are grouped by hour (imports only). An import holds its parsed rows, grouped by hour, up to half its memory allowance, then writes one batch per measurement and hour. A file does not need to be sorted by time: an unsorted file makes about one file per hour per window of that size (one window for most files), where the write endpoints would leave many small files per hour for compaction. CSV and Parquet imports spend some time on it (about 30% on a shuffled year of data). See backfills. Deployments with ingest.decimal_columns set write line-protocol imports chunk by chunk instead, so a large unsorted line-protocol file there makes more, smaller files until compaction merges them.
• No Content-Encoding. A request with a Content-Encoding header is refused with 415. To send a compressed line protocol file, gzip the file itself.