Decimal Precision

Arc supports native Decimal128 columns for precision-sensitive use cases such as financial data, scientific measurements, and cryptocurrency trading. Declare decimal columns via per-measurement configuration, and Arc stores them as native Parquet DECIMAL type — preserving exact precision instead of coercing to float64.

Why Decimal128?

IEEE 754 float64 has ~15-17 significant digits of precision. For financial data, this causes silent rounding:

float64(0.1 + 0.2) = 0.30000000000000004  // not 0.3
float64(9007199254740993) = 9007199254740992  // off by 1

Decimal128 provides up to 38 significant digits with exact precision — no rounding, no surprises.

Configuration

Declare which columns should be stored as DECIMAL using the [ingest] configuration:

[ingest]
# Per-measurement decimal columns: "measurement:col=precision,scale;col2=p,s"
decimal_columns = [
  "trades:price=18,8;amount=18,8",
  "balances:balance=38,18"
]

# Default decimal columns for measurements not listed above (optional)
default_decimal_columns = "value=18,6"

Environment variables:

ARC_INGEST_DECIMAL_COLUMNS="trades:price=18,8;amount=18,8 balances:balance=38,18"
ARC_INGEST_DEFAULT_DECIMAL_COLUMNS="value=18,6"

Format

Each entry follows the pattern measurement:column=precision,scale;column2=precision,scale:

Common configurations:

Ingestion

Decimal conversion happens automatically at ingestion time for configured columns. All ingestion paths are supported:

MessagePack (recommended)

Float values — converted via decimal128.FromFloat64:

import msgpack
import urllib.request

payload = {
    "m": "trades",
    "t": 1711152000000000,
    "fields": {
        "price": 123.45678901,      # float64 → DECIMAL(18,8)
        "amount": 999.12345678,     # float64 → DECIMAL(18,8)
        "volume": 42,               # not configured → stays BIGINT
    },
    "tags": {"symbol": "BTC-USD"}
}

data = msgpack.packb(payload)
req = urllib.request.Request(
    "http://localhost:8000/api/v1/write/msgpack",
    data=data,
    headers={
        "Content-Type": "application/msgpack",
        "x-arc-database": "mydb",
    }
)
urllib.request.urlopen(req)

String values — for highest precision, send values as strings over MessagePack. String-to-decimal conversion is exact with no float64 intermediate:

payload = {
    "m": "trades",
    "t": 1711152000000000,
    "fields": {
        "price": "99999.12345678",           # string → exact DECIMAL(18,8)
        "amount": "0.00000001",              # smallest representable at scale 8
    },
    "tags": {"symbol": "ETH-USD"}
}

Line Protocol

Float values in Line Protocol are converted to the configured decimal type:

curl -X POST "http://localhost:8000/write?db=mydb" \
  -d 'trades,symbol=BTC-USD price=123.45678901,amount=999.12345678,volume=42i'

Querying

Arc reads the Parquet DECIMAL type natively — no query changes needed:

SELECT price, amount, typeof(price) as price_type
FROM trades
LIMIT 5

Response:

{
  "columns": ["price", "amount", "price_type"],
  "data": [
    ["123.45678901", "999.12345678", "DECIMAL(18,8)"],
    ["99999.12345678", "0.00000001", "DECIMAL(18,8)"]
  ]
}

All SQL operations work with decimal columns — aggregation, filtering, ordering, window functions:

-- Aggregation preserves decimal precision
SELECT symbol,
       AVG(price) as avg_price,
       SUM(amount) as total_amount,
       COUNT(*) as trade_count
FROM trades
GROUP BY symbol

-- Filtering works as expected
SELECT * FROM trades WHERE price > 50000.00

-- Arithmetic on decimal columns
SELECT price * amount as notional_value FROM trades

How it works

1. Ingestion: Float64, int64, or string values arriving for declared decimal columns are converted to Arrow Decimal128 at buffer time
2. Storage: Stored as Parquet DECIMAL logical type (16 bytes per value, exact precision)
3. Metadata: Decimal column specs are stored as Parquet metadata (arc:decimals) for self-describing files
4. Querying: Arc reads Parquet DECIMAL natively
5. Compaction: DECIMAL types are preserved automatically during compaction

Performance

• Zero overhead when not configured — one empty map lookup per column during ingestion
• Decimal conversion cost: ~100ns per value. A flush batch of 50K rows with 2 decimal columns adds ~10ms (flush runs in background)
• Storage: 16 bytes per value (vs 8 bytes for float64) — 2x for decimal columns only
• Query performance: Identical to float64 — DECIMAL is handled natively

Best practices

1. Use string values for maximum precision — Send values as strings over MessagePack when precision beyond 15 significant digits is required.

2. Choose precision and scale carefully — Use the smallest precision/scale that fits your data. price=18,8 handles up to 10 billion with 8 decimal places. Over-specifying (e.g., 38,18) wastes no storage but may affect sort performance.

3. Use default decimal columns sparingly — Per-measurement config is preferred. Defaults apply to all measurements not explicitly listed, which may cause unexpected type conversions.

4. Non-decimal columns are unaffected — Columns not listed in the config continue to use automatic type detection (float64, int64, string, bool).

5. Backwards compatible — Files written before enabling decimal config have no decimal metadata and are read normally. You can enable decimal config on existing deployments without any migration.

Troubleshooting

Decimal config not taking effect

Check the Arc startup logs for:

WRN Invalid decimal columns config, decimal support disabled error="..."

This means the configuration format is invalid. Verify the format: "measurement:col=precision,scale".

Values still showing as float64

• Verify the column name in config matches exactly (case-sensitive)
• Check that the measurement name matches
• Use typeof(column) in SQL to verify: SELECT typeof(price) FROM trades LIMIT 1

Scientific notation in query results

Arc may serialize very small decimals using scientific notation (e.g., 1e-08 instead of 0.00000001). The underlying precision is preserved — this is a display format choice. Use CAST(column AS VARCHAR) for explicit string formatting.