Querying with the Python SDK

Query Arc from Python: run SQL and receive dictionaries, a pandas DataFrame, a polars DataFrame, or an Arrow table, and stream result sets too large to hold in memory.

How to query data from Arc using the Python SDK.

Overview

The SDK provides multiple query methods, each returning data in a different format:

Method	Returns	Best For	Performance
`query()`	`QueryResponse` object	Simple queries, inspection	Good
`query_pandas()`	pandas DataFrame	Data science, notebooks	Good
`query_polars()`	polars DataFrame	Large datasets, performance	Better
`query_arrow()`	PyArrow Table	Zero-copy, interop	Best

All query methods are available on client.query.

SQL syntax

Arc uses SQL with the table syntax database.measurement:

SELECT * FROM default.cpu WHERE host = 'server01'

Arc runs a full analytical SQL engine, so you have access to:

Window functions
CTEs (Common Table Expressions)
time_bucket() for time-series aggregation
JSON functions
And more

Basic query (JSON)

The simplest way to query data. Returns a QueryResponse object with columns and data.

import os
from arc_client import ArcClient

ARC_TOKEN = os.environ["ARC_TOKEN"]

with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
    result = client.query.query(
        "SELECT * FROM default.cpu WHERE time > now() - INTERVAL '1 hour' LIMIT 100"
    )

    print(f"Columns: {result.columns}")
    print(f"Row count: {result.row_count}")

    for row in result.data:
        print(row)

QueryResponse object

Property	Type	Description
`success`	`bool`	Whether the query executed successfully
`columns`	`list[str]`	Column names
`data`	`list[list]`	Rows as nested lists
`row_count`	`int`	Rows returned — not a guarantee the full result was delivered
`execution_time_ms`	`float`	Server-side execution time
`timestamp`	`str \| None`	Server timestamp for the query
`error`	`str \| None`	Error message when the query failed
`types`	`list[str]`	Arrow type name per column, parallel to `columns`. Populated only by `query_msgpack()`; the JSON path leaves it empty

Server-side truncation signals: The underlying JSON API response may also contain these fields when a stream fails after returning some rows:

JSON response field	Type	Meaning
`truncated`	`bool` (optional)	Present as `true` when the result is incomplete.
`truncation_reason`	`str` (optional)	Explains why streaming stopped.
`rows_capped`	`bool` (optional)	Present as `true` when a governance row cap stopped the result. Enterprise only.
`row_cap`	`int` (optional)	The cap that was reached, paired with `rows_capped`.

The Python SDK's QueryResponse exposes none of these fields yet (tracked in arc #726). Receiving a QueryResponse, or a row_count that matches expectations, does not prove that the query completed. Applications that need strict completeness should read the raw JSON response and check both truncated and rows_capped, or query through the Arrow IPC endpoint, where an incomplete stream fails to decode and a capped one arrives with the Arc-Rows-Capped trailer set.

The two are not interchangeable: truncated means the result failed part way and must be discarded, while rows_capped means it is valid but stopped at a policy limit. Treating a capped result as a failure is as wrong as treating it as complete.

When to use

✅ Use query() when:

You need to inspect results quickly
Working with small result sets
Don't need DataFrame functionality

❌ Don't use when:

Processing large datasets (use Arrow)
Need DataFrame operations (use pandas/polars)

pandas DataFrame

Returns query results as a pandas DataFrame. Requires pip install arc-tsdb-client[pandas].

import os
from arc_client import ArcClient

ARC_TOKEN = os.environ["ARC_TOKEN"]

with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
    df = client.query.query_pandas(
        "SELECT * FROM default.cpu WHERE host = 'server01' LIMIT 1000"
    )

    print(df.head())
    print(df.dtypes)

    # Use pandas operations
    avg_by_host = df.groupby("host")["usage_idle"].mean()
    print(avg_by_host)

How it works

Query is sent to Arc
Results are returned as Arrow IPC stream
Arrow data is converted to pandas DataFrame (zero-copy where possible)

When to use

✅ Use query_pandas() when:

Working in Jupyter notebooks
Need pandas-specific operations
Integrating with pandas-based tools (matplotlib, seaborn, scikit-learn)

❌ Don't use when:

Processing very large datasets (polars is faster)
Memory is constrained (polars uses less memory)

Polars DataFrame

Returns query results as a polars DataFrame. Requires pip install arc-tsdb-client[polars].

import os
from arc_client import ArcClient

ARC_TOKEN = os.environ["ARC_TOKEN"]

with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
    df = client.query.query_polars(
        "SELECT * FROM default.cpu LIMIT 100000"
    )

    print(df.head())

    # Use polars operations (lazy evaluation, parallel execution)
    result = (
        df.lazy()
        .filter(pl.col("usage_idle") > 90)
        .group_by("host")
        .agg(pl.col("usage_idle").mean().alias("avg_idle"))
        .collect()
    )
    print(result)

Why Polars?

Polars is a DataFrame library written in Rust that offers:

Faster operations: Especially for large datasets
Lower memory usage: Efficient memory layout
Lazy evaluation: Optimize query plans before execution
Parallel execution: Uses all CPU cores automatically

When to use

✅ Use query_polars() when:

Processing large datasets (100K+ rows)
Performance is critical
You prefer polars' API
Running in production pipelines

❌ Don't use when:

Need pandas compatibility for downstream tools
Working in environments that only support pandas

PyArrow table (zero-copy)

Returns query results as a PyArrow Table. This is the most efficient option for large datasets.

import os
from arc_client import ArcClient

ARC_TOKEN = os.environ["ARC_TOKEN"]

with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
    table = client.query.query_arrow(
        "SELECT * FROM default.cpu LIMIT 1000000"
    )

    print(f"Rows: {table.num_rows}")
    print(f"Columns: {table.num_columns}")
    print(f"Schema: {table.schema}")

    # Convert to pandas (zero-copy where possible)
    df = table.to_pandas()

    # Convert to polars
    import polars as pl
    df = pl.from_arrow(table)

    # Save to Parquet
    import pyarrow.parquet as pq
    pq.write_table(table, "output.parquet")

Why Arrow?

Apache Arrow is a columnar memory format that enables:

Zero-copy reads: Data stays in the same memory layout
Interoperability: Share data between pandas, polars, DuckDB, Spark
Efficient serialization: Arrow IPC format is compact and fast

When to use

✅ Use query_arrow() when:

Processing very large datasets
Need to pass data to multiple tools
Saving results to Parquet files
Maximum performance is required

Query estimation

Preview the cost of a query before executing it:

import os
from arc_client import ArcClient

ARC_TOKEN = os.environ["ARC_TOKEN"]

with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
    estimate = client.query.estimate(
        "SELECT * FROM default.cpu WHERE time > now() - INTERVAL '30 days'"
    )

    print(f"Estimated rows: {estimate.estimated_rows}")
    print(f"Warning level: {estimate.warning_level}")  # none, low, medium, high

    if estimate.warning_level == "high":
        print("Consider adding filters or LIMIT clause")

Estimate result

Property	Type	Description
`estimated_rows`	`int`	Approximate row count
`warning_level`	`str`	`none`, `low`, `medium`, `high`

List measurements

Discover what measurements exist in a database:

import os
from arc_client import ArcClient

ARC_TOKEN = os.environ["ARC_TOKEN"]

with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
    measurements = client.query.list_measurements(database="default")

    for m in measurements:
        print(f"{m.measurement}:")
        print(f"  Files: {m.file_count}")
        print(f"  Size: {m.total_size_mb:.1f} MB")

Common query patterns

Time-series aggregation

Use time_bucket() to aggregate data into time intervals:

df = client.query.query_pandas("""
    SELECT
        time_bucket(INTERVAL '5 minutes', time) as bucket,
        host,
        AVG(usage_idle) as avg_idle,
        MAX(usage_system) as max_system,
        COUNT(*) as samples
    FROM default.cpu
    WHERE time > now() - INTERVAL '1 hour'
    GROUP BY bucket, host
    ORDER BY bucket DESC
""")

Latest value per host

df = client.query.query_pandas("""
    SELECT DISTINCT ON (host)
        time, host, usage_idle, usage_system
    FROM default.cpu
    ORDER BY host, time DESC
""")

Percentiles

df = client.query.query_pandas("""
    SELECT
        host,
        percentile_cont(0.50) WITHIN GROUP (ORDER BY usage_idle) as p50,
        percentile_cont(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
        percentile_cont(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
    FROM default.cpu
    WHERE time > now() - INTERVAL '24 hours'
    GROUP BY host
""")

Join measurements

df = client.query.query_pandas("""
    SELECT
        c.time,
        c.host,
        c.usage_idle as cpu_idle,
        m.used_percent as mem_used
    FROM default.cpu c
    JOIN default.mem m
        ON c.time = m.time AND c.host = m.host
    WHERE c.time > now() - INTERVAL '10 minutes'
    ORDER BY c.time DESC
""")

Async queries

All query methods have async equivalents:

import os
import asyncio
from arc_client import AsyncArcClient

ARC_TOKEN = os.environ["ARC_TOKEN"]

async def main():
    async with AsyncArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
        # JSON result
        result = await client.query.query("SELECT * FROM default.cpu LIMIT 10")

        # pandas DataFrame
        df = await client.query.query_pandas("SELECT * FROM default.cpu LIMIT 1000")

        # Polars DataFrame
        pl_df = await client.query.query_polars("SELECT * FROM default.cpu LIMIT 1000")

        # Arrow Table
        table = await client.query.query_arrow("SELECT * FROM default.cpu LIMIT 10000")

asyncio.run(main())

Concurrent queries

Run multiple queries in parallel:

import os
import asyncio
from arc_client import AsyncArcClient

ARC_TOKEN = os.environ["ARC_TOKEN"]

async def main():
    async with AsyncArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
        # Run queries concurrently
        cpu_task = client.query.query_pandas("SELECT * FROM default.cpu LIMIT 1000")
        mem_task = client.query.query_pandas("SELECT * FROM default.mem LIMIT 1000")
        disk_task = client.query.query_pandas("SELECT * FROM default.disk LIMIT 1000")

        cpu_df, mem_df, disk_df = await asyncio.gather(cpu_task, mem_task, disk_task)

        print(f"CPU: {len(cpu_df)} rows")
        print(f"Memory: {len(mem_df)} rows")
        print(f"Disk: {len(disk_df)} rows")

asyncio.run(main())

Error handling

import os
from arc_client import ArcClient
from arc_client.exceptions import (

ARC_TOKEN = os.environ["ARC_TOKEN"]

    ArcQueryError,
    ArcConnectionError,
    ArcAuthenticationError,
)

with ArcClient(host="localhost", token=os.environ["ARC_TOKEN"]) as client:
    try:
        df = client.query.query_pandas("SELECT * FROM nonexistent.table")
    except ArcQueryError as e:
        print(f"Query failed: {e}")  # Invalid SQL or table not found
    except ArcAuthenticationError as e:
        print(f"Auth failed: {e}")  # Invalid token
    except ArcConnectionError as e:
        print(f"Connection error: {e}")  # Server unreachable

Performance tips

1. Filter early

Push filters to Arc rather than filtering in Python:

# ✅ Good: Filter in SQL
df = client.query.query_pandas("""
    SELECT * FROM default.cpu
    WHERE time > now() - INTERVAL '1 hour'
    AND host = 'server01'
""")

# ❌ Bad: Fetch all, filter in Python
df = client.query.query_pandas("SELECT * FROM default.cpu")
df = df[df["host"] == "server01"]

2. Select only needed columns

# ✅ Good: Select specific columns
df = client.query.query_pandas("""
    SELECT time, host, usage_idle FROM default.cpu
""")

# ❌ Bad: Select all columns
df = client.query.query_pandas("SELECT * FROM default.cpu")

3. Use LIMIT for exploration

# ✅ Good: Limit during exploration
df = client.query.query_pandas("SELECT * FROM default.cpu LIMIT 100")

4. Use Arrow for large results

# For 100K+ rows, Arrow is significantly faster
table = client.query.query_arrow("SELECT * FROM default.cpu")
df = table.to_pandas()  # Zero-copy conversion

Next steps

Data Management - Retention, CQs, and deletion
Data Ingestion - Write data to Arc
API Reference - Raw REST API documentation

On this page