BasekickLabs

Querying with the Python SDK

Query an Arc Enterprise cluster from Python and return JSON, pandas, polars, or PyArrow, including how reader-node routing and query governance limits surface to the client.

How to query data from Arc using the Python SDK.

Overview

The SDK provides multiple query methods, each returning data in a different format:

MethodReturnsBest ForPerformance
query()QueryResponse objectSimple queries, inspectionGood
query_pandas()pandas DataFrameData science, notebooksGood
query_polars()polars DataFrameLarge datasets, performanceBetter
query_arrow()PyArrow TableZero-copy, interopBest

All query methods are available on client.query. Snippets that use client without creating it run inside a client opened as in the full examples below.

SQL syntax

From arc-tsdb-client 0.2.1, a query sends an x-arc-database header only for a database you chose: a call's database= argument, or the database given to ArcClient. Without one, bare names (FROM cpu) read default and qualified names (FROM iot.mem) work. With one, write bare names: Arc rejects database.measurement names while the header is set. Up to 0.2.0 the client always sent the header (default unless set), so only bare names worked. See Qualified names and the database header.

SELECT * FROM cpu WHERE host = 'server01'
# Same query against another database
df = client.query.query_pandas("SELECT * FROM cpu LIMIT 10", database="production")

So SELECT * FROM default.cpu fails with 400. To query more than one database in one statement, create the client with database="", which sends no header, and qualify every name. Continuous query definitions are different: they name their source as database.measurement (FROM default.cpu); see Continuous queries.

Arc runs a full analytical SQL engine, so you have access to:

  • Window functions
  • CTEs (Common Table Expressions)
  • time_bucket() for time-series aggregation
  • JSON functions
  • And more

Basic query (JSON)

The simplest way to query data. Returns a QueryResponse object with columns and data.

from arc_client import ArcClient

with ArcClient(host="localhost", token="your-token") as client:
    result = client.query.query(
        "SELECT * FROM cpu WHERE time > now() - INTERVAL '1 hour' LIMIT 100"
    )

    print(f"Columns: {result.columns}")
    print(f"Row count: {result.row_count}")

    for row in result.data:
        print(row)

QueryResponse object

PropertyTypeDescription
successboolWhether the query executed successfully
columnslist[str]Column names
datalist[list]Rows as nested lists
row_countintRows returned — not a guarantee the full result was delivered
execution_time_msfloatServer-side execution time
timestampstr | NoneServer timestamp for the query
errorstr | NoneError message when the query failed
typeslist[str]Arrow type name per column, populated by query_msgpack(); the JSON path leaves it empty. arc-tsdb-client 0.2.0 and later

Server-side truncation signals: The underlying JSON API response may also contain these fields when a stream fails after returning some rows:

JSON response fieldTypeMeaning
truncatedbool (optional)Present as true when the result is incomplete.
truncation_reasonstr (optional)Explains why streaming stopped.
rows_cappedbool (optional)Present as true when a governance row cap stopped the result. Enterprise only.
row_capint (optional)The cap that was reached, paired with rows_capped.

arc-tsdb-client 0.2.0 and later expose all four fields on QueryResponse:

  • query() returns a truncated result rather than raising, so check truncated before using data, and rows_capped to know a cap stopped it.
  • query_msgpack() raises ArcQueryError on an incomplete body and carries rows_capped and row_cap.
  • query_arrow() raises ArcQueryError on a truncated stream and records the cap in the returned table's schema metadata (arc.rows_capped, arc.row_cap). query_pandas() and query_polars() raise the same way, but the DataFrame they return does not carry the cap; use query_arrow() when a cap may apply.

arc-tsdb-client 0.1.2 exposes none of these fields on QueryResponse, and receiving one, or an expected row_count, does not prove the query completed. With Arc 26.09.2 or later, its query_arrow(), query_pandas() and query_polars() raise ArcQueryError on a stream that failed part way, because the server marks the body so it cannot be read as complete. They cannot see a governance row cap, which arrives only in the Arc-Rows-Capped trailer; check rows_capped in the raw JSON response if a cap may apply.

The two are not interchangeable: truncated means the result failed part way and must be discarded, while rows_capped means it is valid but stopped at a policy limit. Treating a capped result as a failure is as wrong as treating it as complete.

When to use

✅ Use query() when:

  • You need to inspect results quickly
  • Working with small result sets
  • Don't need DataFrame functionality

❌ Don't use when:

  • Processing large datasets (use Arrow)
  • Need DataFrame operations (use pandas/polars)

pandas DataFrame

Returns query results as a pandas DataFrame. Requires pip install arc-tsdb-client[pandas].

from arc_client import ArcClient

with ArcClient(host="localhost", token="your-token") as client:
    df = client.query.query_pandas(
        "SELECT * FROM cpu WHERE host = 'server01' LIMIT 1000"
    )

    print(df.head())
    print(df.dtypes)

    # Use pandas operations
    avg_by_host = df.groupby("host")["usage_idle"].mean()
    print(avg_by_host)

How it works

  1. Query is sent to Arc
  2. Results are returned as Arrow IPC stream
  3. Arrow data is converted to pandas DataFrame (zero-copy where possible)

When to use

✅ Use query_pandas() when:

  • Working in Jupyter notebooks
  • Need pandas-specific operations
  • Integrating with pandas-based tools (matplotlib, seaborn, scikit-learn)

❌ Don't use when:

  • Processing very large datasets (polars is faster)
  • Memory is constrained (polars uses less memory)

Polars DataFrame

Returns query results as a polars DataFrame. Requires pip install arc-tsdb-client[polars].

import polars as pl
from arc_client import ArcClient

with ArcClient(host="localhost", token="your-token") as client:
    df = client.query.query_polars(
        "SELECT * FROM cpu LIMIT 100000"
    )

    print(df.head())

    # Use polars operations (lazy evaluation, parallel execution)
    result = (
        df.lazy()
        .filter(pl.col("usage_idle") > 90)
        .group_by("host")
        .agg(pl.col("usage_idle").mean().alias("avg_idle"))
        .collect()
    )
    print(result)

Why Polars?

Polars is a DataFrame library written in Rust that offers:

  • Faster operations: Especially for large datasets
  • Lower memory usage: Efficient memory layout
  • Lazy evaluation: Optimize query plans before execution
  • Parallel execution: Uses all CPU cores automatically

When to use

✅ Use query_polars() when:

  • Processing large datasets (100K+ rows)
  • Performance is critical
  • You prefer polars' API
  • Running in production pipelines

❌ Don't use when:

  • Need pandas compatibility for downstream tools
  • Working in environments that only support pandas

PyArrow table (zero-copy)

Returns query results as a PyArrow Table. This is the most efficient option for large datasets.

from arc_client import ArcClient

with ArcClient(host="localhost", token="your-token") as client:
    table = client.query.query_arrow(
        "SELECT * FROM cpu LIMIT 1000000"
    )

    print(f"Rows: {table.num_rows}")
    print(f"Columns: {table.num_columns}")
    print(f"Schema: {table.schema}")

    # Convert to pandas (zero-copy where possible)
    df = table.to_pandas()

    # Convert to polars
    import polars as pl
    df = pl.from_arrow(table)

    # Save to Parquet
    import pyarrow.parquet as pq
    pq.write_table(table, "output.parquet")

Why Arrow?

Apache Arrow is a columnar memory format that enables:

  • Zero-copy reads: Data stays in the same memory layout
  • Interoperability: Share data between pandas, polars, DuckDB, Spark
  • Efficient serialization: Arrow IPC format is compact and fast

When to use

✅ Use query_arrow() when:

  • Processing very large datasets
  • Need to pass data to multiple tools
  • Saving results to Parquet files
  • Maximum performance is required

Query estimation

Preview the cost of a query before executing it:

from arc_client import ArcClient

with ArcClient(host="localhost", token="your-token") as client:
    estimate = client.query.estimate(
        "SELECT * FROM cpu WHERE time > now() - INTERVAL '30 days'"
    )

    print(f"Estimated rows: {estimate.estimated_rows}")
    print(f"Warning level: {estimate.warning_level}")  # none, low, medium, high

    if estimate.warning_level == "high":
        print("Consider adding filters or LIMIT clause")

Estimate result

PropertyTypeDescription
estimated_rowsintApproximate row count
warning_levelstrnone, low, medium, high

List measurements

Discover what measurements exist in a database:

from arc_client import ArcClient

with ArcClient(host="localhost", token="your-token") as client:
    measurements = client.query.list_measurements(database="default")

    for m in measurements:
        print(f"{m.measurement}:")
        print(f"  Files: {m.file_count}")
        print(f"  Size: {m.total_size_mb:.1f} MB")

Common query patterns

Time-series aggregation

Use time_bucket() to aggregate data into time intervals:

df = client.query.query_pandas("""
    SELECT
        time_bucket(INTERVAL '5 minutes', time) as bucket,
        host,
        AVG(usage_idle) as avg_idle,
        MAX(usage_system) as max_system,
        COUNT(*) as samples
    FROM cpu
    WHERE time > now() - INTERVAL '1 hour'
    GROUP BY bucket, host
    ORDER BY bucket DESC
""")

Latest value per host

df = client.query.query_pandas("""
    SELECT DISTINCT ON (host)
        time, host, usage_idle, usage_system
    FROM cpu
    ORDER BY host, time DESC
""")

Percentiles

df = client.query.query_pandas("""
    SELECT
        host,
        percentile_cont(0.50) WITHIN GROUP (ORDER BY usage_idle) as p50,
        percentile_cont(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
        percentile_cont(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
    FROM cpu
    WHERE time > now() - INTERVAL '24 hours'
    GROUP BY host
""")

Join measurements

df = client.query.query_pandas("""
    SELECT
        c.time,
        c.host,
        c.usage_idle as cpu_idle,
        m.used_percent as mem_used
    FROM cpu c
    JOIN mem m
        ON c.time = m.time AND c.host = m.host
    WHERE c.time > now() - INTERVAL '10 minutes'
    ORDER BY c.time DESC
""")

Async queries

All query methods have async equivalents:

import asyncio
from arc_client import AsyncArcClient

async def main():
    async with AsyncArcClient(host="localhost", token="your-token") as client:
        # JSON result
        result = await client.query.query("SELECT * FROM cpu LIMIT 10")

        # pandas DataFrame
        df = await client.query.query_pandas("SELECT * FROM cpu LIMIT 1000")

        # Polars DataFrame
        pl_df = await client.query.query_polars("SELECT * FROM cpu LIMIT 1000")

        # Arrow Table
        table = await client.query.query_arrow("SELECT * FROM cpu LIMIT 10000")

asyncio.run(main())

Concurrent queries

Run multiple queries in parallel:

import asyncio
from arc_client import AsyncArcClient

async def main():
    async with AsyncArcClient(host="localhost", token="your-token") as client:
        # Run queries concurrently
        cpu_task = client.query.query_pandas("SELECT * FROM cpu LIMIT 1000")
        mem_task = client.query.query_pandas("SELECT * FROM mem LIMIT 1000")
        disk_task = client.query.query_pandas("SELECT * FROM disk LIMIT 1000")

        cpu_df, mem_df, disk_df = await asyncio.gather(cpu_task, mem_task, disk_task)

        print(f"CPU: {len(cpu_df)} rows")
        print(f"Memory: {len(mem_df)} rows")
        print(f"Disk: {len(disk_df)} rows")

asyncio.run(main())

Error handling

from arc_client import ArcClient
from arc_client.exceptions import (
    ArcQueryError,
    ArcAuthenticationError,
)

with ArcClient(host="localhost", token="your-token") as client:
    try:
        df = client.query.query_pandas("SELECT * FROM nonexistent_measurement")
    except ArcAuthenticationError as e:
        print(f"Auth failed: {e}")  # Invalid token
    except ArcQueryError as e:
        # Invalid SQL, measurement not found, or server unreachable:
        # query methods wrap connection errors in ArcQueryError
        print(f"Query failed: {e}")

Performance tips

1. Filter early

Push filters to Arc rather than filtering in Python:

# ✅ Good: Filter in SQL
df = client.query.query_pandas("""
    SELECT * FROM cpu
    WHERE time > now() - INTERVAL '1 hour'
    AND host = 'server01'
""")

# ❌ Bad: Fetch all, filter in Python
df = client.query.query_pandas("SELECT * FROM cpu")
df = df[df["host"] == "server01"]

2. Select only needed columns

# ✅ Good: Select specific columns
df = client.query.query_pandas("""
    SELECT time, host, usage_idle FROM cpu
""")

# ❌ Bad: Select all columns
df = client.query.query_pandas("SELECT * FROM cpu")

3. Use LIMIT for exploration

# ✅ Good: Limit during exploration
df = client.query.query_pandas("SELECT * FROM cpu LIMIT 100")

4. Use Arrow for large results

# For 100K+ rows, Arrow is significantly faster
table = client.query.query_arrow("SELECT * FROM cpu")
df = table.to_pandas()  # Zero-copy conversion

Next steps

On this page