Querying with the Python SDK
Query an Arc Enterprise cluster from Python and return JSON, pandas, polars, or PyArrow, including how reader-node routing and query governance limits surface to the client.
How to query data from Arc using the Python SDK.
Overview
The SDK provides multiple query methods, each returning data in a different format:
| Method | Returns | Best For | Performance |
|---|---|---|---|
query() | QueryResponse object | Simple queries, inspection | Good |
query_pandas() | pandas DataFrame | Data science, notebooks | Good |
query_polars() | polars DataFrame | Large datasets, performance | Better |
query_arrow() | PyArrow Table | Zero-copy, interop | Best |
All query methods are available on client.query. Snippets that use client without creating it run inside a client opened as in the full examples below.
SQL syntax
From arc-tsdb-client 0.2.1, a query sends an x-arc-database header only for a database you chose: a call's database= argument, or the database given to ArcClient. Without one, bare names (FROM cpu) read default and qualified names (FROM iot.mem) work. With one, write bare names: Arc rejects database.measurement names while the header is set. Up to 0.2.0 the client always sent the header (default unless set), so only bare names worked. See Qualified names and the database header.
SELECT * FROM cpu WHERE host = 'server01'# Same query against another database
df = client.query.query_pandas("SELECT * FROM cpu LIMIT 10", database="production")So SELECT * FROM default.cpu fails with 400. To query more than one database in one statement, create the client with database="", which sends no header, and qualify every name. Continuous query definitions are different: they name their source as database.measurement (FROM default.cpu); see Continuous queries.
Arc runs a full analytical SQL engine, so you have access to:
- Window functions
- CTEs (Common Table Expressions)
time_bucket()for time-series aggregation- JSON functions
- And more
Basic query (JSON)
The simplest way to query data. Returns a QueryResponse object with columns and data.
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
result = client.query.query(
"SELECT * FROM cpu WHERE time > now() - INTERVAL '1 hour' LIMIT 100"
)
print(f"Columns: {result.columns}")
print(f"Row count: {result.row_count}")
for row in result.data:
print(row)QueryResponse object
| Property | Type | Description |
|---|---|---|
success | bool | Whether the query executed successfully |
columns | list[str] | Column names |
data | list[list] | Rows as nested lists |
row_count | int | Rows returned — not a guarantee the full result was delivered |
execution_time_ms | float | Server-side execution time |
timestamp | str | None | Server timestamp for the query |
error | str | None | Error message when the query failed |
types | list[str] | Arrow type name per column, populated by query_msgpack(); the JSON path leaves it empty. arc-tsdb-client 0.2.0 and later |
Server-side truncation signals: The underlying JSON API response may also contain these fields when a stream fails after returning some rows:
| JSON response field | Type | Meaning |
|---|---|---|
truncated | bool (optional) | Present as true when the result is incomplete. |
truncation_reason | str (optional) | Explains why streaming stopped. |
rows_capped | bool (optional) | Present as true when a governance row cap stopped the result. Enterprise only. |
row_cap | int (optional) | The cap that was reached, paired with rows_capped. |
arc-tsdb-client 0.2.0 and later expose all four fields on
QueryResponse:
query()returns a truncated result rather than raising, so checktruncatedbefore usingdata, androws_cappedto know a cap stopped it.query_msgpack()raisesArcQueryErroron an incomplete body and carriesrows_cappedandrow_cap.query_arrow()raisesArcQueryErroron a truncated stream and records the cap in the returned table's schema metadata (arc.rows_capped,arc.row_cap).query_pandas()andquery_polars()raise the same way, but the DataFrame they return does not carry the cap; usequery_arrow()when a cap may apply.
arc-tsdb-client 0.1.2
exposes none of these fields on QueryResponse, and receiving one, or an
expected row_count, does not prove the query completed. With Arc 26.09.2 or
later, its query_arrow(), query_pandas() and query_polars() raise
ArcQueryError on a stream that failed part way, because the server marks the
body so it cannot be read as complete. They cannot see a governance row cap,
which arrives only in the Arc-Rows-Capped trailer; check rows_capped in
the raw JSON response if a cap may apply.
The two are not interchangeable: truncated means the result failed part way
and must be discarded, while rows_capped means it is valid but stopped at a
policy limit. Treating a capped result as a failure is as wrong as treating it
as complete.
When to use
✅ Use query() when:
- You need to inspect results quickly
- Working with small result sets
- Don't need DataFrame functionality
❌ Don't use when:
- Processing large datasets (use Arrow)
- Need DataFrame operations (use pandas/polars)
pandas DataFrame
Returns query results as a pandas DataFrame. Requires pip install arc-tsdb-client[pandas].
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
df = client.query.query_pandas(
"SELECT * FROM cpu WHERE host = 'server01' LIMIT 1000"
)
print(df.head())
print(df.dtypes)
# Use pandas operations
avg_by_host = df.groupby("host")["usage_idle"].mean()
print(avg_by_host)How it works
- Query is sent to Arc
- Results are returned as Arrow IPC stream
- Arrow data is converted to pandas DataFrame (zero-copy where possible)
When to use
✅ Use query_pandas() when:
- Working in Jupyter notebooks
- Need pandas-specific operations
- Integrating with pandas-based tools (matplotlib, seaborn, scikit-learn)
❌ Don't use when:
- Processing very large datasets (polars is faster)
- Memory is constrained (polars uses less memory)
Polars DataFrame
Returns query results as a polars DataFrame. Requires pip install arc-tsdb-client[polars].
import polars as pl
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
df = client.query.query_polars(
"SELECT * FROM cpu LIMIT 100000"
)
print(df.head())
# Use polars operations (lazy evaluation, parallel execution)
result = (
df.lazy()
.filter(pl.col("usage_idle") > 90)
.group_by("host")
.agg(pl.col("usage_idle").mean().alias("avg_idle"))
.collect()
)
print(result)Why Polars?
Polars is a DataFrame library written in Rust that offers:
- Faster operations: Especially for large datasets
- Lower memory usage: Efficient memory layout
- Lazy evaluation: Optimize query plans before execution
- Parallel execution: Uses all CPU cores automatically
When to use
✅ Use query_polars() when:
- Processing large datasets (100K+ rows)
- Performance is critical
- You prefer polars' API
- Running in production pipelines
❌ Don't use when:
- Need pandas compatibility for downstream tools
- Working in environments that only support pandas
PyArrow table (zero-copy)
Returns query results as a PyArrow Table. This is the most efficient option for large datasets.
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
table = client.query.query_arrow(
"SELECT * FROM cpu LIMIT 1000000"
)
print(f"Rows: {table.num_rows}")
print(f"Columns: {table.num_columns}")
print(f"Schema: {table.schema}")
# Convert to pandas (zero-copy where possible)
df = table.to_pandas()
# Convert to polars
import polars as pl
df = pl.from_arrow(table)
# Save to Parquet
import pyarrow.parquet as pq
pq.write_table(table, "output.parquet")Why Arrow?
Apache Arrow is a columnar memory format that enables:
- Zero-copy reads: Data stays in the same memory layout
- Interoperability: Share data between pandas, polars, DuckDB, Spark
- Efficient serialization: Arrow IPC format is compact and fast
When to use
✅ Use query_arrow() when:
- Processing very large datasets
- Need to pass data to multiple tools
- Saving results to Parquet files
- Maximum performance is required
Query estimation
Preview the cost of a query before executing it:
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
estimate = client.query.estimate(
"SELECT * FROM cpu WHERE time > now() - INTERVAL '30 days'"
)
print(f"Estimated rows: {estimate.estimated_rows}")
print(f"Warning level: {estimate.warning_level}") # none, low, medium, high
if estimate.warning_level == "high":
print("Consider adding filters or LIMIT clause")Estimate result
| Property | Type | Description |
|---|---|---|
estimated_rows | int | Approximate row count |
warning_level | str | none, low, medium, high |
List measurements
Discover what measurements exist in a database:
from arc_client import ArcClient
with ArcClient(host="localhost", token="your-token") as client:
measurements = client.query.list_measurements(database="default")
for m in measurements:
print(f"{m.measurement}:")
print(f" Files: {m.file_count}")
print(f" Size: {m.total_size_mb:.1f} MB")Common query patterns
Time-series aggregation
Use time_bucket() to aggregate data into time intervals:
df = client.query.query_pandas("""
SELECT
time_bucket(INTERVAL '5 minutes', time) as bucket,
host,
AVG(usage_idle) as avg_idle,
MAX(usage_system) as max_system,
COUNT(*) as samples
FROM cpu
WHERE time > now() - INTERVAL '1 hour'
GROUP BY bucket, host
ORDER BY bucket DESC
""")Latest value per host
df = client.query.query_pandas("""
SELECT DISTINCT ON (host)
time, host, usage_idle, usage_system
FROM cpu
ORDER BY host, time DESC
""")Percentiles
df = client.query.query_pandas("""
SELECT
host,
percentile_cont(0.50) WITHIN GROUP (ORDER BY usage_idle) as p50,
percentile_cont(0.95) WITHIN GROUP (ORDER BY usage_idle) as p95,
percentile_cont(0.99) WITHIN GROUP (ORDER BY usage_idle) as p99
FROM cpu
WHERE time > now() - INTERVAL '24 hours'
GROUP BY host
""")Join measurements
df = client.query.query_pandas("""
SELECT
c.time,
c.host,
c.usage_idle as cpu_idle,
m.used_percent as mem_used
FROM cpu c
JOIN mem m
ON c.time = m.time AND c.host = m.host
WHERE c.time > now() - INTERVAL '10 minutes'
ORDER BY c.time DESC
""")Async queries
All query methods have async equivalents:
import asyncio
from arc_client import AsyncArcClient
async def main():
async with AsyncArcClient(host="localhost", token="your-token") as client:
# JSON result
result = await client.query.query("SELECT * FROM cpu LIMIT 10")
# pandas DataFrame
df = await client.query.query_pandas("SELECT * FROM cpu LIMIT 1000")
# Polars DataFrame
pl_df = await client.query.query_polars("SELECT * FROM cpu LIMIT 1000")
# Arrow Table
table = await client.query.query_arrow("SELECT * FROM cpu LIMIT 10000")
asyncio.run(main())Concurrent queries
Run multiple queries in parallel:
import asyncio
from arc_client import AsyncArcClient
async def main():
async with AsyncArcClient(host="localhost", token="your-token") as client:
# Run queries concurrently
cpu_task = client.query.query_pandas("SELECT * FROM cpu LIMIT 1000")
mem_task = client.query.query_pandas("SELECT * FROM mem LIMIT 1000")
disk_task = client.query.query_pandas("SELECT * FROM disk LIMIT 1000")
cpu_df, mem_df, disk_df = await asyncio.gather(cpu_task, mem_task, disk_task)
print(f"CPU: {len(cpu_df)} rows")
print(f"Memory: {len(mem_df)} rows")
print(f"Disk: {len(disk_df)} rows")
asyncio.run(main())Error handling
from arc_client import ArcClient
from arc_client.exceptions import (
ArcQueryError,
ArcAuthenticationError,
)
with ArcClient(host="localhost", token="your-token") as client:
try:
df = client.query.query_pandas("SELECT * FROM nonexistent_measurement")
except ArcAuthenticationError as e:
print(f"Auth failed: {e}") # Invalid token
except ArcQueryError as e:
# Invalid SQL, measurement not found, or server unreachable:
# query methods wrap connection errors in ArcQueryError
print(f"Query failed: {e}")Performance tips
1. Filter early
Push filters to Arc rather than filtering in Python:
# ✅ Good: Filter in SQL
df = client.query.query_pandas("""
SELECT * FROM cpu
WHERE time > now() - INTERVAL '1 hour'
AND host = 'server01'
""")
# ❌ Bad: Fetch all, filter in Python
df = client.query.query_pandas("SELECT * FROM cpu")
df = df[df["host"] == "server01"]2. Select only needed columns
# ✅ Good: Select specific columns
df = client.query.query_pandas("""
SELECT time, host, usage_idle FROM cpu
""")
# ❌ Bad: Select all columns
df = client.query.query_pandas("SELECT * FROM cpu")3. Use LIMIT for exploration
# ✅ Good: Limit during exploration
df = client.query.query_pandas("SELECT * FROM cpu LIMIT 100")4. Use Arrow for large results
# For 100K+ rows, Arrow is significantly faster
table = client.query.query_arrow("SELECT * FROM cpu")
df = table.to_pandas() # Zero-copy conversionNext steps
- Data Management - Retention, CQs, and deletion
- Data Ingestion - Write data to Arc
- API Reference - Raw REST API documentation
Data Ingestion
Write to Arc Enterprise from Python with write_columnar, write_dataframe, buffered writes, and line protocol, and how writes route to the cluster's writer node.
Data Management
Manage Arc Enterprise retention policies, continuous queries, delete operations, and API tokens from Python, including tokens scoped by the cluster's RBAC roles.