Query routing

You do not pick an engine. Send SQL to oleander and the query routing layer parses it, works out which tables it touches, estimates how much data those tables hold from Iceberg snapshot metadata, and picks both the engine and the machine size to match. Every decision comes back with a reason trail in engine_decision, so a run is never a black box.

What each engine can do

	DuckDB	Polars	Bloom	Spark
Iceberg read	✓	✓	✓	✓
Other catalogs	✓	✗	✗	✓
BigQuery / Snowflake / Postgres	✓	✗	✗	✗
Write SQL	✓	✓	✗	✓
Iceberg DDL	✗	✗	✗	✓
Scripts	✗	✓	✗	✗
Table output	✗	✓	✓ distributed	✓
Returns rows	✓	✓ local	✓ local	✗

Set engine explicitly and oleander validates the request instead of rerouting - an impossible combination returns an engine capability error. Leave it on auto unless you have a reason.

When SQL isn’t enough

The router covers queries. Two engines take arbitrary code when the work is ingest, streaming, or a transformation that SQL cannot express.

Spark

Serverless Spark runs PySpark applications and JARs you upload as versioned artifacts, on managed infrastructure or your own registered cluster. Reach for it when you need to:

Ingest from a source the lake does not attach - a JDBC database, an API, a proprietary format. Postgres imports are exactly this, packaged.
Stream with Spark Structured Streaming, monitoring run events as batches arrive.
Write arbitrary code - ML training, multi-step pipelines, anything with control flow.

Spark is also where the router sends Iceberg DDL and the largest writes, so submitted queries and jobs you wrote yourself land in the same run history.

Polars scripts

Pass script instead of sql and you get a Python Polars DataFrame program against your lake tables, in an isolated sandbox with no infrastructure to provision. Declare what it reads in tables, assign result, and the router sends it to Polars.

result = events.group_by("day").len()

This is the middle ground: real Python for reshaping, joining, and computing things SQL makes awkward, without packaging a Spark job.

Lineage across engines

Every engine emits OpenLineage through the same path, so lineage is continuous no matter what ran a given step. A DuckDB query, a Bloom scan, a Polars script, and a Spark job that read and wrote the same tables produce the same dataset identities and connect into one graph. Dataset names are qualified with the session catalog specifically so they match across engines - oleander.default.events is the same node whichever engine touched it. Each event also carries the routing decision as a facet, so the graph records not just what ran but which engine ran it and why.

Engine	When lineage is emitted
DuckDB	On the query, including failures
Bloom	On the query; distributed runs emit from the workflow once they finish, under the run id the submission returned
Polars	On the run. Script mode needs its reads declared in `tables` - a script with none runs without lineage
Spark	From the OpenLineage listener, with column-level lineage from the execution plan

That is what makes a query and a job you wrote yourself equivalent to impact analysis and column lineage: the graph does not care which engine produced an edge.

Explain before you run

Pass explain: true to get the engine, estimated input size, and sandbox tier without executing anything or spending compute. Worth doing before a query you expect to be large.

const { engine_decision } = await oleander.queryRun({
  sql: "SELECT * FROM oleander.default.events",
  explain: true,
});
console.log(engine_decision?.engine, engine_decision?.size_band);

Machine sizing

Roughly 2 GiB of input per vCPU, snapped up to a provisioned tier, with 2 GB of memory per vCPU.

Estimated input	vCPUs	Memory
unknown or under 2 GiB	2	4 GB
~2-6 GiB	4	8 GB
~6-14 GiB	8	16 GB
~14-30 GiB	16	32 GB
~30 GiB and up	32	64 GB

Past 50 GiB the query leaves the sandbox and goes distributed, where worker count and machine type come from the same estimate:

Input size	Worker machine	Worker count
under 250 GiB	`bloom.2.b`	`ceil(bytes / 32 GiB)`
250 GiB and up	`bloom.4.b`	`ceil(bytes / 64 GiB)`

Worker count is clamped to between 2 and 10, so 900 GiB caps at 10 × bloom.4.b. A size you request explicitly wins outright. The org’s default sandbox size in lake settings still applies to the CLI and the SDKs’ legacy query methods, which bypass the router. Responses report the sandbox that actually served the run under compute - a warm larger sandbox can serve a smaller request, and that is the machine the org is billed for.

Reading the decision

engine_decision carries engine, the ordered reasons trail, size_band, and sandbox. In the lake UI it is the tooltip on the engine badge; distributed and async runs return no rows and report through a toast with the engine, destination table, and a link to the run.

A 402 or 403 on a query is a billing decision, not a transient failure. Read the message and act on it rather than retrying the same query.

Getting Started

Platform

Connections

Coding with agents

CLI

External Compute

What each engine can do

When SQL isn’t enough

Spark

Polars scripts

Lineage across engines

Explain before you run

Machine sizing

Reading the decision

​What each engine can do

​When SQL isn’t enough

​Spark

​Polars scripts

​Lineage across engines

​Explain before you run

​Machine sizing

​Reading the decision

What each engine can do

When SQL isn’t enough

Spark

Polars scripts

Lineage across engines

Explain before you run

Machine sizing

Reading the decision