> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oleander.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Introduction
> The agent-first data lake.
## What is oleander?
oleander is your agent’s data platform team. We host and manage storage and compute tools in a unified platform with automatic query routing, so your agent can query, transform, and run jobs without managing engines or infrastructure.
## Who we are
oleander was founded by the co-creators of **[OpenLineage](https://github.com/OpenLineage/openlineage)** and **[Marquez](https://github.com/MarquezProject/marquez)**. An open standard the data industry uses to emit and track lineage.
## The query routing layer
Send a query without naming an engine and the **query routing layer** decides where it runs. It parses the statement, resolves the tables it touches, and picks an engine from what the query needs: a [connection](/platform/connections/bigquery) or a foreign catalog means DuckDB, Iceberg DDL means Spark, a DataFrame script means Polars. Only once nothing forces a choice does size decide - estimated from Iceberg snapshot metadata, and used to pick the machine as well as the engine.
Every decision carries a reason trail, returned as `engine_decision`, so a run is never a black box. An agent can explain exactly why a query ran where it did, and `explain: true` returns the whole decision without spending any compute.
### Compute scoped to the request
Each query gets its own isolated sandbox, sized from that query's estimated input rather than from an org-wide setting. A small read lands on 2 vCPUs. The same query a year later, against a table that grew tenfold, lands on a bigger one. Past 50 GiB it stops being a sandbox at all and goes distributed across sized workers.
For an agent this matters more than it does for a person: it can fire off an exploratory query without first deciding what it will cost, and a query that turns out to be expensive gets the machine it needs instead of timing out on a shared one. Nothing is provisioned ahead of time and nothing stays running between requests.
### Beyond SQL
Routing covers queries. When the work is ingest, streaming, or code that SQL cannot express, [Serverless Spark](/platform/spark/jobs) runs PySpark applications and JARs on managed infrastructure or your own cluster - JDBC and API ingest, Structured Streaming, ML training, anything with control flow. [Polars scripts](/platform/polars) sit in between: real Python against your lake tables, no job to package.
Both emit the same OpenLineage events as a routed query, so a Spark job and a one-off SELECT are the same kind of node in the graph.
See [Query routing](/platform/query-routing) for the rules in order.
## Agent in the loop
oleander is built around the idea that agents should be first-class participants in your data engineering workflow - not an afterthought bolted onto a dashboard.
Connect your agent via MCP and it immediately has access to your lake, catalogs, run history, lineage graph, costs, logs, and traces. Ask it anything:
> "What's the average petal length in the iris dataset, broken down by species?"
The agent calls `catalogs_list` to find your tables, then `query_run` to run the query - no SQL written by you, no engine specified, no dashboard opened.
```sql theme={null}
SELECT species, AVG(petal_length) AS avg_petal_length
FROM oleander.default.iris
GROUP BY species
ORDER BY avg_petal_length DESC;
```
Agents don't just query - they author jobs too. Give your agent a task and it will write the PySpark job, upload it as a versioned artifact, submit a run, and verify the output:
> "Build a Spark ML classifier that reads `oleander.default.iris`, uses the sepal and petal measurements to predict flower species, trains and evaluates a random forest model, and writes the test-set predictions to `oleander.default.iris_predictions`."
The agent calls `spark_artifacts_upload` to store the script, then `spark_jobs_submit` to run it. Once the run completes, it verifies the output automatically - comparing row counts and byte volumes against the previous window, checking the schema diff, confirming lineage, and benchmarking cost per record. If anything regresses, it opens an investigation and surfaces a fix before you've noticed anything.
Spark Streaming jobs follow the same pattern. The agent writes the streaming job, submits it to a connected cluster, and monitors run events as they arrive. If throughput drops or a batch fails, it surfaces the issue with full context so the engineer can act immediately.
## Self-validating by default
Every run writes lineage, traces, and cost events to the lake automatically. oleander compares each run against the previous window, flags regressions in row counts, byte volumes, and duration, and surfaces anomalies before they reach production.
Engineers and agents see the same signal. When the agent catches something, it opens an investigation with root cause, evidence from telemetry, and a suggested next step. No human has to notice the alert and open four tabs to figure out what happened.
## Multi-engine, cost-aware
| Engine | When oleander uses it |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| **[Bloom](/platform/bloom)** | The default for reads. DataFusion over Iceberg, local on a sandbox or distributed across workers |
| **DuckDB** | Mutations, foreign catalogs, and [BigQuery, Snowflake, and Postgres](/platform/connections/bigquery) connections |
| **[Polars](/platform/polars)** | DataFrame scripts, in-memory transformation, high-throughput row processing |
| **[Spark](/platform/spark/jobs)** | Iceberg DDL, large-scale jobs, existing cluster integrations |
Agents routing through the MCP inherit this logic. A query that fits in a sandbox never starts a cluster. A job that needs distributed compute gets one sized to its input. Cost per record is tracked on every run.
Beyond the managed engines, attach the data you already have as [connections](/platform/connections/bigquery) - BigQuery, Snowflake, and Postgres, queryable alongside your lake - and point the compute you already run at oleander: your own [Spark](/integrations/spark), [Airflow](/integrations/airflow), and [dbt](/integrations/dbt).
## How oleander works
Upload files directly, sync from S3, import from Postgres, or attach BigQuery and Snowflake. Your data lands in the oleander lake as Iceberg tables, queryable immediately.
Add the MCP server in one command. Your agent gets access to 41 tools: routed queries, catalog management, lineage tracing, run inspection, cost attribution, and Spark job submission.
Every job run emits lineage and telemetry. oleander validates each run against historical baselines, flags anomalies, and triggers investigations when something looks wrong.
CLI, API, MCP, tasks, and SDK all read from the same lake and context graph. Engineers and agents work with the same data, the same lineage, and the same cost signal.