DuckDB

Overview

oleander’s lake provides a powerful environment for data analysis, collaboration, and discovery. Execute SQL queries, manage your tables, and contribute to the community with automatic lineage tracking and observability built-in. DuckDB is one of four engines behind the lake, and the most widely capable one. It reaches every external connection and registered catalog, and it handles mutations that Bloom cannot commit. You do not select it by hand: the query router sends work here when the query calls for it. Every organization is automatically provisioned a private Apache Iceberg catalog named oleander. It has two namespaces: default for your own tables and telemetry for platform events (run events, traces, logs). It is cross-compatible with Spark, DuckDB, and any other Iceberg-aware tool. See Catalogs for the full schema. You can also connect BigQuery, Snowflake, and Postgres to query them alongside your lake data, bring your own compute to connect from a local DuckDB or PySpark session, or register an external Iceberg catalog to query your existing S3 Tables data. The lake’s left panel splits into two tabs: Data, for browsing your Iceberg catalogs and external connections, and Schedules, for scheduled queries, transfers (recurring Postgres syncs), and table maintenance, together with their cadence and health status. See Scheduled queries with time-travel below, Table syncing, and Table health and maintenance for what lives in each.

Portable

The lake is portable and can be used with any DuckDB environment. This means you can run queries, import and export data, and leverage all the features of oleander’s lake wherever DuckDB is supported, including on your local machine, in cloud environments, or as part of larger data workflows.

For full lineage coverage, always use the oleander lake API or platform (not a direct DuckDB connection) for queries. API usage ensures all operations are tracked and observable through the lake interface.Direct DuckDB connections are supported for power users; however, operations performed outside the API will not be traceable in lineage or workflows. The CLI’s oleander duckdb shell is the exception: it reports lineage for every statement.

Automatic lineage

Every operation in oleander automatically captures lineage metadata using the OpenLineage specification. No manual configuration id required.

Serverless SQL execution

Run SQL queries directly in oleander’s lake environment. Whether you’re analyzing your own data or exploring public tables, the SQL interface provides:

Fast Query Execution - Execute queries with optimized performance
Interactive Results - View and export query results instantly
Query History - Access your past queries and results
Automatic Lineage - Every query automatically captures lineage metadata

Sync your data

There are several ways to sync your own data into oleander’s lake:

Upload files

Upload a Parquet, CSV, or JSON file as an Iceberg table. Click upload data at the bottom of the lake’s left panel, press ⌘U, choose Create table from file from a namespace’s more-options menu, or drop a file onto a namespace in the catalog tree.

Pick the catalog, namespace, and table name - a new namespace can be created inline - then rename or drop columns before loading. An upload always creates a new table; to add rows to an existing table, load into a new one and INSERT from it. Uploads run in the background, and their status shows in the panel footer.

DuckDB import operations

Connect to your lake and run any DuckDB import operation to load data from various sources

-- Insert flower data from a garden API
INSERT INTO oleander.default.flowers
SELECT
  flower_data->>'name' AS name,
  flower_data->>'species' AS species,
  flower_data->>'color' AS color,
  (flower_data->>'evergreen')::BOOLEAN AS evergreen,
  (flower_data->>'petals')::INTEGER AS petals,
  flower_data->>'notes' AS notes
FROM read_json('https://api.garden.example.com/flowers', format='newline_delimited');

CREATE TABLE statements

Run CREATE TABLE statements to define and populate tables with your data

CREATE TABLE oleander.default.flowers AS
SELECT
  'oleander' AS name,
  'nerium' AS species,
  'pink' AS color,
  TRUE AS evergreen,
  5 AS petals,
  'toxic but beautiful' AS notes;

API-based queries

Use API-based queries to pull data from external sources directly into your lake (see example below)

S3 scheduled sync

Configure a daily sync of parquet files from your S3 bucket into oleander.default. Navigate to lake settings and add a sync under Scheduled S3 syncs. Provide your AWS credentials, bucket name, region, and optional prefix. Only files whose names don’t already exist as tables will be synced on each run.Each sync card shows the outcome of recent runs - Success, Partial (some files failed), or Failed - and the last error.

Scheduled queries with time-travel

Any saved query can be put on a schedule. Open Schedule query on a query tab and set:

Runs - Every 15 minutes, Hourly, or Daily. The 15-minute cadence suits rollups whose work is bounded by what arrived since the last run, so a tighter interval is cheaper per run rather than more expensive.
Engine - Auto, Bloom, DuckDB, Spark, or Polars. Leave it on Auto unless you have a reason to pin one.
Run as - the principal each run executes as. A schedule can run as yourself or as an Application principal; other Humans and Agents are not eligible, even if you can delegate to them. The principal’s own roles govern the run, so it needs warehouse access and permission to execute the selected engine, and you need Describe and Manage grants on it to pick it.
Destination - the namespace.table in the oleander catalog where each run lands, e.g. oleander.default.sch_nyc_taxi_daily.

The execution principal’s membership and compute access are checked again at every run, not just when the schedule is created. Removing a role from that principal stops the schedule’s runs from that point on. Each successful run writes a new Iceberg snapshot to the destination table rather than overwriting it, so past runs stay queryable with time-travel instead of you needing to keep separate historical copies of the data. Please note that these do incur additional storage as each version is kept to preserve time-traveling capabilities.

SELECT * 
FROM oleander.default.sch_nyc_taxi_daily 
AT (TIMESTAMP => now() - INTERVAL '24 HOURS');

Every run of a schedule appears on the Workloads page under one workload, oleander.schedules/<destination table>, with column lineage from the source tables to the destination. Scheduled queries are listed under the lake panel’s Schedules tab, alongside transfers, with their cadence and next run time. A failed run shows what went wrong on hover. Each row’s menu changes the engine or interval, pauses or resumes the schedule, and has a Run as… entry to change its execution principal.

Schedules created before execution principals existed have none and run under the organization’s identity. Set one with Run as… - editing such a schedule’s query requires it, and it is the only way to move the schedule onto Bloom.

Schedules can also be created and managed from an agent - see Saved query schedules.

External connections

Attach BigQuery, Snowflake, Postgres, MySQL, and MongoDB in lake settings and query them alongside your Iceberg data as connection_name.schema.table:

-- Query a BigQuery table directly
SELECT *
FROM my_bq_connection.analytics.events
WHERE _PARTITIONDATE = '2024-01-01'
LIMIT 100;

-- Join BigQuery data with your Iceberg tables
SELECT e.user_id, u.plan
FROM my_bq_connection.analytics.events e
JOIN oleander.default.users u ON e.user_id = u.id
WHERE e.event_type = 'signup';

Connection queries run on Bloom or DuckDB. The query router prefers Bloom for reads and sends writes, Snowflake queries, and anything Bloom can’t take here.

Credentials are stored encrypted and are never returned to the browser after saving.

Bring your own (Iceberg) catalog

We support adding Iceberg via S3 table buckets. This lets you use your existing Iceberg catalog within the oleander lake and consume the oleander compute layer.

This allows the oleander aws account to generate temporary credentials to access your catalog.

Create a role that has access to S3Tables. You can scope this down to specific tables.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3tables:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "PassRoleToS3TablesReplication",
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": [
                        "replication.s3tables.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

Configure your IAM role’s trust relationship with this policy so oleander can assume it. The registration dialog shows the same policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::579897423473:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

Navigate to lake settings, select S3 Tables under External catalogs, and enter your AWS account ID, role ARN, and table bucket name. Your catalog is then listed on the lake page.

Bring your own compute

Connect to your oleander lake from any DuckDB terminal, PySpark application, or client library. You can easily connect to your lake in our settings here.

DuckDB terminal

Use the oleander CLI to launch a connected DuckDB session:

oleander duckdb

-- Query your data
SELECT * FROM oleander.default.flowers LIMIT 10;

Getting started

Get started with oleander’s lake in just a few steps:

Navigate to the Lake in your oleander dashboard to access the SQL interface
Run a SQL query on a public table to get familiar with the interface:

SELECT * FROM oleander.default.iris_dataset LIMIT 10;

Sync your own data to start analyzing your tables. Create a new table with sample data:

CREATE TABLE oleander.default.flowers AS
SELECT
  'oleander' AS name,
  'nerium' AS species,
  'pink' AS color,
  TRUE AS evergreen,
  5 AS petals,
  'toxic but beautiful' AS notes;

Follow up with an insertion if you’d like as well:

INSERT INTO oleander.default.flowers (name, species, color, evergreen, petals, notes)
VALUES (
  'camellia',
  'camellia japonica',
  'white',
  TRUE,
  8,
  'winter-blooming, glossy leaves'
);

Explore lineage to see how data flows through your queries. Every operation automatically captures lineage metadata for complete observability.

The lake provides a complete environment for data analysis, from quick SQL queries to complex Spark jobs, all with observability built in.

FAQs

Why do clock time and query time differ?

Clock time and query time can differ for several reasons:

Clock time measures the total wall-clock time from when a query is submitted until it comes back to you. This also includes lineage extraction and creation.
Query time measures the actual execution time spent processing the query in the database engine running your query.

The difference typically occurs due to:

Network latency
Lineage extraction

You only pay for the query time.

Why doesn't the direct DuckDb connection instrument lineage?

Direct DuckDB connections bypass the oleander API layer, which is responsible for capturing and instrumenting lineage metadata. When you connect directly to DuckDB, your queries execute directly against the database engine without going through oleander’s observability infrastructure.To capture lineage:

Use the oleander lake API or platform for queries
All operations performed through the API are automatically tracked and observable
Lineage metadata is captured using the OpenLineage specification

Direct DuckDB connections are supported for power users who need direct database access, but they operate outside the observability layer. If you need lineage tracking for your queries, use the oleander API instead of a direct connection.

Getting Started

Platform

IAM

Connections

Integrations

Coding with agents

CLI

External Compute

Overview

Portable

Automatic lineage

Serverless SQL execution

Sync your data

Scheduled queries with time-travel

External connections

Bring your own (Iceberg) catalog

Bring your own compute

DuckDB terminal

Getting started

FAQs

​Overview

​Portable

​Automatic lineage

​Serverless SQL execution

​Sync your data

​Scheduled queries with time-travel

​External connections

​Bring your own (Iceberg) catalog

​Bring your own compute

​DuckDB terminal

​Getting started

​FAQs

Overview

Portable

Automatic lineage

Serverless SQL execution

Sync your data

Scheduled queries with time-travel

External connections

Bring your own (Iceberg) catalog

Bring your own compute

DuckDB terminal

Getting started

FAQs