Why DataFusion
DataFusion is a Rust query engine with a vectorized Arrow execution core, a full SQL frontend, and a physical planner that parallelizes across cores. Bloom’s execution layer adds:- An Iceberg bridge. A DataFusion
CatalogProviderover Iceberg REST that resolves only the tables a query actually names, rather than loading every table in every catalog. - Predicate pushdown into Iceberg scans. Supported primitive predicates are pushed down so Iceberg can prune by partition, manifest, file statistics, and reader. DataFusion still re-evaluates the original filters, so pruning never changes results.
- Distributed execution. The coordinator plans file tasks once and streams versioned tasks to workers. Workers never contact or preload the REST catalog.
SQL and functions
For SQL syntax and built-in functions, see the DataFusion SQL reference and scalar functions. Check Bloom’s DataFusion version when consulting upstream feature availability. The Bloom docs cover the features and functions oleander adds to DataFusion. Function references live under Bloom → Functions.Local mode
A local Bloom run executes entirely inside one sandbox and returns rows on the call. This is the default for interactive reads against theoleander catalog. It can also save the result to a table - see Save result below.
Sandbox size comes from the router’s input estimate - 2 vCPU for small queries up to 32 vCPU for inputs approaching 50 GiB. See machine sizing.
Save result
Turn on Save result in a lake tab’s run settings and a local Bloom run persists its rows to an Iceberg table before returning them, the same toggle DuckDB and Auto use. Leave the name blank and the table isoleander.default.query_<hash>, keyed on the query text, so re-running the same query overwrites the same table; give it a name to pick the table instead. Auto runs that land on Bloom honor the toggle too.
Over the API, pass output to POST /api/v1/warehouse/bloom:
output to return rows without saving. An empty output: {} saves to query_<hash>. Only the oleander catalog is supported, namespace defaults to default, and write_mode is OVERWRITE (default) or APPEND. The response carries saved_table_name, and the preview rows are read back from the saved table, so with APPEND the preview includes rows that were already there.
The query runs once to write the table; the preview re-applies the query’s final ORDER BY when reading it back, but not its LIMIT or OFFSET, which already decided what was saved. Order by columns the query outputs - a sort key that is not in the result, or ORDER BY random(), is rejected before anything runs. Turn the toggle off for those queries.
Distributed mode
Past 50 GiB of estimated input, Bloom goes distributed: a coordinator plus 2 to 10 workers, sized from the input.
A distributed run does not return rows. It always commits to a destination table, takes
OVERWRITE (default) or APPEND, and emits lineage from the workflow once it finishes, with the destination as the output dataset.
The final worker stage writes Parquet directly and returns only Iceberg DataFile metadata to the coordinator, which performs one Iceberg commit. Workers use refreshable, prefix-scoped credential leases for both reads and writes, and never receive the catalog token.
Bloom is metered compute and requires a payment method on file. An org without one falls back to DuckDB.
URL tables
Bloom can read a single CSV, JSON, or Parquet file directly from an absolute URL, quoted in place of a table name:http:// and https:// reads work in both modes. Range-capable servers are read by file-scan range; a server that ignores byte ranges or reports no length is downloaded once per runtime and cached in memory.
This is deliberately narrow. It rejects credentials, headers, query strings, fragments, directories, globs, relative paths, and cloud schemes such as s3://, gs://, and azure://. Use a catalog or an external connection for anything beyond one public file.
Writing to a table
When Bloom is asked to produce a table, the write is planned as a DataFusion DML plan whose target is the final table, and the plan is exposed before physical execution so lineage is extracted from the real plan rather than from a re-parse of the SQL.OVERWRITE completely replaces the destination - schema, partitioning, properties, history, and data. The replacement table is unpartitioned. APPEND adds to an existing compatible table, creating it unpartitioned from the result schema if it does not exist.
Versions
Iceberg 0.9.1 exposes no full-table overwrite transaction, so
OVERWRITE is implemented by writing a replacement table and swapping it in with catalog renames after the write succeeds. The logical plan still names only the final destination.