Skip to main content
Bloom is the engine oleander reaches for first. It is built on Apache DataFusion and DataFusion Distributed, reads Iceberg through the same REST catalog as everything else, and runs in two shapes from the same binary: local, inside a single sandbox, and distributed, across a fleet of workers. You do not choose between them. The query router sizes the run from the input and picks the shape.

​
Why DataFusion

DataFusion is a Rust query engine with a vectorized Arrow execution core, a full SQL frontend, and a physical planner that parallelizes across cores. Bloom adds three things to it:
  • An Iceberg bridge. A DataFusion CatalogProvider over Iceberg REST that resolves only the tables a query actually names, rather than loading every table in every catalog.
  • Predicate pushdown into Iceberg scans. Supported primitive predicates are pushed down so Iceberg can prune by partition, manifest, file statistics, and reader. DataFusion still re-evaluates the original filters, so pruning never changes results.
  • Distributed execution. The coordinator plans file tasks once and streams versioned tasks to workers. Workers never contact or preload the REST catalog.

​
Local mode

A local Bloom run executes entirely inside one sandbox and returns rows on the call. This is the default for interactive reads against the oleander catalog. Sandbox size comes from the router’s input estimate - 2 vCPU for small queries up to 32 vCPU for inputs approaching 50 GiB. See machine sizing.

​
Distributed mode

Past 50 GiB of estimated input, Bloom goes distributed: a coordinator plus 2 to 10 workers, sized from the input. A distributed run does not return rows. It always commits to a destination table, takes OVERWRITE (default) or APPEND, and emits lineage from the workflow once it finishes, with the destination as the output dataset. The final worker stage writes Parquet directly and returns only Iceberg DataFile metadata to the coordinator, which performs one Iceberg commit. Workers use refreshable, prefix-scoped credential leases for both reads and writes, and never receive the catalog token.
Bloom is metered compute and requires a payment method on file. An org without one falls back to DuckDB.

​
URL tables

Bloom can read a single CSV, JSON, or Parquet file directly from an absolute URL, quoted in place of a table name:
Anonymous http:// and https:// reads work in both modes. Range-capable servers are read by file-scan range; a server that ignores byte ranges or reports no length is downloaded once per runtime and cached in memory. This is deliberately narrow. It rejects credentials, headers, query strings, fragments, directories, globs, relative paths, and cloud schemes such as s3://, gs://, and azure://. Use a catalog or an external connection for anything beyond one public file.

​
Writing to a table

When Bloom is asked to produce a table, the write is planned as a DataFusion DML plan whose target is the final table, and the plan is exposed before physical execution so lineage is extracted from the real plan rather than from a re-parse of the SQL. OVERWRITE completely replaces the destination - schema, partitioning, properties, history, and data. The replacement table is unpartitioned. APPEND adds to an existing compatible table, creating it unpartitioned from the result schema if it does not exist.

​
Versions

Iceberg 0.9.1 exposes no full-table overwrite transaction, so OVERWRITE is implemented by writing a replacement table and swapping it in with catalog renames after the write succeeds. The logical plan still names only the final destination.