Skip to main content
Streamkap is a managed data streaming platform that uses change data capture (CDC) to continuously replicate database inserts, updates, and deletes to destinations like Apache Iceberg. Connect its Iceberg destination to your oleander catalog to keep tables in your lake up to date and ready to query across compatible engines. For a live example, Bluehouse warehouses the public Bluesky firehose through exactly this connection - a Streamkap Iceberg destination writing into an oleander catalog namespace. Streamkap pipeline from the bluesky-jetstream webhook source to the oleander Iceberg destination

​
What you need

The organization ID is both the OAuth client ID and the catalog warehouse name. The API key is the OAuth client secret.

​
Setup

  1. In Streamkap, open Destinations, click Add, and choose Iceberg.
  2. On the Auth tab, fill in the catalog connection:
  1. On the Settings tab:
  1. Click Save, then attach the destination to a pipeline.
The OAuth2 Credential is the organization ID and API key joined with a colon. Streamkap exchanges it for a short-lived catalog token on your behalf, so you never paste a token by hand.

​
Partitioning

Streaming data is usually time series, so partition new tables by event time. Queries that filter on a time range then read only the matching partitions. Set Partition By on the Settings tab to a comma-separated Iceberg partition spec: Streamkap Partition By field set to day(created_at) Combine them, e.g. day(created_at), bucket(id, 16).
  • The spec applies to tables Streamkap creates after you save. Existing tables keep their layout.
  • Column names are case-sensitive. If a column doesn’t match, Streamkap creates the table unpartitioned, so check the first new table.
  • In upsert mode, partition only on columns whose values never change, such as created_at or bucket(id, 16).
To give individual tables their own spec, see Streamkap’s partitioning docs.

​
Querying the data

Streamkap creates one table per topic under the schema you chose. Query it like any other table in your catalog:
Tables land compacted, so the catalog stays healthy without extra maintenance. For long-running streams, schedule Iceberg maintenance to expire old snapshots and keep query times flat.