Skip to main content
Redpanda is a Kafka-compatible streaming platform. Its Redpanda Connect pipelines include an Iceberg output that writes messages to Iceberg tables through a REST catalog. Point it at your oleander catalog to land topics in your lake, ready to query across compatible engines. Redpanda Connect pipeline reading the hello-world topic, mapping each message, and writing to the hello_world Iceberg table

​
What you need

The organization ID is both the OAuth client ID and the catalog warehouse name. The API key is the OAuth client secret. You also need a Redpanda Cloud cluster. Connect pipelines run on Serverless, Dedicated, and BYOC clusters, and the iceberg output is available from Redpanda Connect 4.80.0.

​
Setup

  1. In Redpanda, create a user for the pipeline under Security → Users, with ACLs to read your topic and its consumer group.
  2. Open the cluster’s Secrets Store and add three secrets:
  1. Open Connect, click Create a pipeline, switch to the YAML tab, and paste this config, replacing <organization-id>:
  1. Click Save, then Start pipeline.
The iceberg output fields that connect it to oleander:
Redpanda Connect exchanges your organization ID and API key for a short-lived catalog token, and oleander vends the storage credentials for each table. The storage.aws_s3.bucket field is required by the output’s schema, but oleander supplies the real table location, so any value works and there are no S3 keys to configure.
The iceberg output writes JSON objects, inferring a column from each field. The mapping processor above wraps the plain-text messages from Redpanda’s quickstart hello-world topic into {message, topic} objects. If your topic already carries JSON, drop the pipeline section. Once the pipeline starts, its logs show the output creating the table on the first write: Redpanda Connect pipeline logs showing the iceberg output becoming active and creating the redpanda.hello_world table

​
Querying the data

Query the table like any other table in your catalog:
The oleander lake querying redpanda.hello_world, returning five rows from the hello-world topic Every batch is an Iceberg commit, and batching.period sets how often one lands. A longer period means fewer, larger files. For long-running pipelines, schedule Iceberg maintenance to compact small files, expire old snapshots, and keep query times flat. For keyed topics, the output can upsert with row_operation and identifier_fields. Prefer merge_strategy: copy-on-write there: the default merge-on-read writes equality deletes, which slow down reads until maintenance compacts them.
This guide uses Redpanda Connect, not Redpanda’s built-in Iceberg Topics. Iceberg Topics only runs on BYOC and BYOVPC clusters and writes its data files to Redpanda’s own bucket, so it can’t use the storage your oleander catalog provides.