Spark SQL - oleander

`submitSparkSql(options)`

Submit a Spark SQL query that writes its result to an Iceberg output table. Returns run info immediately; use getRun() or submitSparkSqlAndWait() to poll status.

const submitted = await client.submitSparkSql({
  namespace: "my-namespace",
  name: "nightly-agg",
  query: "SELECT day, count(*) AS n FROM oleander.default.events GROUP BY day",
  outputTable: "default.daily_counts",
});

console.log(submitted.run_id, submitted.state); // "..." SUBMITTED

Parameters

options.namespace

string

required

Job namespace.

options.name

string

required

Job name.

options.query

string

required

The Spark SQL query to execute.

options.outputTable

string

required

The Iceberg table the query result is written to.

options.writeMode

string

default:"OVERWRITE"

OVERWRITE or APPEND. Case-insensitive on input.

options.driverMachineType

SparkMachineType

default:"spark.1.b"

Driver machine type.

options.executorMachineType

SparkMachineType

default:"spark.1.b"

Executor machine type.

options.executorNumbers

number

default:"2"

Number of executors, from 1 to 20.

Return type: `SparkSqlSubmitResponse`

Field	Type	Description
`run_id`	`string`	The ID of the submitted run
`state`	`string`	Initial run state (for example, `SUBMITTED`)
`namespace`	`string`	Job namespace
`job_name`	`string`	Job name
`pipeline_id`	nullable string	Pipeline ID if one exists
`submitted_at`	`string`	ISO timestamp of submission
`compute`	`object`	`driver_machine_type`, `executor_machine_type`, `executor_numbers`
`output_table`	`string`	The output table
`write_mode`	`string`	The write mode used

`submitSparkSqlAndWait(options)`

Submit a Spark SQL query and poll until the run reaches a terminal state (COMPLETE, FAIL, or ABORT). Throws an error if the timeout is exceeded.

const { runId, state, run } = await client.submitSparkSqlAndWait({
  namespace: "my-namespace",
  name: "nightly-agg",
  query: "SELECT day, count(*) AS n FROM oleander.default.events GROUP BY day",
  outputTable: "default.daily_counts",
  writeMode: "APPEND",
});

if (state !== "COMPLETE") {
  throw new Error(`Run ${runId} ended with state: ${state}`);
}

Accepts all submitSparkSql options plus:

options.pollIntervalMs

number

default:"10000"

Milliseconds between status polls.

options.timeoutMs

number

default:"600000"

Maximum time to wait in milliseconds before throwing a timeout error.

Return shape

Field	Type	Description
`runId`	`string`	The ID of the submitted run
`state`	`string`	Terminal state (`COMPLETE`, `FAIL`, or `ABORT`)
`run`	`RunResponse`	Full run details

​submitSparkSql(options)

​Parameters

​Return type: SparkSqlSubmitResponse

​submitSparkSqlAndWait(options)

​Return shape

`submitSparkSql(options)`

Parameters

Return type: `SparkSqlSubmitResponse`

`submitSparkSqlAndWait(options)`

Return shape