Spark SQL

`submitSparkSql(options)`

Submit a Spark SQL query that writes its result to an Iceberg output table. Returns run info immediately; use getRun() or submitSparkSqlAndWait() to poll status.

const submitted = await client.submitSparkSql({
  namespace: "my-namespace",
  name: "nightly-agg",
  query: "SELECT day, count(*) AS n FROM oleander.default.events GROUP BY day",
  outputTable: "default.daily_counts",
});

console.log(submitted.run_id, submitted.state); // "..." SUBMITTED

Parameters

string

required

Job namespace.

string

required

Job name.

string

required

The Spark SQL query to execute.

string

required

The Iceberg table the query result is written to.

string

default:"OVERWRITE"

OVERWRITE or APPEND. Case-insensitive on input.

SparkMachineType

default:"spark.1.b"

Driver machine type.

SparkMachineType

default:"spark.1.b"

Executor machine type.

number

default:"2"

Number of executors, from 1 to 20.

Return type: `SparkSqlSubmitResponse`

Field	Type	Description
`run_id`	`string`	The ID of the submitted run
`state`	`string`	Initial run state (for example, `SUBMITTED`)
`namespace`	`string`	Job namespace
`job_name`	`string`	Job name
`pipeline_id`	nullable string	Pipeline ID if one exists
`submitted_at`	`string`	ISO timestamp of submission
`compute`	`object`	`driver_machine_type`, `executor_machine_type`, `executor_numbers`
`output_table`	`string`	The output table
`write_mode`	`string`	The write mode used

`submitSparkSqlAndWait(options)`

Submit a Spark SQL query and poll until the run reaches a terminal state (COMPLETE, FAIL, or ABORT). Throws an error if the timeout is exceeded.

const { runId, state, run } = await client.submitSparkSqlAndWait({
  namespace: "my-namespace",
  name: "nightly-agg",
  query: "SELECT day, count(*) AS n FROM oleander.default.events GROUP BY day",
  outputTable: "default.daily_counts",
  writeMode: "APPEND",
});

if (state !== "COMPLETE") {
  throw new Error(`Run ${runId} ended with state: ${state}`);
}

Accepts all submitSparkSql options plus:

number

default:"10000"

Milliseconds between status polls.

number

default:"600000"

Maximum time to wait in milliseconds before throwing a timeout error.

Return shape

Field	Type	Description
`runId`	`string`	The ID of the submitted run
`state`	`string`	Terminal state (`COMPLETE`, `FAIL`, or `ABORT`)
`run`	`RunResponse`	Full run details

TypeScript

Python

`submitSparkSql(options)`

Parameters

Return type: `SparkSqlSubmitResponse`

`submitSparkSqlAndWait(options)`

Return shape

​submitSparkSql(options)

​Parameters

​Return type: SparkSqlSubmitResponse

​submitSparkSqlAndWait(options)

​Return shape

`submitSparkSql(options)`

Parameters

Return type: `SparkSqlSubmitResponse`

`submitSparkSqlAndWait(options)`

Return shape