Spark SQL

`submit_spark_sql(options)`

Submit a Spark SQL query that writes its result to an Iceberg output table. Returns run info immediately; use get_run() or submit_spark_sql_and_wait() to poll status.

from oleander_sdk import SparkSqlSubmitOptions

submitted = await client.submit_spark_sql(SparkSqlSubmitOptions(
    namespace="my-namespace",
    name="nightly-agg",
    query="SELECT day, count(*) AS n FROM oleander.default.events GROUP BY day",
    output_table="default.daily_counts",
))

print(submitted.run_id, submitted.state)  # "..." SUBMITTED

Parameters

str

required

Job namespace.

str

required

Job name.

str

required

The Spark SQL query to execute.

str

required

The Iceberg table the query result is written to.

SparkWriteMode

default:"OVERWRITE"

OVERWRITE or APPEND. Case-insensitive on input.

SparkMachineType

default:"spark.1.b"

Driver machine type.

SparkMachineType

default:"spark.1.b"

Executor machine type.

int

default:"2"

Number of executors, from 1 to 20.

Return type: `SparkSqlSubmitResponse`

Field	Type	Description
`run_id`	`str`	The ID of the submitted run
`state`	`str`	Initial run state (for example, `SUBMITTED`)
`namespace`	`str`	Job namespace
`job_name`	`str`	Job name
`pipeline_id`	`Optional[str]`	Pipeline ID if one exists
`submitted_at`	`str`	ISO timestamp of submission
`compute`	`SparkSqlCompute`	`driver_machine_type`, `executor_machine_type`, `executor_numbers`
`output_table`	`str`	The output table
`write_mode`	`str`	The write mode used

`submit_spark_sql_and_wait(options)`

Submit a Spark SQL query and poll until the run reaches a terminal state (COMPLETE, FAIL, or ABORT). Raises TimeoutError if the timeout is exceeded.

from oleander_sdk import SubmitSparkSqlAndWaitOptions

result = await client.submit_spark_sql_and_wait(SubmitSparkSqlAndWaitOptions(
    namespace="my-namespace",
    name="nightly-agg",
    query="SELECT day, count(*) AS n FROM oleander.default.events GROUP BY day",
    output_table="default.daily_counts",
    write_mode="APPEND",
))

if result.state != "COMPLETE":
    raise Exception(f"Run {result.run_id} ended with state: {result.state}")

Accepts all submit_spark_sql options plus:

int

default:"10000"

Milliseconds between status polls. Must be greater than 0.

int

default:"600000"

Maximum time to wait in milliseconds before raising TimeoutError.

Return type: `SubmitAndWaitResult`

Field	Type	Description
`run_id`	`str`	The ID of the submitted run
`state`	`str`	Terminal state (`COMPLETE`, `FAIL`, or `ABORT`)
`run`	`RunResponse`	Full run details

TypeScript

Python

`submit_spark_sql(options)`

Parameters

Return type: `SparkSqlSubmitResponse`

`submit_spark_sql_and_wait(options)`

Return type: `SubmitAndWaitResult`

​submit_spark_sql(options)

​Parameters

​Return type: SparkSqlSubmitResponse

​submit_spark_sql_and_wait(options)

​Return type: SubmitAndWaitResult

`submit_spark_sql(options)`

Parameters

Return type: `SparkSqlSubmitResponse`

`submit_spark_sql_and_wait(options)`

Return type: `SubmitAndWaitResult`