Spark SQL - oleander

`submit_spark_sql(options)`

Submit a Spark SQL query that writes its result to an Iceberg output table. Returns run info immediately; use get_run() or submit_spark_sql_and_wait() to poll status.

from oleander_sdk import SparkSqlSubmitOptions

submitted = await client.submit_spark_sql(SparkSqlSubmitOptions(
    namespace="my-namespace",
    name="nightly-agg",
    query="SELECT day, count(*) AS n FROM oleander.default.events GROUP BY day",
    output_table="default.daily_counts",
))

print(submitted.run_id, submitted.state)  # "..." SUBMITTED

Parameters

options.namespace

str

required

Job namespace.

options.name

str

required

Job name.

options.query

str

required

The Spark SQL query to execute.

options.output_table

str

required

The Iceberg table the query result is written to.

options.write_mode

SparkWriteMode

default:"OVERWRITE"

OVERWRITE or APPEND. Case-insensitive on input.

options.driver_machine_type

SparkMachineType

default:"spark.1.b"

Driver machine type.

options.executor_machine_type

SparkMachineType

default:"spark.1.b"

Executor machine type.

options.executor_numbers

int

default:"2"

Number of executors, from 1 to 20.

Return type: `SparkSqlSubmitResponse`

Field	Type	Description
`run_id`	`str`	The ID of the submitted run
`state`	`str`	Initial run state (for example, `SUBMITTED`)
`namespace`	`str`	Job namespace
`job_name`	`str`	Job name
`pipeline_id`	`Optional[str]`	Pipeline ID if one exists
`submitted_at`	`str`	ISO timestamp of submission
`compute`	`SparkSqlCompute`	`driver_machine_type`, `executor_machine_type`, `executor_numbers`
`output_table`	`str`	The output table
`write_mode`	`str`	The write mode used

`submit_spark_sql_and_wait(options)`

Submit a Spark SQL query and poll until the run reaches a terminal state (COMPLETE, FAIL, or ABORT). Raises TimeoutError if the timeout is exceeded.

from oleander_sdk import SubmitSparkSqlAndWaitOptions

result = await client.submit_spark_sql_and_wait(SubmitSparkSqlAndWaitOptions(
    namespace="my-namespace",
    name="nightly-agg",
    query="SELECT day, count(*) AS n FROM oleander.default.events GROUP BY day",
    output_table="default.daily_counts",
    write_mode="APPEND",
))

if result.state != "COMPLETE":
    raise Exception(f"Run {result.run_id} ended with state: {result.state}")

Accepts all submit_spark_sql options plus:

options.poll_interval_ms

int

default:"10000"

Milliseconds between status polls. Must be greater than 0.

options.timeout_ms

int

default:"600000"

Maximum time to wait in milliseconds before raising TimeoutError.

Return type: `SubmitAndWaitResult`

Field	Type	Description
`run_id`	`str`	The ID of the submitted run
`state`	`str`	Terminal state (`COMPLETE`, `FAIL`, or `ABORT`)
`run`	`RunResponse`	Full run details

​submit_spark_sql(options)

​Parameters

​Return type: SparkSqlSubmitResponse

​submit_spark_sql_and_wait(options)

​Return type: SubmitAndWaitResult

`submit_spark_sql(options)`

Parameters

Return type: `SparkSqlSubmitResponse`

`submit_spark_sql_and_wait(options)`

Return type: `SubmitAndWaitResult`