> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oleander.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Spark

> Initialize workspaces, upload artifacts, submit jobs, and register clusters from the CLI.

Run your Spark applications on oleander-managed infrastructure or on your own registered clusters. For a full overview of Spark on oleander, see the [Spark documentation](/platform/spark/jobs).

## Oleander-managed Spark

### Initialize a PySpark workspace

Create a new PySpark job workspace:

```bash theme={null}
oleander spark init <dirname>
```

**Example:**

```bash theme={null}
oleander spark init my-job
```

You can also initialize the current directory:

```bash theme={null}
oleander spark init .
```

Pass flags to skip the interactive prompts:

```bash theme={null}
oleander spark init my-job \
  --entrypoint-name pipeline.py \
  --custom-package-name mylib
```

| Flag                           | Description                                             |
| ------------------------------ | ------------------------------------------------------- |
| `--entrypoint-name <file>`     | Name for the main entrypoint file (must end with `.py`) |
| `--custom-package-name <name>` | Name for the custom Python package directory            |

The initialized workspace includes:

* `entrypoint.py` as the Spark job entrypoint
* `mylib/` for Python modules packaged as `pyFiles`
* `pyproject.toml` and `uv.lock` for project and dependency management with `uv`
* `Makefile` targets for building deployable artifacts

Use `uv` to manage dependencies:

```bash theme={null}
uv sync --dev
uv add <package>
uv add --dev <package>
```

Use `make` to build the deployment artifacts:

```bash theme={null}
make
```

This builds:

* `out/pyfiles.zip`
* `out/environment.tar.gz`

You can also build individual artifacts:

```bash theme={null}
make pyfiles
make environment
make rebuild
```

After building, upload and submit from the initialized workspace:

```bash theme={null}
oleander spark jobs upload entrypoint.py \
  --py-files out/pyfiles.zip \
  --virtualenv out/environment.tar.gz
```

```bash theme={null}
oleander spark jobs submit entrypoint.py \
  --namespace <namespace> \
  --name <job-name> \
  --wait
```

### List your Spark artifacts

```bash theme={null}
oleander spark jobs list
```

### Download a Spark artifact

Print the source of an uploaded artifact to stdout:

```bash theme={null}
oleander spark jobs get <entrypoint>
```

To fetch a specific version:

```bash theme={null}
oleander spark jobs get <entrypoint> --artifact-version 3
```

| Flag                     | Description                                     |
| ------------------------ | ----------------------------------------------- |
| `--artifact-version <n>` | Version number to retrieve (defaults to latest) |

### Upload a Spark artifact

Upload a local `.py` or `.jar` artifact to oleander:

```bash theme={null}
oleander spark jobs upload <your_artifact_path>
```

**Example:**

```bash theme={null}
oleander spark jobs upload ./transformations/process_sales_data.py
```

**JAR example:**

```bash theme={null}
oleander spark jobs upload ./jobs/process_sales_data.jar
```

Every upload creates a new artifact version on the backend.

Upload options:

| Flag           | Description                                                                    |
| -------------- | ------------------------------------------------------------------------------ |
| `--namespace`  | Namespace used when uploading. Defaults to `default`.                          |
| `--py-files`   | Local `.zip` or `.egg` dependency archive uploaded alongside a Python artifact |
| `--virtualenv` | Local virtual environment archive uploaded alongside a Python artifact         |
| `--dry-run`    | Show what would be uploaded without uploading anything                         |

### Delete a Spark artifact

```bash theme={null}
oleander spark jobs delete <artifact_name>
```

Use the exact uploaded artifact name, including the file extension.

**Example:**

```bash theme={null}
oleander spark jobs delete process_sales_data.py
```

### Submit a job

Submit an uploaded artifact to the oleander-managed cluster. Use the uploaded artifact name as the `entrypoint`.

```bash theme={null}
oleander spark jobs submit <entrypoint> --namespace <namespace> --name <run_name> --wait
```

**Example:**

```bash theme={null}
oleander spark jobs submit process_sales_data.py --namespace finance --name process-sales-data --wait
```

Common submit options:

| Flag          | Description                                                         |
| ------------- | ------------------------------------------------------------------- |
| `--cluster`   | Cluster name. Defaults to the oleander-managed cluster when omitted |
| `--namespace` | Required job namespace                                              |
| `--name`      | Required job name                                                   |
| `--args`      | Entrypoint arguments                                                |
| `--sparkConf` | Spark configurations, for example `spark.default.parallelism=8`     |
| `--packages`  | Extra package coordinates                                           |
| `--jobTags`   | Job-specific tags                                                   |
| `--runTags`   | Run-specific tags                                                   |
| `--wait`      | Block until the job finishes                                        |

Oleander-managed submit options:

| Flag                    | Description                  |
| ----------------------- | ---------------------------- |
| `--driverMachineType`   | Driver machine type          |
| `--executorMachineType` | Executor machine type        |
| `--executorNumbers`     | Number of executor instances |

### Stop a running job

Abort a run that is currently in progress:

```bash theme={null}
oleander spark jobs stop <run_id>
```

The run ID is returned when you submit a job and is also visible in the oleander UI.

***

## Registered clusters

### List registered clusters

```bash theme={null}
oleander spark clusters list
```

## Registered EMR Serverless

Register your EMR Serverless cluster, then target it by name when submitting jobs.

### Register a cluster

```bash theme={null}
oleander spark clusters register <name> \
  --type emr-serverless \
  --region <region> \
  --account-id <aws_account_id> \
  --controller-role-arn <controller_role_arn> \
  --execution-role-arn <execution_role_arn> \
  --application-id <application_id> \
  --log-bucket <log_bucket>
```

See the [Spark documentation](/platform/spark/jobs#registered-emr-serverless-spark) for IAM policy details.

### Submit a job

Use an EMR-compatible entrypoint, such as an S3 URI to a `.py` or `.jar` artifact.

```bash theme={null}
oleander spark jobs submit <entrypoint> \
  --cluster <cluster_name> \
  --namespace <namespace> \
  --name <run_name> \
  --wait
```

**Example:**

```bash theme={null}
oleander spark jobs submit s3://my-bucket/jobs/process_sales_data.py --cluster my-emr --namespace finance --name process-sales-data --wait
```

Additional EMR Serverless flags:

| Flag                   | Description                                                                                            |
| ---------------------- | ------------------------------------------------------------------------------------------------------ |
| `--pyFiles`            | Zip archive of Python dependencies                                                                     |
| `--virtualenv`         | Virtual environment archive for Python jobs                                                            |
| `--mainClass`          | Main class for JVM jobs; use instead of Python-specific options such as `--pyFiles` and `--virtualenv` |
| `--executionIamPolicy` | IAM policy applied to execution                                                                        |

## Registered Glue

Register your Glue cluster, then target it by name.

### Register a cluster

```bash theme={null}
oleander spark clusters register <name> \
  --type glue \
  --controller-role-arn <controller_role_arn>
```

See the [Spark documentation](/platform/spark/jobs#registered-glue-spark) for IAM policy details.

### Submit a job

For Glue, the `entrypoint` is the Glue job name.

```bash theme={null}
oleander spark jobs submit <job_name> --cluster <cluster_name> --namespace <namespace> --name <run_name> --wait
```

Additional Glue flags:

| Flag                   | Description                                   |
| ---------------------- | --------------------------------------------- |
| `--workerType`         | Glue worker type                              |
| `--numberOfWorkers`    | Number of Glue workers                        |
| `--enableAutoScaling`  | Enable Glue auto scaling                      |
| `--timeoutMinutes`     | Timeout in minutes                            |
| `--executionClass`     | Execution class, such as `STANDARD` or `FLEX` |
| `--executionIamPolicy` | IAM policy applied to execution               |

<Note>
  For Glue jobs, `--args` must be passed as alternating key-value pairs, for example `--args source s3://bucket/in target s3://bucket/out`.
</Note>