apache/beam · error · ValueError

Option spark_rest_url must be set.

Error message

Option spark_rest_url must be set.

What it means

Apache Beam's SparkRunner requires a REST URL when launching via a pre-built spark-submit uber jar. Because the REST URL is the only way the driver communicates with the launched Spark job in uber-jar mode, Beam refuses to start without it and raises ValueError early in default_job_server.

Solutions

1. Pass --spark_rest_url=<scheme>://<spark-master-host>:6066 (or the cluster's REST submission URL) alongside --spark_submit_uber_jar.
2. If you did not intend uber-jar mode, remove --spark_submit_uber_jar so Beam uses the default Java job server.
3. Set spark_rest_url programmatically: options.view_as(SparkRunnerOptions).spark_rest_url = 'http://host:6066' before creating the runner.
4. Verify the REST port is exposed by the Spark cluster (spark.master must be a REST-capable standalone/YARN master).

Example fix

// before
--runner=SparkRunner --spark_submit_uber_jar
// after
--runner=SparkRunner --spark_submit_uber_jar --spark_rest_url=http://spark-master:6066

from apache_beam.options.pipeline_options import PipelineOptions, SparkRunnerOptions
opts = PipelineOptions(pipeline_args)
spark = opts.view_as(SparkRunnerOptions)
if spark.spark_submit_uber_jar and not spark.spark_rest_url:
    raise SystemExit('--spark_rest_url is required with --spark_submit_uber_jar')

When it happens

Trigger: Running with --runner=SparkRunner plus --spark_submit_uber_jar while omitting --spark_rest_url; programmatically calling default_job_server on PipelineOptions whose SparkRunnerOptions has spark_submit_uber_jar truthy but spark_rest_url None.

Common situations: Users enable the uber-jar deployment mode (common on clusters where Gradle-built job servers are inconvenient) but forget the required --spark_rest_url flag pointing at the Spark cluster's REST submission endpoint (e.g. http://spark-host:6066).

Understand the failure class

Background: "Must pass :limit option" / "Missing required option" — required option errors explained — this error's family across 41 libraries.

Related errors

• A schema was provided without a data format (or viceversa)…
• At least one of --render_port or --render_output must be…
• At most one of --create_test and --fix_tests may be…
• Batch size is too large! It should be smaller or equal than
• boolean cross product parameter required to explode more…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b3cf921128381a23. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/spark_runner.py:66

  spark master if one is not given.
  """

  # Inherits run_portable_pipeline from PortableRunner.

  def default_environment(self, options):
    spark_options = options.view_as(pipeline_options.SparkRunnerOptions)
    portable_options = options.view_as(pipeline_options.PortableOptions)
    if (re.match(LOCAL_MASTER_PATTERN, spark_options.spark_master_url) and
        not portable_options.environment_type and
        not portable_options.output_executable_path):
      portable_options.environment_type = 'LOOPBACK'
    return super().default_environment(options)

  def default_job_server(self, options):
    spark_options = options.view_as(pipeline_options.SparkRunnerOptions)
    if spark_options.spark_submit_uber_jar:
      if not spark_options.spark_rest_url:
        raise ValueError('Option spark_rest_url must be set.')
      return spark_uber_jar_job_server.SparkUberJarJobServer(
          spark_options.spark_rest_url, options)
    # Use Java job server by default.
    # Only SparkRunnerOptions and JobServerOptions affect job server
    # configuration, so concat those as the cache key.
    job_server_options = options.view_as(pipeline_options.JobServerOptions)
    options_str = str(spark_options) + str(job_server_options)
    if not options_str in JOB_SERVER_CACHE:
      JOB_SERVER_CACHE[options_str] = job_server.StopOnExitJobServer(
          SparkJarJobServer(options))
    return JOB_SERVER_CACHE[options_str]

  def create_job_service_handle(self, job_service, options):
    return portable_runner.JobServiceHandle(
        job_service,
        options,
        retain_unknown_options=options.view_as(
            pipeline_options.SparkRunnerOptions).spark_submit_uber_jar)

View on GitHub (pinned to 12126d8942)