{"record":{"id":"b3cf921128381a23","repo":"apache/beam","slug":"option-spark-rest-url-must-be-set","errorCode":null,"errorMessage":"Option spark_rest_url must be set.","messagePattern":"Option spark_rest_url must be set\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/portability/spark_runner.py","lineNumber":66,"sourceCode":"  spark master if one is not given.\n  \"\"\"\n\n  # Inherits run_portable_pipeline from PortableRunner.\n\n  def default_environment(self, options):\n    spark_options = options.view_as(pipeline_options.SparkRunnerOptions)\n    portable_options = options.view_as(pipeline_options.PortableOptions)\n    if (re.match(LOCAL_MASTER_PATTERN, spark_options.spark_master_url) and\n        not portable_options.environment_type and\n        not portable_options.output_executable_path):\n      portable_options.environment_type = 'LOOPBACK'\n    return super().default_environment(options)\n\n  def default_job_server(self, options):\n    spark_options = options.view_as(pipeline_options.SparkRunnerOptions)\n    if spark_options.spark_submit_uber_jar:\n      if not spark_options.spark_rest_url:\n        raise ValueError('Option spark_rest_url must be set.')\n      return spark_uber_jar_job_server.SparkUberJarJobServer(\n          spark_options.spark_rest_url, options)\n    # Use Java job server by default.\n    # Only SparkRunnerOptions and JobServerOptions affect job server\n    # configuration, so concat those as the cache key.\n    job_server_options = options.view_as(pipeline_options.JobServerOptions)\n    options_str = str(spark_options) + str(job_server_options)\n    if not options_str in JOB_SERVER_CACHE:\n      JOB_SERVER_CACHE[options_str] = job_server.StopOnExitJobServer(\n          SparkJarJobServer(options))\n    return JOB_SERVER_CACHE[options_str]\n\n  def create_job_service_handle(self, job_service, options):\n    return portable_runner.JobServiceHandle(\n        job_service,\n        options,\n        retain_unknown_options=options.view_as(\n            pipeline_options.SparkRunnerOptions).spark_submit_uber_jar)","sourceCodeStart":48,"sourceCodeEnd":84,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/portability/spark_runner.py#L48-L84","documentation":"Apache Beam's SparkRunner requires a REST URL when launching via a pre-built spark-submit uber jar. Because the REST URL is the only way the driver communicates with the launched Spark job in uber-jar mode, Beam refuses to start without it and raises ValueError early in default_job_server.","triggerScenarios":"Running with --runner=SparkRunner plus --spark_submit_uber_jar while omitting --spark_rest_url; programmatically calling default_job_server on PipelineOptions whose SparkRunnerOptions has spark_submit_uber_jar truthy but spark_rest_url None.","commonSituations":"Users enable the uber-jar deployment mode (common on clusters where Gradle-built job servers are inconvenient) but forget the required --spark_rest_url flag pointing at the Spark cluster's REST submission endpoint (e.g. http://spark-host:6066).","solutions":["Pass --spark_rest_url=<scheme>://<spark-master-host>:6066 (or the cluster's REST submission URL) alongside --spark_submit_uber_jar.","If you did not intend uber-jar mode, remove --spark_submit_uber_jar so Beam uses the default Java job server.","Set spark_rest_url programmatically: options.view_as(SparkRunnerOptions).spark_rest_url = 'http://host:6066' before creating the runner.","Verify the REST port is exposed by the Spark cluster (spark.master must be a REST-capable standalone/YARN master)."],"exampleFix":"// before\n--runner=SparkRunner --spark_submit_uber_jar\n// after\n--runner=SparkRunner --spark_submit_uber_jar --spark_rest_url=http://spark-master:6066","handlingStrategy":"validation","validationCode":"from apache_beam.options.pipeline_options import PipelineOptions, SparkRunnerOptions\nopts = PipelineOptions(pipeline_args)\nspark = opts.view_as(SparkRunnerOptions)\nif spark.spark_submit_uber_jar and not spark.spark_rest_url:\n    raise SystemExit('--spark_rest_url is required with --spark_submit_uber_jar')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pair --spark_submit_uber_jar with --spark_rest_url in launcher scripts","Add a preflight check in CI before submitting","Document the required REST port (usually 6066) in deployment runbooks"],"tags":["apache-beam","spark","configuration","cli"],"backgroundTag":"missing-required-option","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}