{"record":{"id":"45cdf1d738bebf80","repo":"apache/beam","slug":"support-for-spark-2-was-dropped-uber-jar-job-server","errorCode":null,"errorMessage":"Support for Spark 2 was dropped.","messagePattern":"Support for Spark 2 was dropped\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py","lineNumber":78,"sourceCode":"\n  def stop(self):\n    pass\n\n  def executable_jar(self):\n    if self._executable_jar:\n      if not os.path.exists(self._executable_jar):\n        parsed = urllib.parse.urlparse(self._executable_jar)\n        if not parsed.scheme:\n          raise ValueError(\n              'Unable to parse jar URL \"%s\". If using a full URL, make sure '\n              'the scheme is specified. If using a local file path, make sure '\n              'the file exists; you may have to first build the job server '\n              'using `./gradlew runners:spark:3:job-server:shadowJar`.' %\n              self._executable_jar)\n      url = self._executable_jar\n    else:\n      if self._spark_version == '2':\n        raise ValueError('Support for Spark 2 was dropped.')\n      else:\n        url = job_server.JavaJarJobServer.path_to_beam_jar(\n            ':runners:spark:3:job-server:shadowJar')\n    return job_server.JavaJarJobServer.local_jar(\n        url, user_agent=self._user_agent)\n\n  def create_beam_job(self, job_id, job_name, pipeline, options):\n    return SparkBeamJob(\n        self._rest_url,\n        self.executable_jar(),\n        job_id,\n        job_name,\n        pipeline,\n        options,\n        artifact_port=self._artifact_port)\n\n\nclass SparkBeamJob(abstract_job_service.UberJarBeamJob):","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py#L60-L96","documentation":"When SparkUberJarJobServer must fall back to Beam's own job server jar, it checks the configured Spark version; version '2' is no longer supported, so Beam raises ValueError instead of trying to fetch a nonexistent Spark 2 artifact.","triggerScenarios":"Creating SparkUberJarJobServer without an explicit executable jar while _spark_version == '2'; invocation flows through create_beam_job when the pipeline starts.","commonSituations":"Pipelines still pointed at Spark 2 clusters after upgrading Beam; spark_version inferred or set from legacy environment/config; forgotten migration step.","solutions":["Set spark_version to '3' (or leave default) and run against a Spark 3 cluster.","Provide the uber jar explicitly so the version-dependent branch is not taken.","If you must keep Spark 2, downgrade Beam to a release that still supported it.","Audit pipeline options/CI scripts for hardcoded spark_version=2."],"exampleFix":"// before\noptions.view_as(SparkRunnerOptions).spark_version = '2'\n// after\noptions.view_as(SparkRunnerOptions).spark_version = '3'","handlingStrategy":"validation","validationCode":"if server._spark_version == '2':\n    raise SystemExit('Provide a Spark 3 uber jar or set spark_version=3')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Standardize on Spark 3.x across environments","Always pass an explicit uber jar to avoid version-dependent fallbacks","Track Beam release notes for dropped Spark support"],"tags":["apache-beam","spark","versioning","migration"],"backgroundTag":"unsupported-platform","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}