{"record":{"id":"fb60d9b8e53527a5","repo":"apache/beam","slug":"support-for-spark-2-was-dropped","errorCode":null,"errorMessage":"Support for Spark 2 was dropped.","messagePattern":"Support for Spark 2 was dropped\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/portability/spark_runner.py","lineNumber":113,"sourceCode":"    for arg in SPARK_JAR_JOB_SERVER_JVM_ARGS:\n      if arg not in self._jvm_properties:\n        self._jvm_properties.append(arg)\n\n  def path_to_jar(self):\n    if self._jar:\n      if not os.path.exists(self._jar):\n        url = urllib.parse.urlparse(self._jar)\n        if not url.scheme:\n          raise ValueError(\n              'Unable to parse jar URL \"%s\". If using a full URL, make sure '\n              'the scheme is specified. If using a local file path, make sure '\n              'the file exists; you may have to first build the job server '\n              'using `./gradlew runners:spark:3:job-server:shadowJar`.' %\n              self._jar)\n      return self._jar\n    else:\n      if self._spark_version == '2':\n        raise ValueError('Support for Spark 2 was dropped.')\n      return self.path_to_beam_jar(':runners:spark:3:job-server:shadowJar')\n\n  def java_arguments(\n      self, job_port, artifact_port, expansion_port, artifacts_dir):\n    return [\n        '--spark-master-url',\n        self._master_url,\n        '--artifacts-dir',\n        artifacts_dir,\n        '--job-port',\n        job_port,\n        '--artifact-port',\n        artifact_port,\n        '--expansion-port',\n        expansion_port\n    ]\n","sourceCodeStart":95,"sourceCodeEnd":130,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/portability/spark_runner.py#L95-L130","documentation":"Beam dropped support for Spark 2; when no explicit job server jar is supplied, SparkRunner derives one from the spark_version option. If that version is '2', Beam raises ValueError because no Spark 2 job server artifact exists anymore.","triggerScenarios":"Using the default jar selection (no --spark_job_server_jar) with --spark_version=2 (or spark_version set to '2' programmatically) on SparkRunner.","commonSituations":"Migrating legacy pipelines from older Beam versions that supported Spark 2; environments still running Spark 2.x clusters; copied pipeline options from an old project.","solutions":["Upgrade the Spark cluster to Spark 3.x and pass --spark_version=3 (or omit; 3 is the default).","Supply a Spark 3 job server jar explicitly with --spark_job_server_jar so the version branch is skipped.","If Spark 2 is unavoidable, pin an older Beam release that still supported Spark 2 (pre Beam 2.37-ish) — not recommended.","Remove any stale spark_version=2 setting from pipeline options or config files."],"exampleFix":"// before\n--spark_version=2\n// after\n--spark_version=3","handlingStrategy":"validation","validationCode":"spark = options.view_as(SparkRunnerOptions)\nif getattr(spark, 'spark_version', '3') == '2':\n    raise SystemExit('Spark 2 is unsupported; upgrade the cluster to Spark 3.x')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin spark_version=3 in pipeline option templates","Plan cluster upgrades before bumping apache-beam","Grep CI configs for spark_version=2"],"tags":["apache-beam","spark","versioning","deprecation"],"backgroundTag":"unsupported-platform","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}