apache/beam · error · ValueError

Support for Spark 2 was dropped.

Error message

Support for Spark 2 was dropped.

What it means

Beam dropped support for Spark 2; when no explicit job server jar is supplied, SparkRunner derives one from the spark_version option. If that version is '2', Beam raises ValueError because no Spark 2 job server artifact exists anymore.

Solutions

1. Upgrade the Spark cluster to Spark 3.x and pass --spark_version=3 (or omit; 3 is the default).
2. Supply a Spark 3 job server jar explicitly with --spark_job_server_jar so the version branch is skipped.
3. If Spark 2 is unavoidable, pin an older Beam release that still supported Spark 2 (pre Beam 2.37-ish) — not recommended.
4. Remove any stale spark_version=2 setting from pipeline options or config files.

Example fix

// before
--spark_version=2
// after
--spark_version=3

spark = options.view_as(SparkRunnerOptions)
if getattr(spark, 'spark_version', '3') == '2':
    raise SystemExit('Spark 2 is unsupported; upgrade the cluster to Spark 3.x')

When it happens

Trigger: Using the default jar selection (no --spark_job_server_jar) with --spark_version=2 (or spark_version set to '2' programmatically) on SparkRunner.

Common situations: Migrating legacy pipelines from older Beam versions that supported Spark 2; environments still running Spark 2.x clusters; copied pipeline options from an old project.

Understand the failure class

Background: "unsupported platform" / "not supported on this platform" errors: what they mean and how to fix them — this error's family across 47 libraries.

Related errors

• Support for Spark 2 was dropped.
• chunk_to_dict_fn is deprecated, use embeddable_to_dict_fn
• Dicom Client moved to…
• Dicom IO moved to apache_beam.io.gcp.healthcare.dicomcio
• Kafka transforms moved to apache_beam.io.kafka

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fb60d9b8e53527a5. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/spark_runner.py:113

    for arg in SPARK_JAR_JOB_SERVER_JVM_ARGS:
      if arg not in self._jvm_properties:
        self._jvm_properties.append(arg)

  def path_to_jar(self):
    if self._jar:
      if not os.path.exists(self._jar):
        url = urllib.parse.urlparse(self._jar)
        if not url.scheme:
          raise ValueError(
              'Unable to parse jar URL "%s". If using a full URL, make sure '
              'the scheme is specified. If using a local file path, make sure '
              'the file exists; you may have to first build the job server '
              'using `./gradlew runners:spark:3:job-server:shadowJar`.' %
              self._jar)
      return self._jar
    else:
      if self._spark_version == '2':
        raise ValueError('Support for Spark 2 was dropped.')
      return self.path_to_beam_jar(':runners:spark:3:job-server:shadowJar')

  def java_arguments(
      self, job_port, artifact_port, expansion_port, artifacts_dir):
    return [
        '--spark-master-url',
        self._master_url,
        '--artifacts-dir',
        artifacts_dir,
        '--job-port',
        job_port,
        '--artifact-port',
        artifact_port,
        '--expansion-port',
        expansion_port
    ]

View on GitHub (pinned to 12126d8942)