apache/beam · error · ValueError

Support for Spark 2 was dropped.

Error message

Support for Spark 2 was dropped.

What it means

When SparkUberJarJobServer must fall back to Beam's own job server jar, it checks the configured Spark version; version '2' is no longer supported, so Beam raises ValueError instead of trying to fetch a nonexistent Spark 2 artifact.

Solutions

1. Set spark_version to '3' (or leave default) and run against a Spark 3 cluster.
2. Provide the uber jar explicitly so the version-dependent branch is not taken.
3. If you must keep Spark 2, downgrade Beam to a release that still supported it.
4. Audit pipeline options/CI scripts for hardcoded spark_version=2.

Example fix

// before
options.view_as(SparkRunnerOptions).spark_version = '2'
// after
options.view_as(SparkRunnerOptions).spark_version = '3'

if server._spark_version == '2':
    raise SystemExit('Provide a Spark 3 uber jar or set spark_version=3')

When it happens

Trigger: Creating SparkUberJarJobServer without an explicit executable jar while _spark_version == '2'; invocation flows through create_beam_job when the pipeline starts.

Common situations: Pipelines still pointed at Spark 2 clusters after upgrading Beam; spark_version inferred or set from legacy environment/config; forgotten migration step.

Understand the failure class

Background: "unsupported platform" / "not supported on this platform" errors: what they mean and how to fix them — this error's family across 47 libraries.

Related errors

• Support for Spark 2 was dropped.
• Option spark_rest_url must be set.
• producer_properties is no longer supported and will be…
• Property "version" not found in…
• Request to failed with status

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/45cdf1d738bebf80. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py:78

  def stop(self):
    pass

  def executable_jar(self):
    if self._executable_jar:
      if not os.path.exists(self._executable_jar):
        parsed = urllib.parse.urlparse(self._executable_jar)
        if not parsed.scheme:
          raise ValueError(
              'Unable to parse jar URL "%s". If using a full URL, make sure '
              'the scheme is specified. If using a local file path, make sure '
              'the file exists; you may have to first build the job server '
              'using `./gradlew runners:spark:3:job-server:shadowJar`.' %
              self._executable_jar)
      url = self._executable_jar
    else:
      if self._spark_version == '2':
        raise ValueError('Support for Spark 2 was dropped.')
      else:
        url = job_server.JavaJarJobServer.path_to_beam_jar(
            ':runners:spark:3:job-server:shadowJar')
    return job_server.JavaJarJobServer.local_jar(
        url, user_agent=self._user_agent)

  def create_beam_job(self, job_id, job_name, pipeline, options):
    return SparkBeamJob(
        self._rest_url,
        self.executable_jar(),
        job_id,
        job_name,
        pipeline,
        options,
        artifact_port=self._artifact_port)


class SparkBeamJob(abstract_job_service.UberJarBeamJob):

View on GitHub (pinned to 12126d8942)