apache/beam · error · RuntimeError

The Kafka Streams runner is experimental and is not part of…

Error message

The Kafka Streams runner is experimental and is not part of any Apache Beam release, so there is no published job server jar to download. Build one from a Beam source tree with
  ./gradlew -Pwith-kafka-streams-runner :runners:kafka-streams:job-server:shadowJar
and pass it as --kafka_streams_job_server_jar, or point that option at a jar you already have. The runner is opt-in at build time, so the -Pwith-kafka-streams-runner flag is required; without it the runner is not part of the build at all.

What it means

The Kafka Streams runner is experimental and not part of any released Beam artifact, so the normal 'download the Beam job server jar' path always fails. When no locally built shadowJar exists either, path_to_jar raises this RuntimeError explaining that the jar must be built from source with the opt-in Gradle flag.

Solutions

  1. Build from a Beam source tree: ./gradlew -Pwith-kafka-streams-runner :runners:kafka-streams:job-server:shadowJar.
  2. Pass the built jar explicitly with --kafka_streams_job_server_jar=/path/to/shadowJar.
  3. Alternatively point the option at any Kafka Streams job server jar you already have.

Example fix

// before
beam run --runner=KafkaStreamsRunner ...
// after
./gradlew -Pwith-kafka-streams-runner :runners:kafka-streams:job-server:shadowJar
beam run ... --kafka_streams_job_server_jar=runners/kafka-streams/job-server/build/libs/beam-runners-kafka-streams-job-server-<ver>-SNAPSHOT.jar
Defensive patterns

Strategy: validation

Validate before calling

local = subprocess_server.JavaJarServer.path_to_dev_beam_jar(
    ':runners:kafka-streams:job-server:shadowJar')
if not (options.kafka_streams_job_server_jar or os.path.exists(local)):
    raise SystemExit('build the kafka-streams job server jar first')

Try / catch

try:
    jar = runner.path_to_jar()
except RuntimeError:
    subprocess.run(['./gradlew', '-Pwith-kafka-streams-runner',
                    ':runners:kafka-streams:job-server:shadowJar'], check=True)

Prevention

When it happens

Trigger: Running a Kafka Streams pipeline without --kafka_streams_job_server_jar, where the dev jar path produced by JavaJarServer.path_to_dev_beam_jar(':runners:kafka-streams:job-server:shadowJar') does not exist on disk.

Common situations: Using pip-installed Apache Beam (no Beam source tree, so no dev jar); building Beam without -Pwith-kafka-streams-runner so the jar was never produced; assuming a published job server exists for releases.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/398dc2daf47713b0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/kafka_streams_runner.py:115

        url = urllib.parse.urlparse(self._jar)
        if not url.scheme:
          raise ValueError(
              'Unable to parse jar URL "%s". If using a full URL, make sure '
              'the scheme is specified. If using a local file path, make sure '
              'the file exists; you may have to first build the job server '
              'using `./gradlew -Pwith-kafka-streams-runner '
              'runners:kafka-streams:job-server:shadowJar`.' % self._jar)
      return self._jar

    # No jar was given, so look for one built from this source tree. The base
    # class would fall back to Maven Central, but the job server is not
    # published for any Beam release, so that download always fails and says
    # nothing useful about why.
    local_jar = subprocess_server.JavaJarServer.path_to_dev_beam_jar(
        ':runners:kafka-streams:job-server:shadowJar')
    if os.path.exists(local_jar):
      return local_jar
    raise RuntimeError(
        'The Kafka Streams runner is experimental and is not part of any '
        'Apache Beam release, so there is no published job server jar to '
        'download. Build one from a Beam source tree with\n'
        '  ./gradlew -Pwith-kafka-streams-runner '
        ':runners:kafka-streams:job-server:shadowJar\n'
        'and pass it as --kafka_streams_job_server_jar, or point that option '
        'at a jar you already have. The runner is opt-in at build time, so '
        'the -Pwith-kafka-streams-runner flag is required; without it the '
        'runner is not part of the build at all.')

  def java_arguments(
      self, job_port, artifact_port, expansion_port, artifacts_dir):
    return [
        '--artifacts-dir',
        artifacts_dir,
        '--job-port',
        job_port,
        '--artifact-port',

View on GitHub (pinned to 12126d8942)