apache/beam · error · RuntimeError

Service failed to start up with error %s

Error message

Service failed to start up with error %s

What it means

start() launches the Beam Java/Flink job service subprocess and waits for its gRPC channel to become ready. If the process exits (poll() is not None) before the channel is ready, the loop aborts and raises with the process exit code. This means the server died during startup rather than merely being slow.

Source

Thrown at sdks/python/apache_beam/utils/subprocess_server.py:229

            ("grpc.max_send_message_length", -1),
            # Default: 20000ms (20s), increased to 10 minutes for stability
            ("grpc.keepalive_timeout_ms", 600_000),
            # Default: 2, set to 0 to allow unlimited pings without data
            ("grpc.http2.max_pings_without_data", 0),
            # Default: False, set to True to allow keepalive pings when no calls
            ("grpc.keepalive_permit_without_calls", True),
            # Default: 2, set to 0 to allow unlimited ping strikes
            ("grpc.http2.max_ping_strikes", 0),
            # Default: 0 (disabled), enable socket reuse for better handling
            ("grpc.so_reuseport", 1),
        ]
        self._grpc_channel = grpc.insecure_channel(
            endpoint, options=channel_options)
        channel_ready = grpc.channel_ready_future(self._grpc_channel)
        while True:
          if process is not None and process.poll() is not None:
            _LOGGER.error("Failed to start job service with %s", process.args)
            raise RuntimeError(
                'Service failed to start up with error %s' % process.poll())
          try:
            channel_ready.result(timeout=wait_secs)
            break
          except (grpc.FutureTimeoutError, grpc.RpcError):
            wait_secs *= 1.2
            logging.log(
                logging.WARNING if wait_secs > 1 else logging.DEBUG,
                'Waiting for grpc channel to be ready at %s.',
                endpoint)
        return self._stub_class(self._grpc_channel)
      except Exception as e:
        _LOGGER.warning(
            "Error bringing up service (attempt %d of %d): %s",
            attempt + 1,
            max_retries,
            e)
        self.stop_force()

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the surrounding _LOGGER.error('Failed to start job service with %s', process.args) output and the subprocess stdout/stderr to find the actual JVM failure.
2. Verify the jar exists and matches the Beam version; run it manually with java -jar to see the error.
3. Check JAVA_HOME / java -version compatibility with the Beam job server.
4. Free the port or fix gRPC endpoint configuration; check available memory and container limits.
5. Retry the start — transient resource contention can kill startup.

Example fix

// before
with subprocess_server.JavaJarServer.create('flink', jar_path, []) as srv:
    ...
// after
import subprocess, shutil
subprocess.run(['java', '-jar', jar_path, '--help'], check=True)  # surface JVM errors first
with subprocess_server.JavaJarServer.create('flink', jar_path, []) as srv:
    ...

import subprocess
subprocess.run(['java', '-version'], capture_output=True, check=True)  # JVM sanity check before start()

try:
    server.start()
except RuntimeError as e:
    if 'Service failed to start up' in str(e):
        inspect_process_stderr_and_retry()
    else:
        raise

When it happens

Trigger: The spawned java subprocess terminates during startup — bad JVM flags, missing/corrupt jar, port conflicts, OOM kill, or incompatible Java version — detected via process.poll() while polling grpc.channel_ready_future().

Common situations: JAVA_HOME pointing to an incompatible JDK; invalid channel options; a crashed or outdated job-server jar; container memory limits killing the JVM; port already in use.

Related errors

• input is already set.
• Buffer size {size} exceeds GRPC limit {_FLUSH_MAX_SIZE}. Thi
• Header metadata already has a worker_id.
• Returning elements from _SubprocessDoFn.finish_bundle not sa
• Transform service did not start in %s seconds.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/22107a3412a267ef. Report an issue: GitHub.