What causes "Error starting worker: %s" in apache/beam?

Read the %s payload in the message — it contains the harness's own failure reason and fix that underlying issue.

apache/beam · error · RuntimeError

Error starting worker

Error message

Error starting worker: %s

What it means

Raised by the Embedded/external worker handler after sending a ProvisionInfo request to an externally managed (Docker-based) SDK harness: the harness responded with a non-empty error field, meaning it failed to start itself. The runner surfaces the harness's own error text verbatim via RuntimeError.

Solutions

Read the %s payload in the message — it contains the harness's own failure reason and fix that underlying issue.
Verify the container can reach the runner's control/logging endpoints (shared docker network, correct host binding).
Pin --sdk_harness_container_image_overrides to an image matching the runner's Beam version.
Check container logs (docker logs <container>) for startup/dependency errors.

Defensive patterns

Strategy: try-catch

Validate before calling

if response and getattr(response, 'error', None):
    precheck_failed = True

Try / catch

try:
    handler.start_worker()
except RuntimeError as e:
    log.error('harness provision failed: %s', e)
    inspect_container_logs()

Prevention

Use the same docker network for runner and containers
Verify endpoint reachability before start
Pin harness image versions

When it happens

Trigger: Calling start_worker on an external worker handler whose container started but the harness process inside replied with error to the provision request — e.g. bad pipeline options, missing dependencies in the container, or wrong endpoints.

Common situations: Docker image lacking required Python packages; container cannot reach the control/logging endpoints (wrong docker network); incompatible harness image version; invalid environment passed via provision payload.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c8c7f18b916a77c8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py:643

  def start_worker(self):
    # type: () -> None
    _LOGGER.info("Requesting worker at %s", self._external_payload.endpoint.url)
    stub = beam_fn_api_pb2_grpc.BeamFnExternalWorkerPoolStub(
        GRPCChannelFactory.insecure_channel(
            self._external_payload.endpoint.url))
    _LOGGER.info('self.control_address: %s' % self.control_address)
    control_descriptor = endpoints_pb2.ApiServiceDescriptor(
        url=self.control_address)
    response = stub.StartWorker(
        beam_fn_api_pb2.StartWorkerRequest(
            worker_id=self.worker_id,
            control_endpoint=control_descriptor,
            artifact_endpoint=control_descriptor,
            provision_endpoint=control_descriptor,
            logging_endpoint=self.logging_api_service_descriptor(),
            params=self._external_payload.params))
    if response.error:
      raise RuntimeError("Error starting worker: %s" % response.error)

  def stop_worker(self):
    # type: () -> None
    pass

  def host_from_worker(self):
    # type: () -> str
    # TODO(https://github.com/apache/beam/issues/19947): Reconcile across
    # platforms.
    if sys.platform in ['win32', 'darwin']:
      return 'localhost'
    import socket
    return socket.getfqdn()


@WorkerHandler.register_environment(python_urns.EMBEDDED_PYTHON_GRPC, bytes)
class EmbeddedGrpcWorkerHandler(GrpcWorkerHandler):
  def __init__(

View on GitHub (pinned to 12126d8942)