apache/beam · critical · RuntimeError

Failed to start embedded etcd for Dynamo. Process status

Error message

Failed to start embedded etcd for Dynamo. Process status: {process_status}. Install etcd in the worker container or set ETCD_ENDPOINTS to an external etcd service.

What it means

vLLM inference with embedded Dynamo mode tries to start a local etcd instance for coordination. _wait_for_etcd raises RuntimeError if the etcd process never becomes ready, instructing the user to install etcd in the worker container or point ETCD_ENDPOINTS at an external etcd service.

Solutions

Install etcd in the worker container image and ensure it is on PATH
Set the ETCD_ENDPOINTS environment variable to an external etcd cluster (e.g. etcd1:2379,etcd2:2379) to skip embedded etcd
Check _process_status / container logs to see why etcd exited (port conflict, permissions, arch mismatch)

Example fix

# before (Dockerfile)
FROM nvidia/cuda:12.1-base
# after
FROM nvidia/cuda:12.1-base
RUN apt-get update && apt-get install -y etcd-server
ENV PATH="/usr/bin:${PATH}"

Defensive patterns

Strategy: validation

Validate before calling

import os, shutil
if not os.environ.get('ETCD_ENDPOINTS') and shutil.which('etcd') is None:
    raise RuntimeError('Embedded Dynamo needs etcd installed or ETCD_ENDPOINTS set')

Try / catch

try:
    server.start_server()
except RuntimeError as e:
    if 'embedded etcd' in str(e):
        install_or_configure_etcd()  # or set ETCD_ENDPOINTS and rebuild
    raise

Prevention

Bake etcd into worker images used for vLLM/Dynamo pipelines
Prefer external etcd via ETCD_ENDPOINTS in production
Check etcd binary compatibility (arch/OS) with the container

When it happens

Trigger: start_server in embedded Dynamo mode (ETCD_ENDPOINTS unset) where the spawned etcd binary crashes or never becomes healthy within the wait timeout: binary incompatible with the container OS/arch, missing permissions, or port conflicts.

Common situations: Running on Dataflow/custom worker images without etcd installed; etcd binary present but failing to start (bad arch, read-only /tmp, port already in use).

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/42e44ba595f43e92. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/inference/vllm_inference.py:242

        self._use_dynamo and
        _uses_etcd_discovery(self._dynamo_frontend_kwargs) and
        _uses_etcd_discovery(self._vllm_server_kwargs) and
        'ETCD_ENDPOINTS' not in os.environ)

  def _wait_for_etcd(self, endpoint: str, timeout_secs=30) -> None:
    deadline = time.time() + timeout_secs
    health_url = endpoint.rstrip('/') + '/health'
    while time.time() < deadline and self._etcd_process.poll() is None:
      try:
        with urllib.request.urlopen(health_url, timeout=2) as response:
          if response.status < 500:
            return
      except Exception:  # pylint: disable=broad-except
        time.sleep(1)

    process_status = self._process_status()
    self._stop_processes()
    raise RuntimeError(
        "Failed to start embedded etcd for Dynamo. Process status: "
        f"{process_status}. Install etcd in the worker container or set "
        "ETCD_ENDPOINTS to an external etcd service.")

  def _ensure_etcd(self) -> None:
    if not self._uses_embedded_etcd():
      return
    if shutil.which('etcd') is None:
      raise RuntimeError(
          "Embedded Dynamo mode requires etcd when ETCD_ENDPOINTS is not "
          "set. Install etcd in the worker container or set ETCD_ENDPOINTS "
          "to an external etcd service.")

    etcd_name = f'beam-dynamo-etcd-{uuid.uuid4().hex}'
    self._etcd_data_dir = f'/tmp/{etcd_name}'
    peer_port, = subprocess_server.pick_port(None)
    etcd_cmd = [
        'etcd',

View on GitHub (pinned to 12126d8942)