apache/beam · critical · RuntimeError

Embedded Dynamo mode requires etcd when ETCD_ENDPOINTS is…

Error message

Embedded Dynamo mode requires etcd when ETCD_ENDPOINTS is not set. Install etcd in the worker container or set ETCD_ENDPOINTS to an external etcd service.

What it means

_ensure_etcd checks that when embedded Dynamo mode is used (ETCD_ENDPOINTS not set), an etcd binary is available on PATH. If shutil.which('etcd') finds nothing, it raises RuntimeError telling you to install etcd or configure external endpoints.

Solutions

Install etcd in the worker container image (apt-get install etcd-server or equivalent)
Set ETCD_ENDPOINTS to a running external etcd service so embedded etcd is not needed
Verify etcd is on the PATH for the worker process (which etcd resolves)

Example fix

# before
export APACHE_BEAM_VLLM_DYNAMO=1  # embedded mode, no etcd
# after
export ETCD_ENDPOINTS=etcd-host:2379  # or install etcd in the image

Defensive patterns

Strategy: validation

Validate before calling

import os, shutil
if not os.environ.get('ETCD_ENDPOINTS') and shutil.which('etcd') is None:
    raise RuntimeError('Install etcd or set ETCD_ENDPOINTS before enabling embedded Dynamo mode')

Try / catch

try:
    server.start_server()
except RuntimeError as e:
    if 'requires etcd' in str(e):
        os.environ['ETCD_ENDPOINTS'] = 'etcd-host:2379'
    raise

Prevention

Run `which etcd` in the worker image during CI to catch missing binaries
Set ETCD_ENDPOINTS whenever etcd is provided externally
Keep a documented Dockerfile step installing etcd for vLLM workloads

When it happens

Trigger: Calling start_server with embedded Dynamo mode enabled in a container where the etcd binary is not installed and ETCD_ENDPOINTS is unset.

Common situations: Minimal worker images without etcd; relying on the host's etcd not present in the job container; forgetting to set ETCD_ENDPOINTS when etcd is provided externally.

Understand the failure class

Background: "not installed", "pip install", "required for": how missing-dependency errors surface across open-source libraries — this error's family across 34 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/85843d532474b93c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/inference/vllm_inference.py:251

      try:
        with urllib.request.urlopen(health_url, timeout=2) as response:
          if response.status < 500:
            return
      except Exception:  # pylint: disable=broad-except
        time.sleep(1)

    process_status = self._process_status()
    self._stop_processes()
    raise RuntimeError(
        "Failed to start embedded etcd for Dynamo. Process status: "
        f"{process_status}. Install etcd in the worker container or set "
        "ETCD_ENDPOINTS to an external etcd service.")

  def _ensure_etcd(self) -> None:
    if not self._uses_embedded_etcd():
      return
    if shutil.which('etcd') is None:
      raise RuntimeError(
          "Embedded Dynamo mode requires etcd when ETCD_ENDPOINTS is not "
          "set. Install etcd in the worker container or set ETCD_ENDPOINTS "
          "to an external etcd service.")

    etcd_name = f'beam-dynamo-etcd-{uuid.uuid4().hex}'
    self._etcd_data_dir = f'/tmp/{etcd_name}'
    peer_port, = subprocess_server.pick_port(None)
    etcd_cmd = [
        'etcd',
        '--name',
        etcd_name,
        '--listen-client-urls',
        'http://127.0.0.1:{{PORT}}',
        '--advertise-client-urls',
        'http://127.0.0.1:{{PORT}}',
        '--listen-peer-urls',
        f'http://127.0.0.1:{peer_port}',
        '--initial-advertise-peer-urls',

View on GitHub (pinned to 12126d8942)