{"record":{"id":"42e44ba595f43e92","repo":"apache/beam","slug":"failed-to-start-embedded-etcd-for-dynamo-process-status","errorCode":null,"errorMessage":"Failed to start embedded etcd for Dynamo. Process status: {process_status}. Install etcd in the worker container or set ETCD_ENDPOINTS to an external etcd service.","messagePattern":"Failed to start embedded etcd for Dynamo\\. Process status: (.+?)\\. Install etcd in the worker container or set ETCD_ENDPOINTS to an external etcd service\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"sdks/python/apache_beam/ml/inference/vllm_inference.py","lineNumber":242,"sourceCode":"        self._use_dynamo and\n        _uses_etcd_discovery(self._dynamo_frontend_kwargs) and\n        _uses_etcd_discovery(self._vllm_server_kwargs) and\n        'ETCD_ENDPOINTS' not in os.environ)\n\n  def _wait_for_etcd(self, endpoint: str, timeout_secs=30) -> None:\n    deadline = time.time() + timeout_secs\n    health_url = endpoint.rstrip('/') + '/health'\n    while time.time() < deadline and self._etcd_process.poll() is None:\n      try:\n        with urllib.request.urlopen(health_url, timeout=2) as response:\n          if response.status < 500:\n            return\n      except Exception:  # pylint: disable=broad-except\n        time.sleep(1)\n\n    process_status = self._process_status()\n    self._stop_processes()\n    raise RuntimeError(\n        \"Failed to start embedded etcd for Dynamo. Process status: \"\n        f\"{process_status}. Install etcd in the worker container or set \"\n        \"ETCD_ENDPOINTS to an external etcd service.\")\n\n  def _ensure_etcd(self) -> None:\n    if not self._uses_embedded_etcd():\n      return\n    if shutil.which('etcd') is None:\n      raise RuntimeError(\n          \"Embedded Dynamo mode requires etcd when ETCD_ENDPOINTS is not \"\n          \"set. Install etcd in the worker container or set ETCD_ENDPOINTS \"\n          \"to an external etcd service.\")\n\n    etcd_name = f'beam-dynamo-etcd-{uuid.uuid4().hex}'\n    self._etcd_data_dir = f'/tmp/{etcd_name}'\n    peer_port, = subprocess_server.pick_port(None)\n    etcd_cmd = [\n        'etcd',","sourceCodeStart":224,"sourceCodeEnd":260,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/inference/vllm_inference.py#L224-L260","documentation":"vLLM inference with embedded Dynamo mode tries to start a local etcd instance for coordination. _wait_for_etcd raises RuntimeError if the etcd process never becomes ready, instructing the user to install etcd in the worker container or point ETCD_ENDPOINTS at an external etcd service.","triggerScenarios":"start_server in embedded Dynamo mode (ETCD_ENDPOINTS unset) where the spawned etcd binary crashes or never becomes healthy within the wait timeout: binary incompatible with the container OS/arch, missing permissions, or port conflicts.","commonSituations":"Running on Dataflow/custom worker images without etcd installed; etcd binary present but failing to start (bad arch, read-only /tmp, port already in use).","solutions":["Install etcd in the worker container image and ensure it is on PATH","Set the ETCD_ENDPOINTS environment variable to an external etcd cluster (e.g. etcd1:2379,etcd2:2379) to skip embedded etcd","Check _process_status / container logs to see why etcd exited (port conflict, permissions, arch mismatch)"],"exampleFix":"# before (Dockerfile)\nFROM nvidia/cuda:12.1-base\n# after\nFROM nvidia/cuda:12.1-base\nRUN apt-get update && apt-get install -y etcd-server\nENV PATH=\"/usr/bin:${PATH}\"","handlingStrategy":"validation","validationCode":"import os, shutil\nif not os.environ.get('ETCD_ENDPOINTS') and shutil.which('etcd') is None:\n    raise RuntimeError('Embedded Dynamo needs etcd installed or ETCD_ENDPOINTS set')","typeGuard":null,"tryCatchPattern":"try:\n    server.start_server()\nexcept RuntimeError as e:\n    if 'embedded etcd' in str(e):\n        install_or_configure_etcd()  # or set ETCD_ENDPOINTS and rebuild\n    raise","preventionTips":["Bake etcd into worker images used for vLLM/Dynamo pipelines","Prefer external etcd via ETCD_ENDPOINTS in production","Check etcd binary compatibility (arch/OS) with the container"],"tags":["apache-beam","vllm","etcd","dynamo","environment"],"backgroundTag":"missing-dependency","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}