{"record":{"id":"36d8ca38725b4dc8","repo":"bytedance/deer-flow","slug":"gateway-workers-workers-requires-run-ownership-h","errorCode":null,"errorMessage":"GATEWAY_WORKERS={workers} requires run_ownership.heartbeat_enabled=true. Without heartbeat, every run has a NULL lease, so reconciliation treats all inflight runs as orphans — Worker B would kill Worker A's live runs on every rolling update or scale-up. Set run_ownership.heartbeat_enabled=true in config.yaml.","messagePattern":"GATEWAY_WORKERS=(.+?) requires run_ownership\\.heartbeat_enabled=true\\. Without heartbeat, every run has a NULL lease, so reconciliation treats all inflight runs as orphans — Worker B would kill Worker A's live runs on every rolling update or scale-up\\. Set run_ownership\\.heartbeat_enabled=true in config\\.yaml\\.","errorType":"console","errorClass":"SystemExit","httpStatus":null,"severity":"critical","filePath":"backend/app/gateway/deps.py","lineNumber":122,"sourceCode":"\n    if config.scheduler.enabled and not multi_instance_scheduler:\n        raise SystemExit(f\"GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=true because each worker starts its own scheduler. Set GATEWAY_WORKERS=1, scheduler.multi_instance=true, or scheduler.enabled=false.\")\n\n    if _browser_tools_enabled_in_config(config):\n        raise SystemExit(browser_multi_worker_error(workers))\n\n    if backend != \"postgres\":\n        raise SystemExit(f\"GATEWAY_WORKERS={workers} requires database.backend='postgres', but database.backend is '{backend}'. SQLite cannot support concurrent multi-process access. Set GATEWAY_WORKERS=1 or switch to Postgres.\")\n\n    if run_events_backend != \"db\":\n        raise SystemExit(\n            f\"GATEWAY_WORKERS={workers} requires run_events.backend='db', but run_events.backend is '{run_events_backend}'. \"\n            \"Memory and JSONL event stores are process-local, so delivery receipt singleton guarantees cannot hold across workers. \"\n            \"Set GATEWAY_WORKERS=1 or configure run_events.backend: db.\"\n        )\n\n    if run_ownership is None or not run_ownership.heartbeat_enabled:\n        raise SystemExit(\n            f\"GATEWAY_WORKERS={workers} requires run_ownership.heartbeat_enabled=true. \"\n            \"Without heartbeat, every run has a NULL lease, so reconciliation \"\n            \"treats all inflight runs as orphans — Worker B would kill Worker A's \"\n            \"live runs on every rolling update or scale-up. \"\n            \"Set run_ownership.heartbeat_enabled=true in config.yaml.\"\n        )\n\n\ndef _validate_agent_storage(config: AppConfig) -> None:\n    \"\"\"Fail fast on an agent-storage backend the database cannot support.\n\n    ``agent_storage.backend: db`` needs a durable, shared SQL database — a\n    ``memory`` database is per-process, so agent definitions would silently\n    diverge across nodes (and there is no SQL URL to open). Mirrors deermem's\n    create_storage fail-fast and the multi-worker gate above.\n\n    Also warns when a multi-worker Postgres deployment leaves agent storage on\n    ``file``: custom agents created on one node's local disk are invisible to","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/bytedance/deer-flow/blob/1dd6ba1acb03700589994b0366c5d1c7d05e2eff/backend/app/gateway/deps.py#L104-L140","documentation":"SystemExit raised at startup when GATEWAY_WORKERS>1 and run_ownership is missing or heartbeat_enabled is false. Without heartbeats every run row has a NULL lease, so the reconciler classifies all in-flight runs as orphans — during rolling updates or scale-up, worker B would terminate worker A's live runs. The gate makes lease validity a hard precondition for multi-worker operation.","triggerScenarios":"GATEWAY_WORKERS>1 with no run_ownership section or heartbeat_enabled: false; the last of the multi-worker gates, after database and run_events checks.","commonSituations":"Config predating run-ownership (upgrade); disabling heartbeats as an optimization then scaling workers; deploying replicas behind a rolling-update strategy where cross-worker kills are guaranteed.","solutions":["Set run_ownership.heartbeat_enabled: true in config.yaml","Or set GATEWAY_WORKERS=1","Restart the Gateway"],"exampleFix":"# config.yaml\n# before\nrun_ownership:\n  heartbeat_enabled: false\n# after\nrun_ownership:\n  heartbeat_enabled: true","handlingStrategy":"validation","validationCode":"workers = int(os.environ.get(\"GATEWAY_WORKERS\", \"1\"))\nro = getattr(load_config(), \"run_ownership\", None)\nassert not (workers > 1 and (ro is None or not ro.heartbeat_enabled)), \"multi-worker needs heartbeat_enabled\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set run_ownership.heartbeat_enabled: true whenever GATEWAY_WORKERS>1 — otherwise reconcilers kill live runs on deploys","Test rolling updates in staging to verify no cross-worker run kills","Keep the run_ownership block present even if defaults are acceptable, to survive schema changes"],"tags":["config","startup","run-ownership","heartbeat","workers","system-exit","gateway"],"backgroundTag":null,"analyzedSha":"1dd6ba1acb03700589994b0366c5d1c7d05e2eff","analyzedAt":"2026-08-14T21:20:34.804Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}