bytedance/deer-flow · critical · SystemExit

GATEWAY_WORKERS={workers} requires run_ownership.heartbeat_e

Error message

GATEWAY_WORKERS={workers} requires run_ownership.heartbeat_enabled=true. Without heartbeat, every run has a NULL lease, so reconciliation treats all inflight runs as orphans — Worker B would kill Worker A's live runs on every rolling update or scale-up. Set run_ownership.heartbeat_enabled=true in config.yaml.

What it means

SystemExit raised at startup when GATEWAY_WORKERS>1 and run_ownership is missing or heartbeat_enabled is false. Without heartbeats every run row has a NULL lease, so the reconciler classifies all in-flight runs as orphans — during rolling updates or scale-up, worker B would terminate worker A's live runs. The gate makes lease validity a hard precondition for multi-worker operation.

Source

Thrown at backend/app/gateway/deps.py:122

    if config.scheduler.enabled and not multi_instance_scheduler:
        raise SystemExit(f"GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=true because each worker starts its own scheduler. Set GATEWAY_WORKERS=1, scheduler.multi_instance=true, or scheduler.enabled=false.")

    if _browser_tools_enabled_in_config(config):
        raise SystemExit(browser_multi_worker_error(workers))

    if backend != "postgres":
        raise SystemExit(f"GATEWAY_WORKERS={workers} requires database.backend='postgres', but database.backend is '{backend}'. SQLite cannot support concurrent multi-process access. Set GATEWAY_WORKERS=1 or switch to Postgres.")

    if run_events_backend != "db":
        raise SystemExit(
            f"GATEWAY_WORKERS={workers} requires run_events.backend='db', but run_events.backend is '{run_events_backend}'. "
            "Memory and JSONL event stores are process-local, so delivery receipt singleton guarantees cannot hold across workers. "
            "Set GATEWAY_WORKERS=1 or configure run_events.backend: db."
        )

    if run_ownership is None or not run_ownership.heartbeat_enabled:
        raise SystemExit(
            f"GATEWAY_WORKERS={workers} requires run_ownership.heartbeat_enabled=true. "
            "Without heartbeat, every run has a NULL lease, so reconciliation "
            "treats all inflight runs as orphans — Worker B would kill Worker A's "
            "live runs on every rolling update or scale-up. "
            "Set run_ownership.heartbeat_enabled=true in config.yaml."
        )


def _validate_agent_storage(config: AppConfig) -> None:
    """Fail fast on an agent-storage backend the database cannot support.

    ``agent_storage.backend: db`` needs a durable, shared SQL database — a
    ``memory`` database is per-process, so agent definitions would silently
    diverge across nodes (and there is no SQL URL to open). Mirrors deermem's
    create_storage fail-fast and the multi-worker gate above.

    Also warns when a multi-worker Postgres deployment leaves agent storage on
    ``file``: custom agents created on one node's local disk are invisible to

View on GitHub (pinned to 1dd6ba1acb)

Solutions

1. Set run_ownership.heartbeat_enabled: true in config.yaml
2. Or set GATEWAY_WORKERS=1
3. Restart the Gateway

Example fix

# config.yaml
# before
run_ownership:
  heartbeat_enabled: false
# after
run_ownership:
  heartbeat_enabled: true

workers = int(os.environ.get("GATEWAY_WORKERS", "1"))
ro = getattr(load_config(), "run_ownership", None)
assert not (workers > 1 and (ro is None or not ro.heartbeat_enabled)), "multi-worker needs heartbeat_enabled"

When it happens

Trigger: GATEWAY_WORKERS>1 with no run_ownership section or heartbeat_enabled: false; the last of the multi-worker gates, after database and run_events checks.

Common situations: Config predating run-ownership (upgrade); disabling heartbeats as an optimization then scaling workers; deploying replicas behind a rolling-update strategy where cross-worker kills are guaranteed.

Related errors

• scheduler.multi_instance=true requires run_ownership.heartbe
• GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=
• GATEWAY_WORKERS={workers} cannot enable agentic browser tool
• GATEWAY_WORKERS={workers} requires database.backend='postgre
• GATEWAY_WORKERS={workers} requires run_events.backend='db',

AI-assisted analysis of bytedance/deer-flow@1dd6ba1acb (2026-08-14). Data as JSON: /api/errors/36d8ca38725b4dc8. Report an issue: GitHub.