bytedance/deer-flow · critical · SystemExit

GATEWAY_WORKERS={workers} requires run_events.backend='db',

Error message

GATEWAY_WORKERS={workers} requires run_events.backend='db', but run_events.backend is '{run_events_backend}'. Memory and JSONL event stores are process-local, so delivery receipt singleton guarantees cannot hold across workers. Set GATEWAY_WORKERS=1 or configure run_events.backend: db.

What it means

SystemExit raised at startup when GATEWAY_WORKERS>1 but run_events.backend is not 'db'. Memory and JSONL run-event stores are process-local: with multiple workers, the delivery-receipt singleton guarantee (an event delivered exactly once across the fleet) cannot hold because workers cannot see each other's receipts. The gate forces the shared DB store before multi-worker mode is allowed.

Source

Thrown at backend/app/gateway/deps.py:115

    if multi_instance_requested and run_events_backend != "db":
        raise SystemExit(f"scheduler.multi_instance=true requires run_events.backend='db'. run_events.backend is '{run_events_backend}'. Set scheduler.multi_instance=false or configure run_events.backend: db.")
    if multi_instance_requested and (run_ownership is None or not run_ownership.heartbeat_enabled):
        raise SystemExit("scheduler.multi_instance=true requires run_ownership.heartbeat_enabled=true so peer runs retain a valid lease. Set scheduler.multi_instance=false or enable run ownership heartbeats.")

    if workers <= 1:
        return

    if config.scheduler.enabled and not multi_instance_scheduler:
        raise SystemExit(f"GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=true because each worker starts its own scheduler. Set GATEWAY_WORKERS=1, scheduler.multi_instance=true, or scheduler.enabled=false.")

    if _browser_tools_enabled_in_config(config):
        raise SystemExit(browser_multi_worker_error(workers))

    if backend != "postgres":
        raise SystemExit(f"GATEWAY_WORKERS={workers} requires database.backend='postgres', but database.backend is '{backend}'. SQLite cannot support concurrent multi-process access. Set GATEWAY_WORKERS=1 or switch to Postgres.")

    if run_events_backend != "db":
        raise SystemExit(
            f"GATEWAY_WORKERS={workers} requires run_events.backend='db', but run_events.backend is '{run_events_backend}'. "
            "Memory and JSONL event stores are process-local, so delivery receipt singleton guarantees cannot hold across workers. "
            "Set GATEWAY_WORKERS=1 or configure run_events.backend: db."
        )

    if run_ownership is None or not run_ownership.heartbeat_enabled:
        raise SystemExit(
            f"GATEWAY_WORKERS={workers} requires run_ownership.heartbeat_enabled=true. "
            "Without heartbeat, every run has a NULL lease, so reconciliation "
            "treats all inflight runs as orphans — Worker B would kill Worker A's "
            "live runs on every rolling update or scale-up. "
            "Set run_ownership.heartbeat_enabled=true in config.yaml."
        )


def _validate_agent_storage(config: AppConfig) -> None:
    """Fail fast on an agent-storage backend the database cannot support.

View on GitHub (pinned to 1dd6ba1acb)

Solutions

1. Set run_events.backend: db in config.yaml
2. Or set GATEWAY_WORKERS=1
3. Restart the Gateway

Example fix

# config.yaml
# before
run_events:
  backend: jsonl
# after
run_events:
  backend: db

workers = int(os.environ.get("GATEWAY_WORKERS", "1"))
re_backend = getattr(getattr(load_config(), "run_events", None), "backend", None)
assert not (workers > 1 and re_backend != "db"), "multi-worker needs run_events.backend: db"

When it happens

Trigger: GATEWAY_WORKERS>1 with run_events.backend set to memory or jsonl (or defaulted to a non-db value); reached after the postgres-backend check passes.

Common situations: Default run_events configuration never changed when workers were scaled; JSONL store chosen for debugging/inspection and left on; config drift between single-worker dev and multi-worker prod.

Related errors

• scheduler.multi_instance=true requires run_events.backend='d
• GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=
• GATEWAY_WORKERS={workers} cannot enable agentic browser tool
• GATEWAY_WORKERS={workers} requires database.backend='postgre
• GATEWAY_WORKERS={workers} requires run_ownership.heartbeat_e

AI-assisted analysis of bytedance/deer-flow@1dd6ba1acb (2026-08-14). Data as JSON: /api/errors/74fa3c1a82d58d99. Report an issue: GitHub.