bytedance/deer-flow · critical · SystemExit

scheduler.multi_instance=true requires run_ownership.heartbe

Error message

scheduler.multi_instance=true requires run_ownership.heartbeat_enabled=true so peer runs retain a valid lease. Set scheduler.multi_instance=false or enable run ownership heartbeats.

What it means

SystemExit raised at startup when scheduler.multi_instance=true but run_ownership is missing or run_ownership.heartbeat_enabled is not true. Peer schedulers reconcile orphaned runs; without heartbeats every run has a NULL lease and would look orphaned, so one scheduler would kill another scheduler's live runs on every reconcile pass. The gate forces a valid lease protocol before multi-instance mode is allowed.

Source

Thrown at backend/app/gateway/deps.py:100

    try:
        workers = int(os.environ.get("GATEWAY_WORKERS", "1"))
    except (TypeError, ValueError):
        workers = 1

    scheduler = getattr(config, "scheduler", None)
    multi_instance_requested = bool(getattr(scheduler, "multi_instance", False))
    multi_instance_scheduler = bool(getattr(scheduler, "enabled", False) and multi_instance_requested)

    backend = getattr(config.database, "backend", None)
    run_events_backend = getattr(getattr(config, "run_events", None), "backend", None)
    run_ownership = getattr(config, "run_ownership", None)

    if multi_instance_requested and backend != "postgres":
        raise SystemExit(f"scheduler.multi_instance=true requires database.backend='postgres'. database.backend is '{backend}'. Set scheduler.multi_instance=false or configure Postgres.")
    if multi_instance_requested and run_events_backend != "db":
        raise SystemExit(f"scheduler.multi_instance=true requires run_events.backend='db'. run_events.backend is '{run_events_backend}'. Set scheduler.multi_instance=false or configure run_events.backend: db.")
    if multi_instance_requested and (run_ownership is None or not run_ownership.heartbeat_enabled):
        raise SystemExit("scheduler.multi_instance=true requires run_ownership.heartbeat_enabled=true so peer runs retain a valid lease. Set scheduler.multi_instance=false or enable run ownership heartbeats.")

    if workers <= 1:
        return

    if config.scheduler.enabled and not multi_instance_scheduler:
        raise SystemExit(f"GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=true because each worker starts its own scheduler. Set GATEWAY_WORKERS=1, scheduler.multi_instance=true, or scheduler.enabled=false.")

    if _browser_tools_enabled_in_config(config):
        raise SystemExit(browser_multi_worker_error(workers))

    if backend != "postgres":
        raise SystemExit(f"GATEWAY_WORKERS={workers} requires database.backend='postgres', but database.backend is '{backend}'. SQLite cannot support concurrent multi-process access. Set GATEWAY_WORKERS=1 or switch to Postgres.")

    if run_events_backend != "db":
        raise SystemExit(
            f"GATEWAY_WORKERS={workers} requires run_events.backend='db', but run_events.backend is '{run_events_backend}'. "
            "Memory and JSONL event stores are process-local, so delivery receipt singleton guarantees cannot hold across workers. "
            "Set GATEWAY_WORKERS=1 or configure run_events.backend: db."

View on GitHub (pinned to 1dd6ba1acb)

Solutions

Set run_ownership.heartbeat_enabled: true in config.yaml
Or set scheduler.multi_instance: false
Restart the Gateway

Example fix

# config.yaml
# before
run_ownership:
  heartbeat_enabled: false
# after
run_ownership:
  heartbeat_enabled: true

Defensive patterns

Strategy: validation

Validate before calling

cfg = load_config()
mi = bool(getattr(getattr(cfg, "scheduler", None), "multi_instance", False))
ro = getattr(cfg, "run_ownership", None)
assert not (mi and (ro is None or not ro.heartbeat_enabled)), "multi_instance needs heartbeat_enabled"

Prevention

Never disable run_ownership heartbeats on multi-node deployments
After upgrading, grep config.yaml for run_ownership — upgrades add required sections
Bundle the three multi_instance prerequisites into one reviewed config change

When it happens

Trigger: config.yaml with scheduler.multi_instance: true and either no run_ownership section or run_ownership.heartbeat_enabled: false.

Common situations: Config written before run-ownership existed (upgrade path); hand-merged config.yaml dropping the run_ownership block; disabling heartbeats to 'reduce load' while scaling out schedulers.

Related errors

AI-assisted analysis of bytedance/deer-flow@1dd6ba1acb (2026-08-14). Data as JSON: /api/errors/bff942289ae10b73. Report an issue: GitHub.