bytedance/deer-flow · critical · SystemExit

GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=

Error message

GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=true because each worker starts its own scheduler. Set GATEWAY_WORKERS=1, scheduler.multi_instance=true, or scheduler.enabled=false.

What it means

SystemExit raised at startup when GATEWAY_WORKERS>1, scheduler.enabled=true, and scheduler.multi_instance is false (multi_instance_scheduler = enabled AND multi_instance). Each uvicorn worker would start its own scheduler, causing every cron/interval task to fire N times. The gate demands either a single worker, explicit multi-instance coordination, or no scheduler.

Source

Thrown at backend/app/gateway/deps.py:106

    multi_instance_requested = bool(getattr(scheduler, "multi_instance", False))
    multi_instance_scheduler = bool(getattr(scheduler, "enabled", False) and multi_instance_requested)

    backend = getattr(config.database, "backend", None)
    run_events_backend = getattr(getattr(config, "run_events", None), "backend", None)
    run_ownership = getattr(config, "run_ownership", None)

    if multi_instance_requested and backend != "postgres":
        raise SystemExit(f"scheduler.multi_instance=true requires database.backend='postgres'. database.backend is '{backend}'. Set scheduler.multi_instance=false or configure Postgres.")
    if multi_instance_requested and run_events_backend != "db":
        raise SystemExit(f"scheduler.multi_instance=true requires run_events.backend='db'. run_events.backend is '{run_events_backend}'. Set scheduler.multi_instance=false or configure run_events.backend: db.")
    if multi_instance_requested and (run_ownership is None or not run_ownership.heartbeat_enabled):
        raise SystemExit("scheduler.multi_instance=true requires run_ownership.heartbeat_enabled=true so peer runs retain a valid lease. Set scheduler.multi_instance=false or enable run ownership heartbeats.")

    if workers <= 1:
        return

    if config.scheduler.enabled and not multi_instance_scheduler:
        raise SystemExit(f"GATEWAY_WORKERS={workers} cannot run with scheduler.enabled=true because each worker starts its own scheduler. Set GATEWAY_WORKERS=1, scheduler.multi_instance=true, or scheduler.enabled=false.")

    if _browser_tools_enabled_in_config(config):
        raise SystemExit(browser_multi_worker_error(workers))

    if backend != "postgres":
        raise SystemExit(f"GATEWAY_WORKERS={workers} requires database.backend='postgres', but database.backend is '{backend}'. SQLite cannot support concurrent multi-process access. Set GATEWAY_WORKERS=1 or switch to Postgres.")

    if run_events_backend != "db":
        raise SystemExit(
            f"GATEWAY_WORKERS={workers} requires run_events.backend='db', but run_events.backend is '{run_events_backend}'. "
            "Memory and JSONL event stores are process-local, so delivery receipt singleton guarantees cannot hold across workers. "
            "Set GATEWAY_WORKERS=1 or configure run_events.backend: db."
        )

    if run_ownership is None or not run_ownership.heartbeat_enabled:
        raise SystemExit(
            f"GATEWAY_WORKERS={workers} requires run_ownership.heartbeat_enabled=true. "
            "Without heartbeat, every run has a NULL lease, so reconciliation "

View on GitHub (pinned to 1dd6ba1acb)

Solutions

  1. Set GATEWAY_WORKERS=1
  2. Or set scheduler.multi_instance: true (plus the postgres/run_events/heartbeat prerequisites it gates on)
  3. Or set scheduler.enabled: false if scheduled tasks run elsewhere

Example fix

# environment
# before
GATEWAY_WORKERS=4
# after (option A)
GATEWAY_WORKERS=1
# after (option B, in config.yaml)
scheduler:
  enabled: true
  multi_instance: true
Defensive patterns

Strategy: validation

Validate before calling

import os
cfg = load_config()
workers = int(os.environ.get("GATEWAY_WORKERS", "1"))
sched = getattr(cfg, "scheduler", None)
mi = bool(getattr(sched, "multi_instance", False))
assert not (workers > 1 and sched and sched.enabled and not mi), "workers>1 + scheduler needs multi_instance"

Prevention

When it happens

Trigger: Launching with GATEWAY_WORKERS=2 or more (env var) while config.yaml has scheduler.enabled: true and multi_instance unset/false. Only reached when workers>1 (the workers<=1 branch returns earlier).

Common situations: Raising worker count for throughput on a box that also runs scheduled tasks; production Docker image defaults with GATEWAY_WORKERS set in compose; forgetting that the scheduler is enabled from an earlier single-worker setup.

Related errors


AI-assisted analysis of bytedance/deer-flow@1dd6ba1acb (2026-08-14). Data as JSON: /api/errors/270a8a4169a4eccf. Report an issue: GitHub.