What causes "worker registry lock poisoned" in ultraworkers/claw-code?

Track down the first panic that held the worker registry lock (earlier backtrace); fixing it removes the poison source.

worker registry lock poisoned

Error message

worker registry lock poisoned

What it means

WorkerRegistry::create (rust/crates/runtime/src/worker_boot.rs:313) panics at .expect("worker registry lock poisoned") when the worker registry's inner Mutex is poisoned. The critical section allocates worker_<ts>_<counter> ids, evaluates trust_auto_resolve against trusted_roots via path_matches_allowlist, and inserts the new Worker — the panic indicates an earlier panic by another thread while this same lock was held.

Source

Thrown at rust/crates/runtime/src/worker_boot.rs:313

struct WorkerRegistryInner {
    workers: HashMap<String, Worker>,
    counter: u64,
}

impl WorkerRegistry {
    #[must_use]
    pub fn new() -> Self {
        Self::default()
    }

    #[must_use]
    pub fn create(
        &self,
        cwd: &str,
        trusted_roots: &[String],
        auto_recover_prompt_misdelivery: bool,
    ) -> Worker {
        let mut inner = self.inner.lock().expect("worker registry lock poisoned");
        inner.counter += 1;
        let ts = now_secs();
        let worker_id = format!("worker_{:08x}_{}", ts, inner.counter);
        let trust_auto_resolve = trusted_roots
            .iter()
            .any(|root| path_matches_allowlist(cwd, root));
        let mut worker = Worker {
            worker_id: worker_id.clone(),
            cwd: cwd.to_owned(),
            status: WorkerStatus::Spawning,
            trust_auto_resolve,
            trust_gate_cleared: false,
            auto_recover_prompt_misdelivery,
            prompt_delivery_attempts: 0,
            prompt_in_flight: false,
            prompt_sent_at: None,
            last_prompt: None,
            expected_receipt: None,

View on GitHub (pinned to 08106b0c37)

Solutions

Track down the first panic that held the worker registry lock (earlier backtrace); fixing it removes the poison source.
Restart the control-plane process — the registry is in-memory, so poisoning is not persisted.
Recover the guard in the crate: .lock().unwrap_or_else(|e| e.into_inner()); verify counter/worker state consistency before inserting the new worker.
Adopt parking_lot::Mutex (no poisoning) for this registry.
Keep panicky logic (terminal screen parsing, path matching) out of the locked region where possible, or wrap create() callers in catch_unwind.

Example fix

// before
let mut inner = self.inner.lock().expect("worker registry lock poisoned");
inner.counter += 1;

// after
let mut inner = self
    .inner
    .lock()
    .unwrap_or_else(|poisoned| poisoned.into_inner());
inner.counter += 1;

Defensive patterns

Strategy: try-catch

Try / catch

let worker = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
    worker_registry.create(cwd, trusted_roots, auto_recover)
}));
match worker {
    Ok(w) => w,
    Err(_) => return Err("worker registry poisoned; cannot spawn workers until restart"),
}

Prevention

Audit all WorkerRegistry critical sections (observe/send_prompt/observe_completion) for panics — one poison stops all worker creation.
Move panicky classification logic outside the locked region to shrink the poison window.
Recover guards with into_inner() or adopt parking_lot::Mutex if you maintain the crate.
Run each worker's supervision in a thread with catch_unwind so a single worker panic doesn't take down spawning.

When it happens

Trigger: Calling WorkerRegistry::create(cwd, trusted_roots, auto_recover_prompt_misdelivery) after any thread panicked inside a WorkerRegistry method (create/get/observe/observe_startup_preflight/resolve_trust/send_prompt/restart/terminate/observe_completion/observe_startup_timeout) that held the shared lock — e.g. a panic inside observe()'s screen-text classification unwinding mid-critical-section.

Common situations: Multi-worker agent lanes: one worker's observe() panicking on unexpected state poisons the registry, and then spawning any new worker (create) dies with 'worker registry lock poisoned', halting lane bootstrapping until restart.

Related errors

AI-assisted analysis of ultraworkers/claw-code@08106b0c37 (2026-08-18). Data as JSON: /api/errors/2020b11f1300073a. Report an issue: GitHub.