lucid command timed out after {}ms; failed to terminate and

Error message

lucid command timed out after {}ms; failed to terminate and reap child: {cleanup_error}

What it means

A lucid CLI subprocess exceeded its timeout window (recall_timeout_ms / store_timeout_ms, default or [storage.lucid.<alias>] override), and the subsequent terminate-and-reap cleanup also failed. The message reports both the elapsed window and the cleanup error because a failed reap means the child may still be running or become a zombie — that is what distinguishes it from the plain timeout variant.

Source

Thrown at crates/zeroclaw-memory/src/lucid.rs:298

        .await;

        let status = match output {
            Ok(status) => status?,
            Err(_) => {
                let cleanup_error = child.kill().await.err().map(|error| error.to_string());
                ::zeroclaw_log::record!(
                    ERROR,
                    ::zeroclaw_log::Event::new(module_path!(), ::zeroclaw_log::Action::Timeout)
                        .with_outcome(::zeroclaw_log::EventOutcome::Failure)
                        .with_attrs(::serde_json::json!({
                            "command": lucid_cmd,
                            "timeout_ms": timeout_window.as_millis() as u64,
                            "cleanup_error": cleanup_error,
                        })),
                    "lucid command timed out"
                );
                if let Some(cleanup_error) = cleanup_error {
                    anyhow::bail!(
                        "lucid command timed out after {}ms; failed to terminate and reap child: {cleanup_error}",
                        timeout_window.as_millis()
                    );
                }
                anyhow::bail!(
                    "lucid command timed out after {}ms",
                    timeout_window.as_millis()
                );
            }
        };

        if !status.success() {
            let stderr = String::from_utf8_lossy(&stderr_bytes);
            anyhow::bail!("lucid command failed: {stderr}");
        }

        Ok(String::from_utf8_lossy(&stdout_bytes).to_string())
    }

View on GitHub (pinned to 88bb9c8533)

Solutions

Raise the timeout: set `recall_timeout_ms` / `store_timeout_ms` under `[storage.lucid.<alias>]` above observed lucid p99 latency.
Check for leftover lucid processes after this error (ps aux | grep lucid) and kill them manually — reaping already failed.
Verify the lucid binary and endpoint are healthy by running the same lucid command manually with the same environment.

Example fix

# before
[storage.lucid.main]
recall_timeout_ms = 2000

# after
[storage.lucid.main]
recall_timeout_ms = 15000
store_timeout_ms = 20000

Defensive patterns

Strategy: fallback

Try / catch

let entries = match lucid_recall(query, limit, None, since, until).await {
    Ok(v) => v,
    Err(e) if e.to_string().contains("lucid command timed out") => {
        // local sqlite results are still valid: degrade and alert
        // (this variant also means reaping failed — sweep for orphaned
        // lucid processes before continuing)
        sweep_orphaned_lucid_processes();
        local.recall(query, limit, None, since, until).await?
    }
    Err(e) => return Err(e),
};

Prevention

Size recall_timeout_ms/store_timeout_ms to the observed p99 of the lucid binary plus network, not the average.
Monitor timeout rate; a rising rate is an early signal of lucid endpoint degradation.
Run lucid close to the service (same host/network) to keep the timeout headroom meaningful.

When it happens

Trigger: A lucid command hangs (remote endpoint stalls, binary waits on stdin or a stuck lock) past the configured timeout, then killing/waiting on the child errors — e.g. the process exits concurrently with the signal, or OS-level reaping races.

Common situations: Aggressively low lucid timeouts on slow networks; half-open connections to the lucid service; heavy load making SIGTERM delivery race with process exit; containers with restricted process-management privileges.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/a61ab89946c67de0. Report an issue: GitHub.