zeroclaw-labs/zeroclaw · error

live mode is not implemented yet (Phase 0 supports --mode re

Error message

live mode is not implemented yet (Phase 0 supports --mode replay only)

What it means

`run_suite` only implements replay of recorded trace fixtures in Phase 0; passing `Mode::Live` bails before any work happens. Live evaluation against a real model provider is planned but not built, so the guard is intentional and immediate. Note the asymmetry: `Mode::from_str("live")` parses fine, which can mislead users into thinking live mode works.

Source

Thrown at crates/zeroclaw-eval/src/runner.rs:24

use zeroclaw_api::model_provider::ModelProvider;
use zeroclaw_config::schema::MemoryConfig;
use zeroclaw_memory::{Memory, create_memory};
use zeroclaw_runtime::agent::agent::Agent;
use zeroclaw_runtime::agent::dispatcher::NativeToolDispatcher;

use crate::Mode;
use crate::case::{LlmTrace, load_suite};
use crate::grader::evaluate_expects;
use crate::observer::RecordingObserver;
use crate::record::RunRecord;
use crate::replay::TraceLlmProvider;
use crate::report::{CaseReport, SuiteReport};
use crate::tools::default_tools;

/// Run every `*.json` trace fixture in `dir` and return an aggregated report.
pub async fn run_suite(dir: &Path, mode: Mode) -> anyhow::Result<SuiteReport> {
    if mode == Mode::Live {
        anyhow::bail!("live mode is not implemented yet (Phase 0 supports --mode replay only)");
    }

    let traces = load_suite(dir)?;
    if traces.is_empty() {
        anyhow::bail!("no *.json trace fixtures found in {}", dir.display());
    }

    let mut cases = Vec::with_capacity(traces.len());
    for (path, trace) in traces {
        let name = trace.model_name.clone();
        let source = path
            .file_name()
            .and_then(|f| f.to_str())
            .unwrap_or("<unknown>")
            .to_string();

        let report = match run_case(&trace).await {
            Ok(record) => CaseReport {

View on GitHub (pinned to 88bb9c8533)

Solutions

1. Run with `--mode replay` and a directory of recorded `*.json` trace fixtures
2. Gate live mode in your own wrapper: check `mode == Mode::Live` first and print a clear 'not yet available' message
3. Track the milestone that ships live mode rather than working around the guard

Example fix

# before
zeroclaw-eval --mode live ./traces
# after
zeroclaw-eval --mode replay ./traces

if mode == zeroclaw_eval::Mode::Live {
    eprintln!("live mode arrives in a later phase; using replay");
    mode = zeroclaw_eval::Mode::Replay;
}

When it happens

Trigger: Calling `run_suite(dir, Mode::Live)` or running the eval CLI with `--mode live` — rejected at the top of the function, before the fixture directory is even scanned.

Common situations: Assuming the accepted `live` parse means the mode is usable; scripts written against future documentation; CI invoked with a mode flag from a newer release.

Related errors

• unknown eval mode '{other}' (expected 'replay' or 'live')
• TraceLlmProvider({}): turn {turn_index} scripted {leftover}
• TraceLlmProvider({}): turn {current} requested more LLM resp
• no *.json trace fixtures found in {}
• unsupported room visibility '{other}': expected private or p

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/157f0e78982eb22c. Report an issue: GitHub.