{"record":{"id":"157f0e78982eb22c","repo":"zeroclaw-labs/zeroclaw","slug":"live-mode-is-not-implemented-yet-phase-0-supports","errorCode":null,"errorMessage":"live mode is not implemented yet (Phase 0 supports --mode replay only)","messagePattern":"live mode is not implemented yet \\(Phase 0 supports --mode replay only\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/zeroclaw-eval/src/runner.rs","lineNumber":24,"sourceCode":"use zeroclaw_api::model_provider::ModelProvider;\nuse zeroclaw_config::schema::MemoryConfig;\nuse zeroclaw_memory::{Memory, create_memory};\nuse zeroclaw_runtime::agent::agent::Agent;\nuse zeroclaw_runtime::agent::dispatcher::NativeToolDispatcher;\n\nuse crate::Mode;\nuse crate::case::{LlmTrace, load_suite};\nuse crate::grader::evaluate_expects;\nuse crate::observer::RecordingObserver;\nuse crate::record::RunRecord;\nuse crate::replay::TraceLlmProvider;\nuse crate::report::{CaseReport, SuiteReport};\nuse crate::tools::default_tools;\n\n/// Run every `*.json` trace fixture in `dir` and return an aggregated report.\npub async fn run_suite(dir: &Path, mode: Mode) -> anyhow::Result<SuiteReport> {\n    if mode == Mode::Live {\n        anyhow::bail!(\"live mode is not implemented yet (Phase 0 supports --mode replay only)\");\n    }\n\n    let traces = load_suite(dir)?;\n    if traces.is_empty() {\n        anyhow::bail!(\"no *.json trace fixtures found in {}\", dir.display());\n    }\n\n    let mut cases = Vec::with_capacity(traces.len());\n    for (path, trace) in traces {\n        let name = trace.model_name.clone();\n        let source = path\n            .file_name()\n            .and_then(|f| f.to_str())\n            .unwrap_or(\"<unknown>\")\n            .to_string();\n\n        let report = match run_case(&trace).await {\n            Ok(record) => CaseReport {","sourceCodeStart":6,"sourceCodeEnd":42,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-eval/src/runner.rs#L6-L42","documentation":"`run_suite` only implements replay of recorded trace fixtures in Phase 0; passing `Mode::Live` bails before any work happens. Live evaluation against a real model provider is planned but not built, so the guard is intentional and immediate. Note the asymmetry: `Mode::from_str(\"live\")` parses fine, which can mislead users into thinking live mode works.","triggerScenarios":"Calling `run_suite(dir, Mode::Live)` or running the eval CLI with `--mode live` — rejected at the top of the function, before the fixture directory is even scanned.","commonSituations":"Assuming the accepted `live` parse means the mode is usable; scripts written against future documentation; CI invoked with a mode flag from a newer release.","solutions":["Run with `--mode replay` and a directory of recorded `*.json` trace fixtures","Gate live mode in your own wrapper: check `mode == Mode::Live` first and print a clear 'not yet available' message","Track the milestone that ships live mode rather than working around the guard"],"exampleFix":"# before\nzeroclaw-eval --mode live ./traces\n# after\nzeroclaw-eval --mode replay ./traces","handlingStrategy":"validation","validationCode":"if mode == zeroclaw_eval::Mode::Live {\n    eprintln!(\"live mode arrives in a later phase; using replay\");\n    mode = zeroclaw_eval::Mode::Replay;\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Do not advertise a live flag in wrappers until the phase ships it","Pin your eval workflow and docs to --mode replay","Watch release notes for the live-mode milestone instead of probing with --mode live"],"tags":["eval","not-implemented","phase-0","zeroclaw-eval"],"backgroundTag":"unsupported-feature","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}