{"record":{"id":"fb40f7e0fda78734","repo":"Hmbown/CodeWhale","slug":"offline-evaluation-harness-reported-failure","errorCode":null,"errorMessage":"offline evaluation harness reported failure","messagePattern":"offline evaluation harness reported failure","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/tui/src/lib.rs","lineNumber":2815,"sourceCode":"        let failed_steps: Vec<_> = report.steps.iter().filter(|s| !s.success).collect();\n        if !failed_steps.is_empty() {\n            println!(\"failed_steps:\");\n            for step in failed_steps {\n                let error = step.error.as_deref().unwrap_or(\"unknown error\");\n                println!(\n                    \"  {} tool={} error={}\",\n                    step.kind.tool_name(),\n                    step.tool_name,\n                    error\n                );\n            }\n        }\n    }\n\n    if report.metrics.success {\n        Ok(())\n    } else {\n        bail!(\"offline evaluation harness reported failure\")\n    }\n}\n\n/// Score a run's token/cache/cost from recorded turns and (optionally) flag\n/// regressions against a committed baseline. Offline: reads recorded usage from\n/// a JSON file, reuses the pricing layer, never calls a model. Exits non-zero\n/// when a baseline is supplied and a metric regresses past the threshold, so it\n/// can be wired as a release gate (#3388).\nfn run_scorecard(args: ScorecardArgs) -> Result<()> {\n    use crate::scorecard::{RecordedTurn, Scorecard, ScorecardMetrics};\n\n    let raw = std::fs::read_to_string(&args.input)\n        .with_context(|| format!(\"failed to read scorecard input {}\", args.input.display()))?;\n    let recorded: Vec<RecordedTurn> = serde_json::from_str(&raw)\n        .with_context(|| format!(\"failed to parse scorecard input {}\", args.input.display()))?;\n\n    let card = Scorecard::from_recorded_turns(&recorded);\n","sourceCodeStart":2797,"sourceCodeEnd":2833,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/0c42157ee52f9d55af2b506d71b46249910f77d3/crates/tui/src/lib.rs#L2797-L2833","documentation":"run_eval executes the offline evaluation scenario (list/read/search/edit/patch/shell steps) and reports metrics; metrics.success is true only when every step succeeded. After printing the human summary — including a failed_steps block naming each failing step's tool and error — the command exits non-zero with this bail, making `codewhale eval` usable as a CI gate without any model calls.","triggerScenarios":"Any eval step failing during the scenario run: a tool error in list/read/search/edit/patch/shell, an environment where a step's tool behaves unexpectedly, or an intentionally injected failure via `--fail-step <STEP>` used to test the harness itself.","commonSituations":"Wiring `codewhale eval` as an offline smoke test in CI; verifying the failure path with --fail-step; environment drift (PATH, workspace layout) making a harness step error.","solutions":["Read the failed_steps section of the output: each line names the step kind, tool, and error — fix that step's root cause first","If the failure was injected with --fail-step while testing the harness, remove the flag","Re-run in an environment matching the harness expectations: same workspace root, required tools on PATH","If wiring CI, treat a non-zero exit as a red build and archive the full output"],"exampleFix":"# before\ncodewhale eval --fail-step shell    # exits 1: offline evaluation harness reported failure\n\n# after\ncodewhale eval                       # green run exits 0 and gates CI","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if ! codewhale eval 2>&1 | tee eval.log; then\n  grep -A20 'failed_steps:' eval.log   # step kind, tool, error\n  exit 1\nfi","preventionTips":["Never pass --fail-step outside deliberate harness tests","Keep the eval workspace and PATH stable across CI runs","Archive eval output so failed_steps is diagnosable after the gate trips"],"tags":["eval","testing","ci","offline"],"backgroundTag":"eval-harness-failed","analyzedSha":"0c42157ee52f9d55af2b506d71b46249910f77d3","analyzedAt":"2026-08-20T21:50:45.477Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}