Hmbown/CodeWhale · error · anyhow::Error

offline evaluation harness reported failure

Error message

offline evaluation harness reported failure

What it means

run_eval executes the offline evaluation scenario (list/read/search/edit/patch/shell steps) and reports metrics; metrics.success is true only when every step succeeded. After printing the human summary — including a failed_steps block naming each failing step's tool and error — the command exits non-zero with this bail, making `codewhale eval` usable as a CI gate without any model calls.

Source

Thrown at crates/tui/src/lib.rs:2815

        let failed_steps: Vec<_> = report.steps.iter().filter(|s| !s.success).collect();
        if !failed_steps.is_empty() {
            println!("failed_steps:");
            for step in failed_steps {
                let error = step.error.as_deref().unwrap_or("unknown error");
                println!(
                    "  {} tool={} error={}",
                    step.kind.tool_name(),
                    step.tool_name,
                    error
                );
            }
        }
    }

    if report.metrics.success {
        Ok(())
    } else {
        bail!("offline evaluation harness reported failure")
    }
}

/// Score a run's token/cache/cost from recorded turns and (optionally) flag
/// regressions against a committed baseline. Offline: reads recorded usage from
/// a JSON file, reuses the pricing layer, never calls a model. Exits non-zero
/// when a baseline is supplied and a metric regresses past the threshold, so it
/// can be wired as a release gate (#3388).
fn run_scorecard(args: ScorecardArgs) -> Result<()> {
    use crate::scorecard::{RecordedTurn, Scorecard, ScorecardMetrics};

    let raw = std::fs::read_to_string(&args.input)
        .with_context(|| format!("failed to read scorecard input {}", args.input.display()))?;
    let recorded: Vec<RecordedTurn> = serde_json::from_str(&raw)
        .with_context(|| format!("failed to parse scorecard input {}", args.input.display()))?;

    let card = Scorecard::from_recorded_turns(&recorded);

View on GitHub (pinned to 0c42157ee5)

Solutions

1. Read the failed_steps section of the output: each line names the step kind, tool, and error — fix that step's root cause first
2. If the failure was injected with --fail-step while testing the harness, remove the flag
3. Re-run in an environment matching the harness expectations: same workspace root, required tools on PATH
4. If wiring CI, treat a non-zero exit as a red build and archive the full output

Example fix

# before
codewhale eval --fail-step shell    # exits 1: offline evaluation harness reported failure

# after
codewhale eval                       # green run exits 0 and gates CI

if ! codewhale eval 2>&1 | tee eval.log; then
  grep -A20 'failed_steps:' eval.log   # step kind, tool, error
  exit 1
fi

When it happens

Trigger: Any eval step failing during the scenario run: a tool error in list/read/search/edit/patch/shell, an environment where a step's tool behaves unexpectedly, or an intentionally injected failure via `--fail-step <STEP>` used to test the harness itself.

Common situations: Wiring `codewhale eval` as an offline smoke test in CI; verifying the failure path with --fail-step; environment drift (PATH, workspace layout) making a harness step error.

Related errors

• external credential consent was not saved: non-interactive u
• lane `{}` was stopped before tmux dry-run start completed
• workspace appears empty after File.list: {}
• search root does not exist: {}
• patch missing *** Begin Patch header

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/fb40f7e0fda78734. Report an issue: GitHub.