Hmbown/CodeWhale · error · anyhow::Error

{} metric(s) regressed past the {:.1}% threshold

Error message

{} metric(s) regressed past the {:.1}% threshold

What it means

run_scorecard scores recorded turns from --input JSON into token/cache/cost metrics and, when --baseline is supplied, flags every metric that increased more than --threshold percent (default 5.0). A non-empty regression list fails the command so it can act as a release gate; the printed REGRESSION lines name each metric with its baseline -> current movement and percent increase.

Source

Thrown at crates/tui/src/lib.rs:2865

            "per_turn": card.per_turn,
            "metrics": card.metrics,
            "regressions": regressions,
        });
        println!("{}", serde_json::to_string_pretty(&out)?);
    } else {
        print!("{}", card.to_summary());
        for r in &regressions {
            println!(
                "REGRESSION {}: baseline {:.4} -> current {:.4} (+{:.1}%)",
                r.metric, r.baseline, r.current, r.pct_increase
            );
        }
    }

    if regressions.is_empty() {
        Ok(())
    } else {
        bail!(
            "{} metric(s) regressed past the {:.1}% threshold",
            regressions.len(),
            args.threshold
        )
    }
}

async fn run_fleet_command(workspace: &Path, config: &Config, args: FleetArgs) -> Result<()> {
    use crate::fleet::alerts::{
        FleetAlertAdapterConfig, FleetAlertConfig, FleetAlertDispatcher, FleetAlertEvent,
        FleetEnvSecretResolver,
    };
    use crate::fleet::control as fleet_control;
    use crate::fleet::executor::FleetExecutor;
    use crate::fleet::manager::{FleetManager, FleetStatusSnapshot, FleetWorkerInspection};
    use codewhale_lane::{ControlOperation, ControlSurface};
    use codewhale_protocol::fleet::{FleetAlertEventClass, FleetArtifactKind, FleetRunId};

View on GitHub (pinned to 0c42157ee5)

Solutions

  1. Read each REGRESSION line and identify which metric (cost, tokens, cache) drove it
  2. Check for real regressions first: routing changes, cache-busting prompt edits, context growth per turn
  3. If the growth is accepted, regenerate the baseline from the current run so the gate tracks the new normal
  4. Only if a metric is known-noisy, widen --threshold deliberately and record why

Example fix

# before
codewhale scorecard --input turns.json --baseline baseline.json
# 2 metric(s) regressed past the 5.0% threshold

# after (accepted growth: re-baseline the new normal)
codewhale scorecard --input turns.json --json > baseline.json
git add baseline.json && git commit -m 'scorecard: re-baseline after routing change'
Defensive patterns

Strategy: try-catch

Validate before calling

# Dry-check before wiring the gate: run without --baseline to inspect metrics first
codewhale scorecard --input turns.json --json | jq '.metrics'

Try / catch

if ! codewhale scorecard --input turns.json --baseline baseline.json; then
  # output lists each REGRESSION metric; triage real regressions vs accepted growth
  exit 1
fi

Prevention

When it happens

Trigger: `codewhale scorecard --input turns.json --baseline baseline.json` where any scored metric grew past the threshold: costlier routing per turn, lost prompt-cache hits, larger contexts, or a baseline recorded against a cheaper model mix. Also a deliberately strict threshold applied to known-noisy metrics.

Common situations: Release/CI gates tripping after a model or routing change; prompt changes that bust the cache; baselines from an older pricing tier; legitimate traffic growth mistaken for regression.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/43f08ae26af880d8. Report an issue: GitHub.