Hmbown/CodeWhale · error · RuntimeError

exact library metric test {metric_test_name(test_name)} ran

Error message

exact library metric test {metric_test_name(test_name)} ran zero tests

What it means

scripts/measure-runtime-contract.py builds the provider-free runtime-contract receipt by running one exact ignored library test (cargo test --locked -p codewhale-tui --lib core::engine::tests::<name> -- --ignored --exact --nocapture --test-threads=1) and parsing a marker line. This error means the test harness output contained 'running 0 tests', i.e. the exact filter selected nothing, so no metric could be collected. The script fails fast instead of emitting an empty receipt.

Source

Thrown at scripts/measure-runtime-contract.py:52

        "--ignored",
        "--exact",
        "--nocapture",
        "--test-threads=1",
    ]


def run_metric(test_name: str, marker: str) -> dict:
    cmd = metric_command(test_name)
    proc = subprocess.run(cmd, text=True, capture_output=True, check=False)
    sys.stderr.write(proc.stderr)
    if proc.returncode != 0:
        sys.stdout.write(proc.stdout)
        proc.check_returncode()

    combined = proc.stdout.splitlines() + proc.stderr.splitlines()
    if re.search(r"\brunning\s+0\s+tests?\b", "\n".join(combined)):
        sys.stdout.write(proc.stdout)
        raise RuntimeError(
            f"exact library metric test {metric_test_name(test_name)} ran zero tests"
        )
    for line in combined:
        if marker in line:
            return json.loads(line.split(marker, 1)[1])

    sys.stdout.write(proc.stdout)
    raise RuntimeError(
        f"missing {marker.rstrip()} marker from exact library metric test "
        f"{metric_test_name(test_name)}"
    )


def main() -> int:
    tool_metrics = run_metric(
        "print_mode_tool_catalog_metrics",
        "TOOL_CATALOG_METRICS ",
    )

View on GitHub (pinned to 8880682c63)

Solutions

1. Run the exact command the script builds and confirm the test exists: cargo test --locked -p codewhale-tui --lib core::engine::tests::print_mode_tool_catalog_metrics -- --ignored --exact --nocapture
2. Restore #[ignore] on the metric test if it was removed -- the runner passes --ignored, so only ignored tests are selected
3. Update METRIC_TEST_MODULE or the test-name argument in scripts/measure-runtime-contract.py to the current module path and test name
4. If the test was intentionally removed, delete the corresponding run_metric() call in main()

Example fix

// crates/tui: before
#[test]
fn print_mode_tool_catalog_metrics() { ... }

// after
#[test]
#[ignore]
fn print_mode_tool_catalog_metrics() { ... }

# Before running the metric, confirm the exact ignored test exists:
import subprocess, sys

def metric_test_exists(test_name: str, module: str = "core::engine::tests") -> bool:
    listing = subprocess.run(
        ["cargo", "test", "--locked", "-p", "codewhale-tui", "--lib",
         f"{module}::{test_name}", "--", "--ignored", "--exact", "--list"],
        capture_output=True, text=True, check=False,
    )
    return f"{module}::{test_name}: test" in listing.stdout

if not metric_test_exists("print_mode_tool_catalog_metrics"):
    sys.exit("metric test missing/renamed/not #[ignore]d -- fix before measuring")

try:
    metrics = run_metric("print_mode_tool_catalog_metrics", "TOOL_CATALOG_METRICS ")
except RuntimeError as e:
    if "ran zero tests" in str(e):
        # stale test name/module: re-check the test inventory, do not retry blindly
        ...

When it happens

Trigger: The metric test was renamed, moved out of the core::engine::tests module, or deleted while scripts/measure-runtime-contract.py still names it; the test lost its #[ignore] attribute so the --ignored flag excludes it; the METRIC_TEST_MODULE constant or the test name passed to run_metric() (print_mode_tool_catalog_metrics, print_mode_runtime_contract_metrics, print_representative_runtime_context_metrics, print_skill_discovery_turn_metrics) has a typo or is stale.

Common situations: Refactoring or relocating tui tests (e.g. moving print_mode_tool_catalog_metrics to another module), removing #[ignore] during test cleanup, or adding a new run_metric() call in main() with a name that does not exactly match an existing #[ignore]d test.

Related errors

• missing {marker.rstrip()} marker from exact library metric t
• measurement command failed
• runtime-contract measurement failed with exit code {proc.ret
• baseline provenance build profile/sample count changed
• baseline receipt {field} does not match baseline_observation

AI-assisted analysis of Hmbown/CodeWhale@8880682c63 (2026-08-16). Data as JSON: /api/errors/f409ee8db8eff10e. Report an issue: GitHub.