Hmbown/CodeWhale · error · RuntimeError

exact library metric test

Error message

exact library metric test {metric_test_name(test_name)} ran zero tests

What it means

scripts/measure-runtime-contract.py runs an exact library metric test as a subprocess and parses a JSON marker from its output. This RuntimeError is raised when the combined stdout/stderr shows the test run reported 'running 0 tests', meaning the filter matched no tests and no metric was measured. The script refuses to emit a fabricated/empty metric, so it aborts loudly instead.

Solutions

1. Run the test filter manually (e.g. cargo test <filter>) and confirm at least one test matches; fix the test name in the script if it was renamed
2. Check that the crate containing the test is included in the workspace/packages being built and the required cargo features are enabled
3. Update the metric test list in scripts/measure-runtime-contract.py to reflect current test names
4. Re-run the script after cargo build to ensure the test binary is fresh

Example fix

// before (script references removed test)
run_metric("bench_exact_token_count")
// after
cargo test exact_token_count  # verify it exists, then:
run_metric("exact_token_count")

import subprocess
output = subprocess.run(["cargo", "test", test_filter, "--", "--list"], capture_output=True, text=True)
if not output.stdout.strip():
    raise SystemExit(f"filter {test_filter!r} matches no tests; fix before running measure-runtime-contract.py")

try:
    metric = run_metric(test_name)
except RuntimeError as e:
    if "ran zero tests" in str(e):
        print(f"test {test_name} not found; check rename/removal")
        sys.exit(2)
    raise

When it happens

Trigger: The test name passed via metric_test_name/test_name no longer matches any test in the test binary (renamed or removed test), or the filter string passed to the test runner is misspelled or over-narrow, so cargo's test harness prints 'running 0 tests'.

Common situations: A test was renamed during refactoring but scripts/measure-runtime-contract.py still references the old name; a feature-gated test is compiled out so the filter matches nothing; running the script on a workspace subset where the crate containing the test is not built; typos when adding a new metric entry.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

• missing marker from exact library metric test
• AT-SPI action failed
• cargo metadata failed with exit code
• exact library measurement test
• exact library measurement test

AI-assisted analysis of Hmbown/CodeWhale@433685b202 (2026-09-15). Data as JSON: /api/errors/f409ee8db8eff10e. Report an issue: GitHub.

Appendix: source

Thrown at scripts/measure-runtime-contract.py:52

        "--ignored",
        "--exact",
        "--nocapture",
        "--test-threads=1",
    ]


def run_metric(test_name: str, marker: str) -> dict:
    cmd = metric_command(test_name)
    proc = subprocess.run(cmd, text=True, capture_output=True, check=False)
    sys.stderr.write(proc.stderr)
    if proc.returncode != 0:
        sys.stdout.write(proc.stdout)
        proc.check_returncode()

    combined = proc.stdout.splitlines() + proc.stderr.splitlines()
    if re.search(r"\brunning\s+0\s+tests?\b", "\n".join(combined)):
        sys.stdout.write(proc.stdout)
        raise RuntimeError(
            f"exact library metric test {metric_test_name(test_name)} ran zero tests"
        )
    for line in combined:
        if marker in line:
            return json.loads(line.split(marker, 1)[1])

    sys.stdout.write(proc.stdout)
    raise RuntimeError(
        f"missing {marker.rstrip()} marker from exact library metric test "
        f"{metric_test_name(test_name)}"
    )


def main() -> int:
    tool_metrics = run_metric(
        "print_mode_tool_catalog_metrics",
        "TOOL_CATALOG_METRICS ",
    )

View on GitHub (pinned to 433685b202)