Hmbown/CodeWhale · error · RuntimeError

missing marker from exact library metric test

Error message

missing {marker.rstrip()} marker from exact library metric test {metric_test_name(test_name)}

What it means

scripts/measure-runtime-contract.py runs a metric test subprocess and expects one output line containing a sentinel marker followed by JSON metrics. This RuntimeError is raised when the process succeeded and ran tests, but no line contained the expected marker, so no metric payload could be parsed.

Solutions

1. Confirm the test actually prints a line containing the marker (grep the test source for the marker prefix)
2. Re-run the test manually and inspect its raw output for the marker line
3. If the marker format changed, update either the script's marker or the test's print statement so they match again
4. Check the test is not ignored or feature-gated out at runtime

Example fix

// before (test no longer prints marker)
println!("RESULT {}", json)
// after (match the script's marker)
println!("{}{}", MARKER, json)

import subprocess
out = subprocess.run(["cargo", "test", test_name, "--", "--nocapture"], capture_output=True, text=True)
assert marker in out.stdout + out.stderr, f"test {test_name} does not emit marker {marker!r}"

try:
    metric = run_metric(test_name)
except RuntimeError as e:
    if "marker" in str(e):
        print(f"{test_name} ran but never printed {marker!r}; restore the metric print")
        sys.exit(3)
    raise

When it happens

Trigger: The test ran (not zero tests) but did not print the marker line — e.g. the test's metric-printing code was removed or renamed, the test was skipped/ignored so it never executed its print, or stdout/stderr buffering or encoding mangled the marker line.

Common situations: Refactoring a benchmark/metric test and forgetting to keep the println! with the agreed marker prefix; marking the test #[ignore]; marker string divergence between the script and the test after an edit; the test panics partway (though that usually surfaces as a nonzero exit first).

Related errors

• exact library metric test
• AT-SPI action failed
• cargo metadata failed with exit code
• Cargo metadata is not valid JSON
• could not parse match block after

AI-assisted analysis of Hmbown/CodeWhale@433685b202 (2026-09-15). Data as JSON: /api/errors/d6e063e5a04600b8. Report an issue: GitHub.

Appendix: source

Thrown at scripts/measure-runtime-contract.py:60

    cmd = metric_command(test_name)
    proc = subprocess.run(cmd, text=True, capture_output=True, check=False)
    sys.stderr.write(proc.stderr)
    if proc.returncode != 0:
        sys.stdout.write(proc.stdout)
        proc.check_returncode()

    combined = proc.stdout.splitlines() + proc.stderr.splitlines()
    if re.search(r"\brunning\s+0\s+tests?\b", "\n".join(combined)):
        sys.stdout.write(proc.stdout)
        raise RuntimeError(
            f"exact library metric test {metric_test_name(test_name)} ran zero tests"
        )
    for line in combined:
        if marker in line:
            return json.loads(line.split(marker, 1)[1])

    sys.stdout.write(proc.stdout)
    raise RuntimeError(
        f"missing {marker.rstrip()} marker from exact library metric test "
        f"{metric_test_name(test_name)}"
    )


def main() -> int:
    tool_metrics = run_metric(
        "print_mode_tool_catalog_metrics",
        "TOOL_CATALOG_METRICS ",
    )
    prompt_metrics = run_metric(
        "print_mode_runtime_contract_metrics",
        "RUNTIME_CONTRACT_METRICS ",
    )
    representative_context_metrics = run_metric(
        "print_representative_runtime_context_metrics",
        "REPRESENTATIVE_CONTEXT_METRICS ",
    )

View on GitHub (pinned to 433685b202)