Hmbown/CodeWhale · error · RuntimeError

missing {marker.rstrip()} marker from exact library metric t

Error message

missing {marker.rstrip()} marker from exact library metric test {metric_test_name(test_name)}

What it means

run_metric() ran the metric test to a zero exit code but no line of combined stdout/stderr contained the expected marker string (e.g. 'TOOL_CATALOG_METRICS ') followed by JSON. The script extracts the metrics object by splitting each line on the marker, so a missing marker makes the receipt impossible to build and the run aborts after dumping the test stdout for inspection.

Source

Thrown at scripts/measure-runtime-contract.py:60

    cmd = metric_command(test_name)
    proc = subprocess.run(cmd, text=True, capture_output=True, check=False)
    sys.stderr.write(proc.stderr)
    if proc.returncode != 0:
        sys.stdout.write(proc.stdout)
        proc.check_returncode()

    combined = proc.stdout.splitlines() + proc.stderr.splitlines()
    if re.search(r"\brunning\s+0\s+tests?\b", "\n".join(combined)):
        sys.stdout.write(proc.stdout)
        raise RuntimeError(
            f"exact library metric test {metric_test_name(test_name)} ran zero tests"
        )
    for line in combined:
        if marker in line:
            return json.loads(line.split(marker, 1)[1])

    sys.stdout.write(proc.stdout)
    raise RuntimeError(
        f"missing {marker.rstrip()} marker from exact library metric test "
        f"{metric_test_name(test_name)}"
    )


def main() -> int:
    tool_metrics = run_metric(
        "print_mode_tool_catalog_metrics",
        "TOOL_CATALOG_METRICS ",
    )
    prompt_metrics = run_metric(
        "print_mode_runtime_contract_metrics",
        "RUNTIME_CONTRACT_METRICS ",
    )
    representative_context_metrics = run_metric(
        "print_representative_runtime_context_metrics",
        "REPRESENTATIVE_CONTEXT_METRICS ",
    )

View on GitHub (pinned to 8880682c63)

Solutions

Grep both sides for the marker and align them exactly, including the trailing space: rg "TOOL_CATALOG_METRICS" crates/tui scripts/measure-runtime-contract.py
Run the cargo command by hand with --nocapture and confirm the marker line actually appears in the output
Ensure the test prints the marker unconditionally (println!("TOOL_CATALOG_METRICS {json}")) after computing metrics

Example fix

// crates/tui test: before
println!("TOOLCAT_METRICS {}", serde_json::to_string(&metrics)?);

// after (matches the marker argument in main())
println!("TOOL_CATALOG_METRICS {}", serde_json::to_string(&metrics)?);

Defensive patterns

Strategy: validation

Validate before calling

# Before running, assert the test source prints the exact marker (incl. trailing space):
import re, subprocess

source = subprocess.run(
    ["rg", "-l", "core::engine::tests"], capture_output=True, text=True
)  # or read the known test file
marker = "TOOL_CATALOG_METRICS "
test_src = open("crates/tui/src/engine.rs").read()  # path of the metric test
assert marker.strip() in test_src, f"test no longer prints {marker.strip()}"

Try / catch

try:
    metrics = run_metric(test_name, marker)
except RuntimeError as e:
    if "missing" in str(e) and "marker" in str(e):
        # marker drift between script and test; align the literals, then re-run
        ...

Prevention

Define the marker string once (a shared constant) instead of duplicating the literal in the script and the test
Keep the marker println! unconditional and last in the test so it always reaches stdout under --nocapture
Include the trailing space convention in the marker constant on both sides

When it happens

Trigger: The marker literal printed by the test drifted from the string passed to run_metric() (renamed, or the trailing space in 'TOOL_CATALOG_METRICS ' was dropped); the print statement was made conditional or moved behind an early return; the marker is emitted by a different test than the one named; output went to a stream the script does not combine (it does combine stdout and stderr, so this is rare).

Common situations: Renaming the metrics printer or its marker constant on one side (test vs script) only; refactoring the test so the marker prints only on some code paths; a new metric added to the test without updating main()'s marker argument.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@8880682c63 (2026-08-16). Data as JSON: /api/errors/d6e063e5a04600b8. Report an issue: GitHub.