mvanhorn/last30days-skill · error · RuntimeError

{repo_dir.name} emitted the agent JSON profile; the evaluato

Error message

{repo_dir.name} emitted the agent JSON profile; the evaluator requires the raw Report (--json-profile=raw).

What it means

A shape guard in run_repo(): the evaluator needs the raw Report JSON (with ranked_candidates) to score rankings, but the subprocess emitted the agent profile JSON (identified by schema_version without ranked_candidates). The engine's --json-profile flag detection failed, so the evaluator refuses to score an empty result instead of silently passing.

Source

Thrown at skills/last30days/scripts/evaluate_search_quality.py:363

    if mock:
        cmd.append("--mock")
    result = subprocess.run(
        cmd,
        cwd=repo_dir,
        env=env,
        capture_output=True,
        text=True,
        timeout=timeout_seconds,
        check=False,
    )
    if result.returncode != 0:
        raise RuntimeError(f"{repo_dir.name} failed for '{topic}' with exit {result.returncode}\n{result.stderr.strip()}")
    payload = json.loads(result.stdout)
    # Shape guard: the evaluator compares raw Report fields. If the engine
    # emitted the agent profile anyway (flag detection missed a future
    # spelling), fail loudly instead of scoring empty ranked_candidates.
    if "schema_version" in payload and "ranked_candidates" not in payload:
        raise RuntimeError(
            f"{repo_dir.name} emitted the agent JSON profile; the evaluator "
            "requires the raw Report (--json-profile=raw)."
        )
    return payload


def create_worktree(rev: str) -> Path:
    worktree_dir = Path(tempfile.mkdtemp(prefix="last30days-eval-"))
    subprocess.run(
        ["git", "worktree", "add", "--detach", str(worktree_dir), rev],
        cwd=REPO_ROOT,
        check=True,
        capture_output=True,
        text=True,
    )
    return worktree_dir

View on GitHub (pinned to c7460f6114)

Solutions

1. Confirm the cmd list in run_repo() actually includes --json-profile=raw for the revision under test.
2. Check the revision's argparse: if the flag was renamed, pass the new spelling or pin the evaluator to revisions that support it.
3. If the default changed, pass the raw-profile flag explicitly rather than relying on defaults.

Example fix

# before
cmd = [sys.executable, 'scripts/last30days.py', topic, '--json']

# after
cmd = [sys.executable, 'scripts/last30days.py', topic, '--json', '--json-profile=raw']

payload = json.loads(result.stdout)
if 'ranked_candidates' not in payload:
    raise RuntimeError(
        f'unexpected engine output shape; keys={sorted(payload)[:8]}; '
        'expected raw Report with ranked_candidates'
    )

def is_raw_report(payload: Any) -> TypeGuard[dict]:
    return isinstance(payload, dict) and 'ranked_candidates' in payload

When it happens

Trigger: Running the engine with --json-profile=raw (or omitting agent profile flags) but a future/renamed flag spelling means the agent profile is still emitted: payload has 'schema_version' key and lacks 'ranked_candidates'. Typical when the evaluated revision renamed the flag or changed default emit behavior.

Common situations: Benchmarking across revisions where the JSON profile flag changed name; engine default flipped to agent profile; the harness's cmd construction (around line 340-356) missing the --json-profile=raw argument for a new revision.

Related errors

• {repo_dir.name} failed for '{topic}' with exit {result.retur
• [CompetitorsPlan] Top-level must be a dict of {entity: {targ
• {label} {path} must be a top-level JSON object, got {type(pa
• Nominations bundle {path} must carry a top-level "nomination
• Pending discovery report {path} must carry a top-level \"rep

AI-assisted analysis of mvanhorn/last30days-skill@c7460f6114 (2026-08-15). Data as JSON: /api/errors/97146e6500c44dda. Report an issue: GitHub.