{"record":{"id":"97146e6500c44dda","repo":"mvanhorn/last30days-skill","slug":"repo-dir-name-emitted-the-agent-json-profile-th","errorCode":null,"errorMessage":"{repo_dir.name} emitted the agent JSON profile; the evaluator requires the raw Report (--json-profile=raw).","messagePattern":"(.+?) emitted the agent JSON profile; the evaluator requires the raw Report \\(--json-profile=raw\\)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"skills/last30days/scripts/evaluate_search_quality.py","lineNumber":363,"sourceCode":"    if mock:\n        cmd.append(\"--mock\")\n    result = subprocess.run(\n        cmd,\n        cwd=repo_dir,\n        env=env,\n        capture_output=True,\n        text=True,\n        timeout=timeout_seconds,\n        check=False,\n    )\n    if result.returncode != 0:\n        raise RuntimeError(f\"{repo_dir.name} failed for '{topic}' with exit {result.returncode}\\n{result.stderr.strip()}\")\n    payload = json.loads(result.stdout)\n    # Shape guard: the evaluator compares raw Report fields. If the engine\n    # emitted the agent profile anyway (flag detection missed a future\n    # spelling), fail loudly instead of scoring empty ranked_candidates.\n    if \"schema_version\" in payload and \"ranked_candidates\" not in payload:\n        raise RuntimeError(\n            f\"{repo_dir.name} emitted the agent JSON profile; the evaluator \"\n            \"requires the raw Report (--json-profile=raw).\"\n        )\n    return payload\n\n\ndef create_worktree(rev: str) -> Path:\n    worktree_dir = Path(tempfile.mkdtemp(prefix=\"last30days-eval-\"))\n    subprocess.run(\n        [\"git\", \"worktree\", \"add\", \"--detach\", str(worktree_dir), rev],\n        cwd=REPO_ROOT,\n        check=True,\n        capture_output=True,\n        text=True,\n    )\n    return worktree_dir\n\n","sourceCodeStart":345,"sourceCodeEnd":381,"githubUrl":"https://github.com/mvanhorn/last30days-skill/blob/c7460f6114449ddfe6ea3fc2f23c3d910c0e740c/skills/last30days/scripts/evaluate_search_quality.py#L345-L381","documentation":"A shape guard in run_repo(): the evaluator needs the raw Report JSON (with ranked_candidates) to score rankings, but the subprocess emitted the agent profile JSON (identified by schema_version without ranked_candidates). The engine's --json-profile flag detection failed, so the evaluator refuses to score an empty result instead of silently passing.","triggerScenarios":"Running the engine with --json-profile=raw (or omitting agent profile flags) but a future/renamed flag spelling means the agent profile is still emitted: payload has 'schema_version' key and lacks 'ranked_candidates'. Typical when the evaluated revision renamed the flag or changed default emit behavior.","commonSituations":"Benchmarking across revisions where the JSON profile flag changed name; engine default flipped to agent profile; the harness's cmd construction (around line 340-356) missing the --json-profile=raw argument for a new revision.","solutions":["Confirm the cmd list in run_repo() actually includes --json-profile=raw for the revision under test.","Check the revision's argparse: if the flag was renamed, pass the new spelling or pin the evaluator to revisions that support it.","If the default changed, pass the raw-profile flag explicitly rather than relying on defaults."],"exampleFix":"# before\ncmd = [sys.executable, 'scripts/last30days.py', topic, '--json']\n\n# after\ncmd = [sys.executable, 'scripts/last30days.py', topic, '--json', '--json-profile=raw']","handlingStrategy":"validation","validationCode":"payload = json.loads(result.stdout)\nif 'ranked_candidates' not in payload:\n    raise RuntimeError(\n        f'unexpected engine output shape; keys={sorted(payload)[:8]}; '\n        'expected raw Report with ranked_candidates'\n    )","typeGuard":"def is_raw_report(payload: Any) -> TypeGuard[dict]:\n    return isinstance(payload, dict) and 'ranked_candidates' in payload","tryCatchPattern":null,"preventionTips":["Always pass the raw-profile flag explicitly; never rely on engine defaults in harnesses.","Pin evaluated revisions whose flag surface you have verified."],"tags":["evaluation","json","schema","subprocess"],"backgroundTag":null,"analyzedSha":"c7460f6114449ddfe6ea3fc2f23c3d910c0e740c","analyzedAt":"2026-08-15T03:34:49.540Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}