abhigyanpatwari/GitNexus · error · ValueError

task {task_id} oracle requires exactly command and files

Error message

task {task_id} oracle requires exactly command and files

What it means

Raised by validate_oracle_declaration when a benchmark task's 'oracle' field is not a dict whose key set is exactly {"command", "files"}. The harness requires a precise declarative shape so the hidden behavioral oracle is unambiguous; any missing key, extra key, wrong type, or absent oracle is rejected before any file is read. The task_id in the message comes from task.get("id", "<unknown>") so it identifies which task declaration is malformed.

Source

Thrown at eval/workflow_bench/oracle_assets.py:89

    if not isinstance(value, str) or not value:
        raise ValueError(f"{label} must be a nonblank relative path")
    if "\\" in value or "\x00" in value or len(value.encode()) > MAX_ORACLE_PATH_BYTES:
        raise ValueError(f"{label} is not a bounded portable path: {value!r}")
    relative = PurePosixPath(value)
    if relative.is_absolute() or not relative.parts or any(part in {"", ".", ".."} for part in relative.parts):
        raise ValueError(f"{label} must not be absolute or traverse parents: {value!r}")
    if relative.parts[0] == ".git":
        raise ValueError(f"{label} cannot target git metadata: {value!r}")
    return relative


def validate_oracle_declaration(task: dict[str, Any]) -> None:
    """Validate the declarative shape without reading harness-owned files."""

    task_id = str(task.get("id", "<unknown>"))
    oracle = task.get("oracle")
    if not isinstance(oracle, dict) or set(oracle) != {"command", "files"}:
        raise ValueError(f"task {task_id} oracle requires exactly command and files")
    command = oracle.get("command")
    if (
        not isinstance(command, str)
        or not command.strip()
        or len(command.encode()) > MAX_ORACLE_COMMAND_BYTES
        or "\x00" in command
    ):
        raise ValueError(f"task {task_id} oracle command must be nonblank and bounded")
    files = oracle.get("files")
    if not isinstance(files, list) or not files or len(files) > MAX_ORACLE_FILES:
        raise ValueError(f"task {task_id} oracle files must contain 1..{MAX_ORACLE_FILES} entries")
    sources: set[str] = set()
    targets: set[str] = set()
    for index, declaration in enumerate(files):
        if not isinstance(declaration, dict) or set(declaration) != {"source", "target"}:
            raise ValueError(f"task {task_id} oracle file {index} requires exactly source and target")
        source = _bounded_relative_path(declaration.get("source"), label=f"task {task_id} oracle source")
        target = _bounded_relative_path(declaration.get("target"), label=f"task {task_id} oracle target")

View on GitHub (pinned to d540b00184)

Solutions

  1. Set task["oracle"] to a dict with exactly two keys: 'command' (str) and 'files' (list) — nothing else.
  2. Run validate_oracle_declaration(task) locally before invoking capture_task_oracle to surface schema errors early.
  3. If you added metadata fields, move them to the top-level task dict, not inside 'oracle'.
  4. Check for typos: it is 'files' (plural) and 'command', not 'file'/'cmd'.

Example fix

// before
task = {"id": "t1", "oracle": {"command": "pytest"}}
// after
task = {"id": "t1", "oracle": {"command": "pytest", "files": [{"source": "expect.txt", "target": "expect.txt"}]}}
Defensive patterns

Strategy: validation

Validate before calling

from eval.workflow_bench.oracle_assets import validate_oracle_declaration

def safe_validate(task):
    oracle = task.get("oracle")
    if not isinstance(oracle, dict) or set(oracle) != {"command", "files"}:
        raise ValueError("oracle must be a dict with exactly keys 'command' and 'files'")
    validate_oracle_declaration(task)

Type guard

def is_well_shaped_oracle(task: dict) -> bool:
    oracle = task.get("oracle")
    return isinstance(oracle, dict) and set(oracle) == {"command", "files"}

Try / catch

try:
    validate_oracle_declaration(task)
except ValueError as exc:
    raise SystemExit(f"Bad oracle declaration for task {task.get('id')}: {exc}") from exc

Prevention

When it happens

Trigger: Calling validate_oracle_declaration(task) or capture_task_oracle(task) where task["oracle"] is None, a string, a list, or a dict containing keys other than exactly 'command' and 'files' (e.g. missing 'files', or an extra 'description'/'timeout' key, or a typo like 'file' instead of 'files').

Common situations: Authoring a new benchmark task YAML/JSON and forgetting the 'files' array; adding documentation metadata inside the oracle dict; renaming keys during a schema refactor; loading a task fixture that predates this strict schema; typos like 'file' vs 'files' or 'cmd' vs 'command'.

Related errors

AI-assisted analysis of abhigyanpatwari/GitNexus@d540b00184 (2026-08-12). Data as JSON: /api/errors/f28eef4a245c85d9. Report an issue: GitHub.