{"record":{"id":"93c289a584f4992d","repo":"google-gemini/gemini-cli","slug":"eval-config-must-be-a-json-object-got-type-cfg","errorCode":null,"errorMessage":"EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}","messagePattern":"EVAL_CONFIG must be a JSON object, got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"tools/caretaker-agent/evals/triage/cloud_runner.py","lineNumber":17,"sourceCode":"\"\"\"\nCloud Run Job Entrypoint for Gemini CLI Triage Evaluation Suite.\nReads EVAL_CONFIG JSON environment variable, invokes run_suite(), and syncs results to GCS.\n\"\"\"\n\nimport os\nimport json\nfrom evals.triage.runner import run_suite\nfrom evals.triage.helpers.sync_to_gcs import sync_results_to_gcs\n\n\ndef main() -> None:\n    config_str = os.environ.get(\"EVAL_CONFIG\", \"{}\")\n    try:\n        cfg = json.loads(config_str) if config_str else {}\n        if not isinstance(cfg, dict):\n            raise ValueError(f\"EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}\")\n    except json.JSONDecodeError as e:\n        raise ValueError(f\"Invalid EVAL_CONFIG JSON: {e}\") from e\n\n    print(\"========================================================\")\n    print(\" 🚀 Running Gemini CLI Triage Evaluation Suite (Cloud Run)\")\n    print(\"========================================================\")\n    if cfg:\n        print(f\"[EVAL_CONFIG] Loaded configuration: {cfg}\")\n\n    # 1. Execute benchmark suite directly via run_suite()\n    run_suite(\n        filter_issues=cfg.get(\"issues\"),\n        concurrency=cfg.get(\"concurrency\", 5),\n        note=cfg.get(\"note\")\n    )\n\n    # 2. Sync evaluation run results to GCS bucket\n    sync_results_to_gcs()","sourceCodeStart":1,"sourceCodeEnd":35,"githubUrl":"https://github.com/google-gemini/gemini-cli/blob/5024443c7217464a66e98f80d73172a26440bd8f/tools/caretaker-agent/evals/triage/cloud_runner.py#L1-L35","documentation":"This ValueError is raised in cloud_runner.main() when the EVAL_CONFIG environment variable parses as valid JSON but the top-level value is not a JSON object (dict). The runner expects a configuration object with keys like 'issues', 'concurrency', and 'note'; a JSON array, string, number, or boolean is rejected. It is a configuration-shape guard at process startup.","triggerScenarios":"Setting EVAL_CONFIG to a JSON array (e.g. EVAL_CONFIG='[1,2,3]'), a bare string (e.g. EVAL_CONFIG='\"hello\"'), a number (e.g. EVAL_CONFIG='5'), or the literal string 'null'. json.loads succeeds but the isinstance(cfg, dict) check at cloud_runner.py:16 fails.","commonSituations":"A user wraps the issue list in brackets thinking EVAL_CONFIG='[28052,25693]' is the issues array directly, instead of EVAL_CONFIG='{\"issues\":[28052,25693]}'. A CI pipeline template injects EVAL_CONFIG from a variable that was serialized as a JSON list rather than an object. Copy-pasting a partial JSON fragment that omits the outer braces.","solutions":["Set EVAL_CONFIG to a JSON object: EVAL_CONFIG='{\"issues\":[28052,25693],\"concurrency\":5,\"note\":\"run\"}'.","If you only need defaults, unset EVAL_CONFIG entirely so it defaults to '{}' (empty object).","Validate the JSON shape locally with: python -c \"import json,os; print(type(json.loads(os.environ['EVAL_CONFIG'])))\" before deploying to Cloud Run.","Check the Cloud Run job's env-var configuration in the console or gcloud for stray quotes or array syntax."],"exampleFix":"# before\nEVAL_CONFIG='[28052,25693]'\n# after\nEVAL_CONFIG='{\"issues\":[28052,25693]}'","handlingStrategy":"validation","validationCode":"import json, os\n\ndef load_eval_config() -> dict:\n    raw = os.environ.get('EVAL_CONFIG', '{}')\n    cfg = json.loads(raw) if raw else {}\n    if not isinstance(cfg, dict):\n        raise ValueError(f'EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}')\n    return cfg","typeGuard":"import json\n\ndef is_eval_config_object(raw: str) -> bool:\n    try:\n        return isinstance(json.loads(raw or '{}'), dict)\n    except json.JSONDecodeError:\n        return False","tryCatchPattern":"try:\n    cfg = load_eval_config()\nexcept ValueError as e:\n    print(f'[CONFIG] Falling back to empty config: {e}')\n    cfg = {}","preventionTips":["Always wrap EVAL_CONFIG in a JSON object, never a bare array or scalar.","Provide a documented example in the repo README and Cloud Run job definition.","Add a startup assertion that logs the parsed config type before run_suite is called."],"tags":["configuration","env-var","cloud-run","evals","json","python"],"backgroundTag":null,"analyzedSha":"5024443c7217464a66e98f80d73172a26440bd8f","analyzedAt":"2026-08-12T06:01:53.711Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}