{"record":{"id":"c2c9b4c9df5f6f38","repo":"google-gemini/gemini-cli","slug":"invalid-eval-config-json-e","errorCode":null,"errorMessage":"Invalid EVAL_CONFIG JSON: {e}","messagePattern":"Invalid EVAL_CONFIG JSON: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"tools/caretaker-agent/evals/triage/cloud_runner.py","lineNumber":19,"sourceCode":"\"\"\"\nCloud Run Job Entrypoint for Gemini CLI Triage Evaluation Suite.\nReads EVAL_CONFIG JSON environment variable, invokes run_suite(), and syncs results to GCS.\n\"\"\"\n\nimport os\nimport json\nfrom evals.triage.runner import run_suite\nfrom evals.triage.helpers.sync_to_gcs import sync_results_to_gcs\n\n\ndef main() -> None:\n    config_str = os.environ.get(\"EVAL_CONFIG\", \"{}\")\n    try:\n        cfg = json.loads(config_str) if config_str else {}\n        if not isinstance(cfg, dict):\n            raise ValueError(f\"EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}\")\n    except json.JSONDecodeError as e:\n        raise ValueError(f\"Invalid EVAL_CONFIG JSON: {e}\") from e\n\n    print(\"========================================================\")\n    print(\" 🚀 Running Gemini CLI Triage Evaluation Suite (Cloud Run)\")\n    print(\"========================================================\")\n    if cfg:\n        print(f\"[EVAL_CONFIG] Loaded configuration: {cfg}\")\n\n    # 1. Execute benchmark suite directly via run_suite()\n    run_suite(\n        filter_issues=cfg.get(\"issues\"),\n        concurrency=cfg.get(\"concurrency\", 5),\n        note=cfg.get(\"note\")\n    )\n\n    # 2. Sync evaluation run results to GCS bucket\n    sync_results_to_gcs()\n\n","sourceCodeStart":1,"sourceCodeEnd":37,"githubUrl":"https://github.com/google-gemini/gemini-cli/blob/5024443c7217464a66e98f80d73172a26440bd8f/tools/caretaker-agent/evals/triage/cloud_runner.py#L1-L37","documentation":"This ValueError is raised in cloud_runner.main() when the EVAL_CONFIG environment variable contains text that cannot be parsed as JSON at all (json.JSONDecodeError). The inner try/except catches the decode error and re-raises it as a ValueError with the parser's error message. It fires before any eval logic runs, at config deserialization time.","triggerScenarios":"EVAL_CONFIG is set to malformed JSON such as an unclosed brace ('{issues:[1]}'), single quotes ('{'issues':[1]}'), trailing commas, or a raw non-JSON string like 'issues=1,2,3'. A shell variable expansion injected a value with unescaped quotes.","commonSituations":"Setting EVAL_CONFIG in a shell without proper quoting so the shell strips quotes, leaving invalid JSON. A CI/CD secret or template variable contains a placeholder like '{{issues}}' that was never substituted. Hand-editing the env var and introducing a syntax error.","solutions":["Validate EVAL_CONFIG with a JSON linter or python -c \"import json; json.loads(open('/dev/stdin').read())\" before launching the Cloud Run job.","Ensure the shell quoting preserves the JSON: use single quotes around the whole value (EVAL_CONFIG='{\"issues\":[1]}') in bash.","If the value comes from a file or secret manager, verify the stored content is complete and not truncated.","Leave EVAL_CONFIG unset to accept the '{}' default and pass options via CLI flags or other env vars instead."],"exampleFix":"# before (bash strips inner double quotes)\nEVAL_CONFIG={\"issues\":[28052]}\n# after\nEVAL_CONFIG='{\"issues\":[28052]}'","handlingStrategy":"try-catch","validationCode":"import json\n\ndef validate_eval_config_json(raw: str) -> bool:\n    try:\n        json.loads(raw)\n        return True\n    except json.JSONDecodeError:\n        return False\n\n# Use before launch:\n# raw = os.environ.get('EVAL_CONFIG', '{}')\n# assert validate_eval_config_json(raw), f'EVAL_CONFIG is not valid JSON: {raw}'","typeGuard":null,"tryCatchPattern":"try:\n    cfg = json.loads(os.environ.get('EVAL_CONFIG', '{}') or '{}')\nexcept json.JSONDecodeError as e:\n    print(f'[CONFIG] EVAL_CONFIG JSON invalid: {e}. Using empty config.')\n    cfg = {}","preventionTips":["Single-quote the entire JSON value in bash to preserve inner double quotes.","Lint the env var with a JSON validator in CI before deploying.","Prefer leaving EVAL_CONFIG unset to accept the safe '{}' default when no overrides are needed."],"tags":["configuration","env-var","cloud-run","json","parsing","python"],"backgroundTag":null,"analyzedSha":"5024443c7217464a66e98f80d73172a26440bd8f","analyzedAt":"2026-08-12T06:01:53.711Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}