google-gemini/gemini-cli · error · ValueError

Invalid EVAL_CONFIG JSON: {e}

Error message

Invalid EVAL_CONFIG JSON: {e}

What it means

This ValueError is raised in cloud_runner.main() when the EVAL_CONFIG environment variable contains text that cannot be parsed as JSON at all (json.JSONDecodeError). The inner try/except catches the decode error and re-raises it as a ValueError with the parser's error message. It fires before any eval logic runs, at config deserialization time.

Source

Thrown at tools/caretaker-agent/evals/triage/cloud_runner.py:19

"""
Cloud Run Job Entrypoint for Gemini CLI Triage Evaluation Suite.
Reads EVAL_CONFIG JSON environment variable, invokes run_suite(), and syncs results to GCS.
"""

import os
import json
from evals.triage.runner import run_suite
from evals.triage.helpers.sync_to_gcs import sync_results_to_gcs


def main() -> None:
    config_str = os.environ.get("EVAL_CONFIG", "{}")
    try:
        cfg = json.loads(config_str) if config_str else {}
        if not isinstance(cfg, dict):
            raise ValueError(f"EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}")
    except json.JSONDecodeError as e:
        raise ValueError(f"Invalid EVAL_CONFIG JSON: {e}") from e

    print("========================================================")
    print(" 🚀 Running Gemini CLI Triage Evaluation Suite (Cloud Run)")
    print("========================================================")
    if cfg:
        print(f"[EVAL_CONFIG] Loaded configuration: {cfg}")

    # 1. Execute benchmark suite directly via run_suite()
    run_suite(
        filter_issues=cfg.get("issues"),
        concurrency=cfg.get("concurrency", 5),
        note=cfg.get("note")
    )

    # 2. Sync evaluation run results to GCS bucket
    sync_results_to_gcs()

View on GitHub (pinned to 5024443c72)

Solutions

1. Validate EVAL_CONFIG with a JSON linter or python -c "import json; json.loads(open('/dev/stdin').read())" before launching the Cloud Run job.
2. Ensure the shell quoting preserves the JSON: use single quotes around the whole value (EVAL_CONFIG='{"issues":[1]}') in bash.
3. If the value comes from a file or secret manager, verify the stored content is complete and not truncated.
4. Leave EVAL_CONFIG unset to accept the '{}' default and pass options via CLI flags or other env vars instead.

Example fix

# before (bash strips inner double quotes)
EVAL_CONFIG={"issues":[28052]}
# after
EVAL_CONFIG='{"issues":[28052]}'

import json

def validate_eval_config_json(raw: str) -> bool:
    try:
        json.loads(raw)
        return True
    except json.JSONDecodeError:
        return False

# Use before launch:
# raw = os.environ.get('EVAL_CONFIG', '{}')
# assert validate_eval_config_json(raw), f'EVAL_CONFIG is not valid JSON: {raw}'

try:
    cfg = json.loads(os.environ.get('EVAL_CONFIG', '{}') or '{}')
except json.JSONDecodeError as e:
    print(f'[CONFIG] EVAL_CONFIG JSON invalid: {e}. Using empty config.')
    cfg = {}

When it happens

Trigger: EVAL_CONFIG is set to malformed JSON such as an unclosed brace ('{issues:[1]}'), single quotes ('{'issues':[1]}'), trailing commas, or a raw non-JSON string like 'issues=1,2,3'. A shell variable expansion injected a value with unescaped quotes.

Common situations: Setting EVAL_CONFIG in a shell without proper quoting so the shell strips quotes, leaving invalid JSON. A CI/CD secret or template variable contains a placeholder like '{{issues}}' that was never substituted. Hand-editing the env var and introducing a syntax error.

Related errors

• EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}
• Missing required environment variable '{name}'. Please ensur
• ${errorMessages.join('\n')}\nPlease fix the configuration fi
• Environment variable 'FIRESTORE_DOC' is required but was not
• Firestore document specification must be a JSON object.

AI-assisted analysis of google-gemini/gemini-cli@5024443c72 (2026-08-12). Data as JSON: /api/errors/c2c9b4c9df5f6f38. Report an issue: GitHub.