google-gemini/gemini-cli · error · ValueError

EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}

Error message

EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}

What it means

This ValueError is raised in cloud_runner.main() when the EVAL_CONFIG environment variable parses as valid JSON but the top-level value is not a JSON object (dict). The runner expects a configuration object with keys like 'issues', 'concurrency', and 'note'; a JSON array, string, number, or boolean is rejected. It is a configuration-shape guard at process startup.

Source

Thrown at tools/caretaker-agent/evals/triage/cloud_runner.py:17

"""
Cloud Run Job Entrypoint for Gemini CLI Triage Evaluation Suite.
Reads EVAL_CONFIG JSON environment variable, invokes run_suite(), and syncs results to GCS.
"""

import os
import json
from evals.triage.runner import run_suite
from evals.triage.helpers.sync_to_gcs import sync_results_to_gcs


def main() -> None:
    config_str = os.environ.get("EVAL_CONFIG", "{}")
    try:
        cfg = json.loads(config_str) if config_str else {}
        if not isinstance(cfg, dict):
            raise ValueError(f"EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}")
    except json.JSONDecodeError as e:
        raise ValueError(f"Invalid EVAL_CONFIG JSON: {e}") from e

    print("========================================================")
    print(" 🚀 Running Gemini CLI Triage Evaluation Suite (Cloud Run)")
    print("========================================================")
    if cfg:
        print(f"[EVAL_CONFIG] Loaded configuration: {cfg}")

    # 1. Execute benchmark suite directly via run_suite()
    run_suite(
        filter_issues=cfg.get("issues"),
        concurrency=cfg.get("concurrency", 5),
        note=cfg.get("note")
    )

    # 2. Sync evaluation run results to GCS bucket
    sync_results_to_gcs()

View on GitHub (pinned to 5024443c72)

Solutions

1. Set EVAL_CONFIG to a JSON object: EVAL_CONFIG='{"issues":[28052,25693],"concurrency":5,"note":"run"}'.
2. If you only need defaults, unset EVAL_CONFIG entirely so it defaults to '{}' (empty object).
3. Validate the JSON shape locally with: python -c "import json,os; print(type(json.loads(os.environ['EVAL_CONFIG'])))" before deploying to Cloud Run.
4. Check the Cloud Run job's env-var configuration in the console or gcloud for stray quotes or array syntax.

Example fix

# before
EVAL_CONFIG='[28052,25693]'
# after
EVAL_CONFIG='{"issues":[28052,25693]}'

import json, os

def load_eval_config() -> dict:
    raw = os.environ.get('EVAL_CONFIG', '{}')
    cfg = json.loads(raw) if raw else {}
    if not isinstance(cfg, dict):
        raise ValueError(f'EVAL_CONFIG must be a JSON object, got {type(cfg).__name__}')
    return cfg

import json

def is_eval_config_object(raw: str) -> bool:
    try:
        return isinstance(json.loads(raw or '{}'), dict)
    except json.JSONDecodeError:
        return False

try:
    cfg = load_eval_config()
except ValueError as e:
    print(f'[CONFIG] Falling back to empty config: {e}')
    cfg = {}

When it happens

Trigger: Setting EVAL_CONFIG to a JSON array (e.g. EVAL_CONFIG='[1,2,3]'), a bare string (e.g. EVAL_CONFIG='"hello"'), a number (e.g. EVAL_CONFIG='5'), or the literal string 'null'. json.loads succeeds but the isinstance(cfg, dict) check at cloud_runner.py:16 fails.

Common situations: A user wraps the issue list in brackets thinking EVAL_CONFIG='[28052,25693]' is the issues array directly, instead of EVAL_CONFIG='{"issues":[28052,25693]}'. A CI pipeline template injects EVAL_CONFIG from a variable that was serialized as a JSON list rather than an object. Copy-pasting a partial JSON fragment that omits the outer braces.

Related errors

• Invalid EVAL_CONFIG JSON: {e}
• Missing required environment variable '{name}'. Please ensur
• ${errorMessages.join('\n')}\nPlease fix the configuration fi
• Environment variable 'FIRESTORE_DOC' is required but was not
• Firestore document specification must be a JSON object.

AI-assisted analysis of google-gemini/gemini-cli@5024443c72 (2026-08-12). Data as JSON: /api/errors/93c289a584f4992d. Report an issue: GitHub.