mlflow/mlflow · error · ValueError

Test cases at indices {indices_with_invalid_context} must ha

Error message

Test cases at indices {indices_with_invalid_context} must have 'context' as a dict when provided.

What it means

The optional 'context' field of a test case, when provided and not a missing-value sentinel (None, NaN, etc.), must be a dict mapping variable names to values used for prompt templating. _validate_test_cases raises this ValueError listing indices whose context is a non-dict value (e.g. a string or list).

Source

Thrown at mlflow/genai/simulators/simulator.py:544

        if not test_cases:
            raise ValueError("test_cases cannot be empty")

        missing_goal_indices = [
            i for i, test_case in enumerate(test_cases) if not test_case.get("goal")
        ]
        if missing_goal_indices:
            raise ValueError(f"Test cases at indices {missing_goal_indices} must have 'goal' field")

        indices_with_invalid_context = [
            i
            for i, test_case in enumerate(test_cases)
            if not (
                isinstance(test_case.get("context"), dict)
                or _is_missing_context_value(test_case.get("context"))
            )
        ]
        if indices_with_invalid_context:
            raise ValueError(
                f"Test cases at indices {indices_with_invalid_context} must have 'context' as "
                "a dict when provided."
            )

        indices_with_reserved_context_keys = [
            i
            for i, test_case in enumerate(test_cases)
            if isinstance(test_case.get("context"), dict)
            and set(test_case["context"]) & _RESERVED_CONTEXT_KEYS
        ]
        if indices_with_reserved_context_keys:
            raise ValueError(
                f"Test cases at indices {indices_with_reserved_context_keys} have context keys "
                f"that conflict with keys reserved by ConversationSimulator "
                f"({_RESERVED_CONTEXT_KEYS}). These keys are used to inject conversation "
                "history ('input', 'messages') or session ID ('mlflow_session_id'). "
                "Rename the conflicting keys in the test case context."
            )

View on GitHub (pinned to 6a27f2decc)

Solutions

Wrap the context value in a dict, e.g. context={'product': 'MLflow tracing'}
Remove the context key entirely if not needed (None/missing is allowed)
Normalize string contexts to {'input': value} or another named variable

Example fix

// before
{'goal': 'Ask about limits', 'context': 'free tier user'}
// after
{'goal': 'Ask about limits', 'context': {'plan': 'free tier'}}

Defensive patterns

Strategy: type-guard

Validate before calling

bad = [i for i, c in enumerate(test_cases)
       if 'context' in c and c['context'] is not None and not isinstance(c['context'], dict)]
if bad:
    raise ValueError(f'Cases {bad}: context must be a dict')

Type guard

def has_valid_context(case: dict) -> bool:
    ctx = case.get('context')
    return ctx is None or isinstance(ctx, dict)

Try / catch

try:
    sim = ConversationSimulator(test_cases=cases)
except ValueError as e:
    if "must have 'context' as a dict" in str(e):
        for c in cases:
            if c.get('context') is not None and not isinstance(c['context'], dict):
                c['context'] = {'value': c['context']}
    else:
        raise

Prevention

Always model context as dict[str, Any] in your data generation
Parse YAML/JSON context strings into dicts before building test cases
Add a schema check (pydantic/dataclass) for test case records

When it happens

Trigger: Setting ConversationSimulator.test_cases where some case has context='product X' or context=['a','b'] instead of a dict — via ConversationSimulator(test_cases=...) or sim.test_cases = ....

Common situations: Copying context from YAML/JSON configs where it is a flat string; assuming context is free-form like a chat history list; pandas records conversion turning dicts into NaN mixes.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/af408334eddf0545. Report an issue: GitHub.