mlflow/mlflow · error · ValueError

Test cases at indices {missing_goal_indices} must have 'goal

Error message

Test cases at indices {missing_goal_indices} must have 'goal' field

What it means

Every test case must include a truthy 'goal' field describing what the simulated user wants to accomplish. _validate_test_cases collects the indices of cases where test_case.get('goal') is falsy (missing, None, or empty string) and raises this ValueError listing them.

Source

Thrown at mlflow/genai/simulators/simulator.py:533

                    "EvaluationDataset passed to ConversationSimulator must contain "
                    "conversational test cases with a 'goal' field in the 'inputs' column"
                )
            return records

        if isinstance(test_cases, DataFrame):
            return test_cases.to_dict("records")

        return test_cases

    def _validate_test_cases(self, test_cases: list[dict[str, Any]]) -> None:
        if not test_cases:
            raise ValueError("test_cases cannot be empty")

        missing_goal_indices = [
            i for i, test_case in enumerate(test_cases) if not test_case.get("goal")
        ]
        if missing_goal_indices:
            raise ValueError(f"Test cases at indices {missing_goal_indices} must have 'goal' field")

        indices_with_invalid_context = [
            i
            for i, test_case in enumerate(test_cases)
            if not (
                isinstance(test_case.get("context"), dict)
                or _is_missing_context_value(test_case.get("context"))
            )
        ]
        if indices_with_invalid_context:
            raise ValueError(
                f"Test cases at indices {indices_with_invalid_context} must have 'context' as "
                "a dict when provided."
            )

        indices_with_reserved_context_keys = [
            i
            for i, test_case in enumerate(test_cases)

View on GitHub (pinned to 6a27f2decc)

Solutions

Add a non-empty 'goal' string to every test case dict at the reported indices
Fix key typos so the field is exactly 'goal'
Filter or repair records with falsy goals before constructing the simulator

Example fix

// before
cases = [{'goal': 'Ask about refunds'}, {'persona': 'angry customer'}]
// after
cases = [{'goal': 'Ask about refunds', 'persona': 'angry customer'}]

Defensive patterns

Strategy: validation

Validate before calling

bad = [i for i, c in enumerate(test_cases) if not (isinstance(c, dict) and c.get('goal'))]
if bad:
    raise ValueError(f'Cases {bad} need a non-empty goal')

Type guard

def has_goal(case: dict) -> bool:
    return isinstance(case.get('goal'), str) and case['goal'].strip() != ''

Try / catch

try:
    sim = ConversationSimulator(test_cases=cases)
except ValueError as e:
    if "must have 'goal' field" in str(e):
        import re
        bad = re.findall(r'\[([^\]]+)\]', str(e))
        for i in ast.literal_eval(bad[0]):
            cases[i]['goal'] = default_goal
    else:
        raise

Prevention

Validate every case has a non-empty 'goal' before assignment
Avoid typos: use a constant GOAL = 'goal' as the key
LLM-distilled goals should be checked for None/empty before use

When it happens

Trigger: Setting ConversationSimulator.test_cases to a list where one or more dicts lack 'goal' or have goal=None/'' — e.g. [{'goal': 'x'}, {'persona': 'admin'}] fails at index 1.

Common situations: Merging test case sets from different sources where some lack 'goal'; typos like 'goals' or 'objective'; records built by LLM distillation where goal extraction failed and returned None.

Understand the failure class

Background: "Missing required field" and "field is required" errors: why libraries reject payloads that omit mandatory fields — this error's family across 20 libraries.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/aade68d8a3b32a6e. Report an issue: GitHub.