LLM evaluator expects all input lists to have the same lengt

Error message

LLM evaluator expects all input lists to have the same length but received {inputs} with lengths {[len(_input) for _input in inputs]}.

What it means

All input lists of the LLM evaluator must have the same length so items can be paired per example. If lengths differ it raises ValueError showing the received values and their lengths.

Source

Thrown at haystack/components/evaluators/llm_evaluator.py:467

                raise ValueError(msg)

        # Validate that all received inputs are lists
        if not all(isinstance(_input, list) for _input in received.values()):
            msg = (
                "LLM evaluator expects all input values to be lists but received "
                f"{[type(_input) for _input in received.values()]}."
            )
            raise ValueError(msg)

        # Validate that all received inputs are of the same length
        inputs = received.values()
        length = len(next(iter(inputs)))
        if not all(len(_input) == length for _input in inputs):
            msg = (
                f"LLM evaluator expects all input lists to have the same length but received {inputs} with lengths "
                f"{[len(_input) for _input in inputs]}."
            )
            raise ValueError(msg)

View on GitHub (pinned to e318778c9b)

Solutions

Make all input lists the same length before calling run(), pairing each question with its response
If some answers are missing, pad with placeholders (e.g. "") to keep alignment rather than dropping from one list
Log/assert len() equality on all inputs before evaluation

Example fix

// before
evaluator.run({"questions": questions[:5], "responses": responses})
// after
evaluator.run({"questions": questions[:5], "responses": responses[:5]})

Defensive patterns

Strategy: validation

Validate before calling

lengths = {k: len(v) for k, v in inputs.items()}
if len(set(lengths.values())) > 1:
    raise ValueError(f"Input lengths differ: {lengths}")

Try / catch

try:
    result = evaluator.run(inputs)
except ValueError as e:
    logger.error("Unequal input lengths: %s", e)
    raise

Prevention

Filter paired lists together using zip so indices stay aligned
Assert len() equality on all evaluator inputs before evaluation
Avoid independent slicing/filtering of questions and answers

When it happens

Trigger: Calling run()/run_async() with e.g. 10 questions but 8 responses; inputs = {"questions": [...10], "responses": [...8]} causes the length-equality check in validate_input_parameters to fail.

Common situations: Filtering or dropping failed answers from one list but not the other before evaluation; an upstream generator producing fewer responses than prompts; off-by-one slicing of test data.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/317405a741a15438. Report an issue: GitHub.