{"record":{"id":"c51d4c115ed95d37","repo":"run-llama/llama_index","slug":"all-inputs-must-have-the-same-length","errorCode":null,"errorMessage":"All inputs must have the same length.","messagePattern":"All inputs must have the same length\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"llama-index-core/llama_index/core/evaluation/batch_runner.py","lineNumber":140,"sourceCode":"\n        \"\"\"\n        assert len(inputs_list) > 0\n        # first, make sure at least one of queries or response_strs is not None\n        input_len: Optional[int] = None\n        for inputs in inputs_list:\n            if inputs is not None:\n                input_len = len(inputs)\n                break\n        if input_len is None:\n            raise ValueError(\"At least one item in inputs_list must be provided.\")\n\n        new_inputs_list = []\n        for inputs in inputs_list:\n            if inputs is None:\n                new_inputs_list.append([None] * input_len)\n            else:\n                if len(inputs) != input_len:\n                    raise ValueError(\"All inputs must have the same length.\")\n                new_inputs_list.append(inputs)\n        return new_inputs_list\n\n    def _validate_nested_eval_kwargs_types(\n        self, eval_kwargs_lists: Dict[str, Any]\n    ) -> Dict[str, Any]:\n        \"\"\"\n        Ensure eval kwargs are acceptable format.\n            either a Dict[str, List] or a Dict[str, Dict[str, List]].\n\n        Allows use of different kwargs (e.g. references) with different evaluators\n            while keeping backwards compatibility for single evaluators\n\n        \"\"\"\n        if not isinstance(eval_kwargs_lists, dict):\n            raise ValueError(\n                f\"eval_kwargs_lists must be a dict. Got {eval_kwargs_lists}\"\n            )","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/run-llama/llama_index/blob/afd0fef371831f9bda13e5af7167cf4e981278ab/llama-index-core/llama_index/core/evaluation/batch_runner.py#L122-L158","documentation":"Raised by BatchRunner._validate_and_clean_inputs when two or more provided kwargs lists have different lengths. After the first non-None list fixes input_len, every other non-None list must match that length, because kwargs are zipped index-by-index with the queries/responses.","triggerScenarios":"Calling aevaluate_responses (directly or via aevaluate) with e.g. len(queries)=10 but len(responses)=8, or a references list whose length differs from the queries list.","commonSituations":"Building queries and references from different sources (queries from a dataset, references hand-written); filtering one list (dropping failed generations) without filtering the other; off-by-one when slicing lists.","solutions":["Log len() of every list you pass and make them equal before calling the runner.","Align lists at construction time: build (query, response, reference) tuples first, then unzip, so lengths cannot diverge.","If some items lack a reference, pad with [None] * len(queries) at the right positions instead of shortening the list."],"exampleFix":"# before\nawait runner.aevaluate_responses(\n    queries=queries,               # len 10\n    responses=responses,           # len 8  -> raises\n)\n\n# after\nn = min(len(queries), len(responses))\nawait runner.aevaluate_responses(\n    queries=queries[:n], responses=responses[:n]\n)","handlingStrategy":"validation","validationCode":"def assert_aligned(queries, responses, **kwargs_lists):\n    n = len(queries)\n    for name, lst in {\"responses\": responses, **kwargs_lists}.items():\n        if lst is not None and len(lst) != n:\n            raise ValueError(f\"{name} has {len(lst)} items, expected {n}\")\n    return True","typeGuard":"def same_len(*lists) -> bool:\n    lens = {len(x) for x in lists if x is not None}\n    return len(lens) <= 1","tryCatchPattern":null,"preventionTips":["Build (query, response, reference) tuples first and unzip, so lengths cannot diverge.","Log len() of every list right before calling aevaluate_responses during development."],"tags":["batch-evaluation","input-validation","length-mismatch"],"backgroundTag":null,"analyzedSha":"afd0fef371831f9bda13e5af7167cf4e981278ab","analyzedAt":"2026-08-15T05:42:58.429Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}