run-llama/llama_index · error · ValueError

Query id {query_id} not in responses

Error message

Query id {query_id} not in responses

What it means

Raised by the qr_pairs property of the QueryResponseDataset-style class when a query id present in self.queries has no matching key in self.responses. The property iterates queries and requires a one-to-one id mapping before zipping them into (query, response) tuples.

Source

Thrown at llama-index-core/llama_index/core/evaluation/dataset_generation.py:88

    @classmethod
    def from_qr_pairs(
        cls,
        qr_pairs: List[Tuple[str, str]],
    ) -> QueryResponseDataset:
        """Create from qr pairs."""
        # define ids as simple integers
        queries = {str(idx): query for idx, (query, _) in enumerate(qr_pairs)}
        responses = {str(idx): response for idx, (_, response) in enumerate(qr_pairs)}
        return cls(queries=queries, responses=responses)

    @property
    def qr_pairs(self) -> List[Tuple[str, str]]:
        """Get pairs."""
        # if query_id not in response, throw error
        for query_id in self.queries:
            if query_id not in self.responses:
                raise ValueError(f"Query id {query_id} not in responses")

        return [
            (self.queries[query_id], self.responses[query_id])
            for query_id in self.queries
        ]

    @property
    def questions(self) -> List[str]:
        """Get questions."""
        return list(self.queries.values())

    def save_json(self, path: str) -> None:
        """Save json."""
        with open(path, "w", encoding="utf-8") as f:
            json.dump(self.model_dump(), f, indent=4)

    @classmethod
    def from_json(cls, path: str) -> QueryResponseDataset:

View on GitHub (pinned to afd0fef371)

Solutions

1. Reconstruct the dataset from aligned pairs via QueryResponseDataset.from_qr_pairs(zip(queries, responses)) so ids are generated consistently.
2. Normalize keys to strings and drop unmatched ids before construction: ids = queries.keys() & responses.keys().
3. If you add a query later, add its response under the same id at the same time.

Example fix

# before
ds = cls(queries={"0": q0, "1": q1}, responses={"0": r0})
pairs = ds.qr_pairs  # raises: id "1" missing from responses

# after
common = set(queries) & set(responses)
ds = cls(
    queries={k: queries[k] for k in common},
    responses={k: responses[k] for k in common},
)
pairs = ds.qr_pairs

missing = set(ds.queries) - set(ds.responses)
if missing:
    raise ValueError(f"responses missing for ids: {sorted(missing)}")

def ids_aligned(ds) -> bool:
    return set(ds.queries.keys()) <= set(ds.responses.keys())

When it happens

Trigger: Constructing the object with queries and responses dicts whose keys diverge — e.g. queries={"0": ..., "1": ...} but responses={"0": ...}, or ids generated by different schemes (integers vs strings, or separately enumerated). Then accessing .qr_pairs.

Common situations: Building the dataset manually from two separately collected dicts (questions from one file, answers from another with dropped items); mutating queries after construction (adding a query without its response); type mismatches like int keys vs the str(idx) keys produced by from_qr_pairs.

Related errors

• query and response must be provided
• The response is invalid
• Both query and contexts must be provided
• The response is invalid
• query, and response must be provided

AI-assisted analysis of run-llama/llama_index@afd0fef371 (2026-08-15). Data as JSON: /api/errors/b7e2dffbac6eb3e8. Report an issue: GitHub.