Unsupported document_comparison_field: '{self.document_compa

Error message

Unsupported document_comparison_field: '{self.document_comparison_field}'. Use 'content', 'id', or 'meta.<key>'.

What it means

DocumentNDCGEvaluator._get_comparison_value resolves document comparison values and raises ValueError when document_comparison_field is not 'content', 'id', or 'meta.<key>'. It is reached from both _build_relevance_map (ground truth docs) and calculate_dcg (retrieved docs) during run().

Source

Thrown at haystack/components/evaluators/document_ndcg.py:71

        Extract the comparison value from a document based on the configured field.
        """
        if self.document_comparison_field == "content":
            return doc.content
        if self.document_comparison_field == "id":
            return doc.id
        if self.document_comparison_field.startswith("meta."):
            parts = self.document_comparison_field[5:].split(".")
            value = doc.meta
            for part in parts:
                if not isinstance(value, dict) or part not in value:
                    return None
                value = value[part]
            return value
        msg = (
            f"Unsupported document_comparison_field: '{self.document_comparison_field}'. "
            "Use 'content', 'id', or 'meta.<key>'."
        )
        raise ValueError(msg)

    def _build_relevance_map(self, gt_docs: list[Document]) -> dict[Any, float]:
        """
        Map each ground truth comparison value to its relevance score.

        Documents whose comparison value cannot be determined (e.g. missing meta key) are skipped,
        since they can never be matched during retrieval either. Documents that share a comparison
        value are collapsed to a single entry keeping the highest relevance, so `calculate_dcg` and
        `calculate_idcg` credit the same unique relevant set with the same scores.
        """
        relevant_value_to_score: dict[Any, float] = {}
        for doc in gt_docs:
            value = self._get_comparison_value(doc)
            if value is None:
                continue
            relevance = doc.score if doc.score is not None else 1.0
            relevant_value_to_score[value] = max(relevant_value_to_score.get(value, relevance), relevance)
        return relevant_value_to_score

View on GitHub (pinned to e318778c9b)

Solutions

Set document_comparison_field to 'content', 'id', or 'meta.<key>' exactly
Prefix meta keys: `document_comparison_field="meta.score_key"`
Validate the option in __init__ or at pipeline-load time to fail fast

Example fix

// before
DocumentNDCGEvaluator(document_comparison_field="content ")  # trailing space
// after
DocumentNDCGEvaluator(document_comparison_field="content")

Defensive patterns

Strategy: validation

Validate before calling

field = evaluator.document_comparison_field
assert field in ("content", "id") or field.startswith("meta.")

Type guard

def is_valid_comparison_field(field: object) -> bool:
    return isinstance(field, str) and (field in ("content", "id") or field.startswith("meta."))

Try / catch

try:
    result = evaluator.run(ground_truth_documents=gt, retrieved_documents=ret)
except ValueError as e:
    if "Unsupported document_comparison_field" in str(e):
        evaluator.document_comparison_field = "content"
        result = evaluator.run(ground_truth_documents=gt, retrieved_documents=ret)
    else:
        raise

Prevention

Validate the field specifier before pipelines run
For meta fields write 'meta.<key>' including nested paths like 'meta.source.url'
Check YAML for case errors ('Content' is invalid)
Test from_dict on your serialized pipelines

When it happens

Trigger: Constructing DocumentNDCGEvaluator with an unsupported field like 'text', 'title' (without 'meta.'), or an empty string; first run() call then fails while building the relevance map.

Common situations: YAML config typos or case mistakes ('Content'); using a bare meta key without the 'meta.' prefix; copying a config from another evaluator with different option values.

Related errors

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/5bc207edf630c88d. Report an issue: GitHub.