deepset-ai/haystack · error · ValueError

header_split_levels must not contain duplicate values.

Error message

header_split_levels must not contain duplicate values.

What it means

MarkdownHeaderSplitter's __init__ rejects duplicate values in header_split_levels. Duplicates have no semantic effect and would cause redundant checks during splitting, so a ValueError is raised.

Source

Thrown at haystack/components/preprocessors/markdown_header_splitter.py:69

        :param split_overlap: The number of overlapping units for each split when using secondary splitting.
            Defaults to 0.
        :param split_threshold: The minimum number of units per split when using secondary splitting. Defaults to 0.
        :param skip_empty_documents: Choose whether to skip documents with empty content. Default is True.
            Set to False when downstream components in the Pipeline (like LLMDocumentContentExtractor) can extract text
            from non-textual documents.
        """
        if header_split_levels is None:
            header_split_levels = [1, 2, 3, 4, 5, 6]

        if not isinstance(header_split_levels, list) or len(header_split_levels) == 0:
            raise ValueError("header_split_levels must be a non-empty list.")
        invalid = [lvl for lvl in header_split_levels if not isinstance(lvl, int) or lvl < 1 or lvl > 6]
        if invalid:
            raise ValueError(
                f"header_split_levels contains invalid values: {invalid}. All levels must be integers between 1 and 6."
            )
        if len(header_split_levels) != len(set(header_split_levels)):
            raise ValueError("header_split_levels must not contain duplicate values.")

        self.page_break_character = page_break_character
        self.secondary_split = secondary_split
        self.split_length = split_length
        self.split_overlap = split_overlap
        self.split_threshold = split_threshold
        self.skip_empty_documents = skip_empty_documents
        self.keep_headers = keep_headers
        self.header_split_levels = header_split_levels
        self._header_split_levels_set = set(header_split_levels)
        self._header_pattern = re.compile(r"(?m)^(#{1,6}) (.+)$")  # ATX-style .md-headers

        # Matches fenced code blocks delimited by triple backticks (```) or triple tildes (~~~).
        # Broken down:
        #   ^                 - fence must start at the beginning of a line (MULTILINE)
        #   (?P<fence>`{3,}|~{3,})
        #                     - named capture group "fence": three or more backticks OR three or
        #                       more tildes. Capturing it allows the closing fence to be matched

View on GitHub (pinned to e318778c9b)

Solutions

1. Deduplicate the list before construction: header_split_levels=sorted(set(levels)).
2. Keep only unique values while preserving order: list(dict.fromkeys(levels)).
3. Remove the duplicated literal from a hand-written list.

Example fix

// before
MarkdownHeaderSplitter(header_split_levels=[1, 2, 2, 3])
// after
MarkdownHeaderSplitter(header_split_levels=sorted(set([1, 2, 2, 3])))  # [1, 2, 3]

levels = list(dict.fromkeys(raw_levels))  # dedupe, preserve order
assert len(levels) == len(set(levels))

def has_no_duplicates(levels) -> bool:
    return isinstance(levels, list) and len(levels) == len(set(levels))

try:
    splitter = MarkdownHeaderSplitter(header_split_levels=levels)
except ValueError as e:
    levels = sorted(set(levels))
    splitter = MarkdownHeaderSplitter(header_split_levels=levels)

When it happens

Trigger: Calling MarkdownHeaderSplitter(header_split_levels=[1, 1, 2]) or building the list by concatenation/range where levels repeat (e.g. list(range(1,4))*2).

Common situations: Programmatically assembling levels from multiple config sources and merging without deduplication, or copy-pasting level lists in code.

Related errors

• 'response_fn' must return an assistant ChatMessage, got '{re
• Cannot stream multiple responses, please set n=1.
• A `ChatMessage` must contain at least one `TextContent`, `To
• top_k must be greater than 0.
• top_k must not be negative.

AI-assisted analysis of deepset-ai/haystack@e318778c9b (2026-08-30). Data as JSON: /api/errors/959b1d669478a276. Report an issue: GitHub.