pandas-dev/pandas · error · ValueError

invalid normalization form

Error message

invalid normalization form

What it means

Raised by pyarrow-backed Series.str.normalize(form) when form is not one of the four Unicode normalization forms NFC, NFD, NFKC, NFKD. These are the only forms accepted by unicodedata.normalize and pyarrow's utf8_normalize. pandas validates the form before dispatching to the kernel.

Source

Thrown at pandas/core/arrays/_arrow_string_mixins.py:182

                pa_pad = partial(pc.utf8_center, lean_left_on_odd_padding=lean_left)
        else:
            raise ValueError(
                f"Invalid side: {side}. Side must be one of 'left', 'right', 'both'"
            )
        return self._from_pyarrow_array(
            pa_pad(self._pa_array, width=width, padding=fillchar)
        )

    def _str_zfill(self, width: int) -> Self:
        if pa_version_under21p0:
            predicate = lambda val: val.zfill(width)
            result = self._apply_elementwise(predicate)
            return self._from_pyarrow_array(pa.chunked_array(result))
        return self._from_pyarrow_array(pc.utf8_zfill(self._pa_array, width))

    def _str_normalize(self, form: Literal["NFC", "NFD", "NFKC", "NFKD"]) -> Self:
        if form not in ("NFC", "NFD", "NFKC", "NFKD"):
            raise ValueError("invalid normalization form")
        if form in ("NFC", "NFKC"):
            # GH#64359 pc.utf8_normalize only decomposes; it skips the canonical
            #  composition step, so for the composing forms it returns decomposed
            #  output. Fall back to unicodedata for these.
            predicate = lambda val: unicodedata.normalize(form, val)
            result = self._apply_elementwise(predicate)
            return self._from_pyarrow_array(pa.chunked_array(result))
        return self._from_pyarrow_array(pc.utf8_normalize(self._pa_array, form=form))

    def _str_get(self, i: int) -> Self:
        lengths = pc.utf8_length(self._pa_array)
        if i >= 0:
            out_of_bounds = pc.greater_equal(i, lengths)
            start = i
            stop = i + 1
            step = 1
        else:
            out_of_bounds = pc.greater(-i, lengths)

View on GitHub (pinned to 71959b8cb9)

Solutions

1. Use exactly one of NFC, NFD, NFKC, NFKD (uppercase).
2. If form comes from user input, validate it against the 4-tuple before calling str.normalize.

Example fix

// before
s.str.normalize("nfc")
// after
s.str.normalize("NFC")

def safe_normalize(s, form="NFC"):
    if form not in ("NFC", "NFD", "NFKC", "NFKD"):
        raise ValueError(f"invalid normalization form {form!r}")
    return s.str.normalize(form)

When it happens

Trigger: Calling s.str.normalize('nfc') (lowercase), 'NFKA' (typo), or any non-standard string on a string[pyarrow] Series.

Common situations: Case-sensitivity mistakes, typos in the form name, or form values read from external config without validation.

Related errors

• Invalid side: {side}. Side must be one of 'left', 'right', '
• replace is not supported with a re.Pattern, callable repl, c
• contains not implemented with {flags=}
• {pat=} must contain a symbolic group name.
• Must specify a valid frequency: {freq}

AI-assisted analysis of pandas-dev/pandas@71959b8cb9 (2026-08-07). Data as JSON: /api/errors/7c47d6e2bcc405e8. Report an issue: GitHub.