pola-rs/polars · error

unable to parse buffer as utf-8

Error message

unable to parse buffer as utf-8

What it means

This panic happens in polars' file-like writer's `write` implementation when it must slice a Python string by character count but the buffer cannot be interpreted as UTF-8. The code uses `.expect("unable to parse buffer as utf-8")` after computing character offsets, so a non-UTF-8 or corrupted buffer triggers an unrecoverable panic instead of a Python exception.

Solutions

1. Ensure all data written is valid UTF-8 text when using text-mode wrappers.
2. Do not mix bytes and str writes to the same file-like object.
3. Validate/repair the input string's encoding before writing.
4. Upgrade polars if a version-specific UTF-8 handling bug is suspected.

Example fix

// before
f.write(b'\xff\xfe binary into text buffer')  # corrupts UTF-8 assumption
// after
f.write(b'\xff\xfe'.decode('utf-8', errors='replace'))  # keep buffer valid UTF-8

text.encode('utf-8')  # raises UnicodeEncodeError early if invalid
# and never mix: write only str to text buffers, only bytes to binary buffers

def is_valid_utf8(s):
    return isinstance(s, str)

try:
    f.write(text)
except BaseException as e:
    if 'utf-8' in str(e).lower():
        raise UnicodeError('buffer corrupted: non-UTF-8 content written to text writer') from e
    raise

When it happens

Trigger: Writing to a polars-wrapped file object where truncation by character count (n_chars) is requested but the underlying buffer content is not valid UTF-8, e.g. when bytes were written into a str-typed buffer or the buffer was mutated externally.

Common situations: Mixing binary and text writes to the same wrapper; passing a StringIO-like object whose internal content was replaced with non-UTF-8 bytes; locale/encoding issues where the source string was decoded incorrectly upstream.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Expecting to be able to downcast into bytes from read…
• activate object
• `columns` arg should only have unique values, got
• could not read
• could not read from input

AI-assisted analysis of pola-rs/polars@fe841f959e (2026-09-18). Data as JSON: /api/errors/788c718b713f8715. Report an issue: GitHub.

Appendix: source

Thrown at crates/polars-python/src/file.rs:240

                        "write",
                        (PyString::new(
                            py,
                            std::str::from_utf8(buf).map_err(io::Error::other)?,
                        ),),
                        None,
                    )
                    .map_err(pyerr_to_io_err)?;
                let n_chars: usize = number_chars_written.extract(py).map_err(pyerr_to_io_err)?;
                // calculate n_bytes
                if n_chars > 0 {
                    std::str::from_utf8(buf)
                        .map(|str| {
                            str.char_indices()
                                .nth(n_chars - 1)
                                .map(|(i, ch)| i + ch.len_utf8())
                                .unwrap()
                        })
                        .expect("unable to parse buffer as utf-8")
                } else {
                    0
                }
            } else {
                let number_bytes_written = self
                    .inner
                    .call_method(py, "write", (PyBytes::new(py, buf),), None)
                    .map_err(pyerr_to_io_err)?;
                number_bytes_written.extract(py).map_err(pyerr_to_io_err)?
            };
            Ok(n_bytes)
        })
    }

    fn flush(&mut self) -> Result<(), io::Error> {
        Self::flush(self)
    }
}

View on GitHub (pinned to fe841f959e)