pola-rs/polars · error

PyString is not valid UTF-8

Error message

PyString is not valid UTF-8

What it means

Internal expect() in PyFileLikeObject::to_buffer: the Python read() call returned a str, but converting it with to_cow() failed because the string's contents are not valid UTF-8, so it cannot become a byte buffer.

Source

Thrown at crates/polars-python/src/file.rs:94

    }

    pub(crate) fn to_buffer(&self) -> Buffer<u8> {
        Python::attach(|py| {
            let bytes = self
                .inner
                .call_method(py, "read", (), None)
                .expect("no read method found");

            if let Ok(b) = bytes.cast_bound::<PyBytes>(py) {
                // SAFETY: we keep the underlying python object alive.
                let slice = b.as_bytes();
                let owner = bytes.clone_ref(py);
                let ss = unsafe { SharedStorage::from_slice_with_owner(slice, owner) };
                return Buffer::from_storage(ss);
            }

            if let Ok(b) = bytes.cast_bound::<PyString>(py) {
                return match b.to_cow().expect("PyString is not valid UTF-8") {
                    Cow::Borrowed(v) => {
                        // SAFETY: we keep the underlying python object alive.
                        let slice = v.as_bytes();
                        let owner = bytes.clone_ref(py);
                        let ss = unsafe { SharedStorage::from_slice_with_owner(slice, owner) };
                        return Buffer::from_storage(ss);
                    },
                    Cow::Owned(v) => Buffer::from_vec(v.into_bytes()),
                };
            }

            panic!("Expecting to be able to downcast into bytes from read result.");
        })
    }

    /// Validates that the underlying
    /// python object has a `read`, `write`, and `seek` methods in respect to parameters.
    /// Will return a `TypeError` if object does not have `read`, `seek`, and `write` methods.

View on GitHub (pinned to 5d8ebabf11)

Solutions

1. The Python string is not valid UTF-8; decode/encode the data as UTF-8 before passing it.
2. Pass bytes instead of a str if the content is not text.

When it happens

Trigger: This panic/expect fires when execution reaches an unguarded state described by: "PyString is not valid UTF-8". Typical triggers: unsupported dtype or feature-gated code path reached at runtime, invalid user input or environment variable value, calling API methods in the wrong order or on mismatched types, or data (lengths, offsets, ranges) violating the function's preconditions.

Common situations: Encountered when input data or configuration does not meet the preconditions of the throwing code path ("PyString is not valid UTF-8"). Common cases: missing Cargo feature flags, mismatched dtypes/lengths between arrays or series, out-of-range temporal values, malformed environment variables, or operations on unsupported/complex nested types.

AI-assisted analysis of pola-rs/polars@5d8ebabf11 (2026-08-19). Data as JSON: /api/errors/a7e597899f538ede. Report an issue: GitHub.