{"record":{"id":"788c718b713f8715","repo":"pola-rs/polars","slug":"unable-to-parse-buffer-as-utf-8","errorCode":null,"errorMessage":"unable to parse buffer as utf-8","messagePattern":"unable to parse buffer as utf-8","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/polars-python/src/file.rs","lineNumber":240,"sourceCode":"                        \"write\",\n                        (PyString::new(\n                            py,\n                            std::str::from_utf8(buf).map_err(io::Error::other)?,\n                        ),),\n                        None,\n                    )\n                    .map_err(pyerr_to_io_err)?;\n                let n_chars: usize = number_chars_written.extract(py).map_err(pyerr_to_io_err)?;\n                // calculate n_bytes\n                if n_chars > 0 {\n                    std::str::from_utf8(buf)\n                        .map(|str| {\n                            str.char_indices()\n                                .nth(n_chars - 1)\n                                .map(|(i, ch)| i + ch.len_utf8())\n                                .unwrap()\n                        })\n                        .expect(\"unable to parse buffer as utf-8\")\n                } else {\n                    0\n                }\n            } else {\n                let number_bytes_written = self\n                    .inner\n                    .call_method(py, \"write\", (PyBytes::new(py, buf),), None)\n                    .map_err(pyerr_to_io_err)?;\n                number_bytes_written.extract(py).map_err(pyerr_to_io_err)?\n            };\n            Ok(n_bytes)\n        })\n    }\n\n    fn flush(&mut self) -> Result<(), io::Error> {\n        Self::flush(self)\n    }\n}","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/pola-rs/polars/blob/fe841f959ef4d2ceefc05a310d33ed7b1ab24e5e/crates/polars-python/src/file.rs#L222-L258","documentation":"This panic happens in polars' file-like writer's `write` implementation when it must slice a Python string by character count but the buffer cannot be interpreted as UTF-8. The code uses `.expect(\"unable to parse buffer as utf-8\")` after computing character offsets, so a non-UTF-8 or corrupted buffer triggers an unrecoverable panic instead of a Python exception.","triggerScenarios":"Writing to a polars-wrapped file object where truncation by character count (n_chars) is requested but the underlying buffer content is not valid UTF-8, e.g. when bytes were written into a str-typed buffer or the buffer was mutated externally.","commonSituations":"Mixing binary and text writes to the same wrapper; passing a StringIO-like object whose internal content was replaced with non-UTF-8 bytes; locale/encoding issues where the source string was decoded incorrectly upstream.","solutions":["Ensure all data written is valid UTF-8 text when using text-mode wrappers.","Do not mix bytes and str writes to the same file-like object.","Validate/repair the input string's encoding before writing.","Upgrade polars if a version-specific UTF-8 handling bug is suspected."],"exampleFix":"// before\nf.write(b'\\xff\\xfe binary into text buffer')  # corrupts UTF-8 assumption\n// after\nf.write(b'\\xff\\xfe'.decode('utf-8', errors='replace'))  # keep buffer valid UTF-8","handlingStrategy":"validation","validationCode":"text.encode('utf-8')  # raises UnicodeEncodeError early if invalid\n# and never mix: write only str to text buffers, only bytes to binary buffers","typeGuard":"def is_valid_utf8(s):\n    return isinstance(s, str)","tryCatchPattern":"try:\n    f.write(text)\nexcept BaseException as e:\n    if 'utf-8' in str(e).lower():\n        raise UnicodeError('buffer corrupted: non-UTF-8 content written to text writer') from e\n    raise","preventionTips":["Never write bytes into a text-mode wrapper.","Decode/repair input with errors='replace' before writing.","Keep text and binary writers separate."],"tags":["python","io","utf-8","panic","encoding"],"backgroundTag":"invalid-argument-format","analyzedSha":"fe841f959ef4d2ceefc05a310d33ed7b1ab24e5e","analyzedAt":"2026-09-18T22:14:11.667Z","contentChangedAt":"2026-09-18T22:14:11.667Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}