{"record":{"id":"25b98259b566e4a0","repo":"tokio-rs/tokio","slug":"utf-8-error-io-error-new-io-errorkind-invalid","errorCode":null,"errorMessage":"utf-8 error (io::Error::new(io::ErrorKind::InvalidData, err))","messagePattern":"utf-8 error \\(io::Error::new\\(io::ErrorKind::InvalidData, err\\)\\)","errorType":"exception","errorClass":"io::Error (InvalidData)","httpStatus":null,"severity":"error","filePath":"tokio/src/io/util/lines.rs","lineNumber":139,"sourceCode":"        let n = ready!(read_until_internal(me.reader, cx, b'\\n', me.buf, &mut read))?;\n\n        if n == 0 && me.buf.is_empty() {\n            return Poll::Ready(Ok(None));\n        }\n\n        let mut bytes = mem::take(me.buf);\n\n        if bytes.last() == Some(&b'\\n') {\n            bytes.pop();\n\n            if bytes.last() == Some(&b'\\r') {\n                bytes.pop();\n            }\n        }\n\n        match String::from_utf8(bytes) {\n            Ok(line) => Poll::Ready(Ok(Some(line))),\n            Err(err) => Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, err))),\n        }\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n\n    #[test]\n    fn assert_unpin() {\n        crate::is_unpin::<Lines<()>>();\n    }\n}\n","sourceCodeStart":121,"sourceCodeEnd":153,"githubUrl":"https://github.com/tokio-rs/tokio/blob/7d0d729d8f03a0033d6752730d0fb5928962560e/tokio/src/io/util/lines.rs#L121-L153","documentation":"tokio's Lines codec (AsyncBufReadExt::lines / Lines::next_line) splits the stream on newline bytes and requires each line to be valid UTF-8. When String::from_utf8 fails on the accumulated line bytes, it discards them and returns this io::Error with ErrorKind::InvalidData, wrapping the underlying Utf8Error as the source. The library throws it because Rust's String type cannot hold arbitrary bytes, so non-UTF-8 input is unrecoverable at this API level.","triggerScenarios":"Calling next_line() (or polling Lines::poll_next_line) on a stream whose current line contains bytes that are not valid UTF-8 — e.g. reading binary data, a file in Latin-1/UTF-16/GBK encoding, compressed (gzip) bytes, or a split multi-byte character truncated at a read boundary that later resolves invalid. The error surfaces on the exact poll where the invalid byte sequence is completed.","commonSituations":"Piping a subprocess that emits binary or non-UTF-8 locale output (Windows cp1252), reading log files written in a legacy encoding, pointing the reader at a socket or file that is not line-oriented text (images, databases, gzipped data), or a peer sending corrupted/partially-transcoded bytes.","solutions":["Read raw bytes instead of lines: use AsyncBufReadExt::read_until(b'\\n', &mut buf) and decode each buffer yourself with String::from_utf8_lossy or an encoding_rs decoder for the actual source encoding.","Fix the data source to emit UTF-8: set the subprocess/producer's locale or encoding (e.g. LANG=C.UTF-8) or transcode the file.","If bytes only arrive split across reads but are valid overall, buffer with read_until across the whole record rather than relying on line decoding.","Detect and skip binary input up front (e.g. sniff for a NUL byte) so Lines is only used on verified text streams."],"exampleFix":"// before: panics-less but errors on non-UTF-8 lines\nlet mut lines = reader.lines();\nwhile let Some(line) = lines.next_line().await? { /* ... */ }\n\n// after: tolerate arbitrary bytes per line\nuse tokio::io::AsyncBufReadExt;\nlet mut buf = Vec::new();\nloop {\n    buf.clear();\n    let n = reader.read_until(b'\\n', &mut buf).await?;\n    if n == 0 { break; }\n    let line = String::from_utf8_lossy(&buf).into_owned();\n    // or: encoding_rs::WINDOWS_1252.decode(&buf) for legacy encodings\n}","handlingStrategy":"fallback","validationCode":"// Peek/probe before using Lines: ensure the stream is text and decode manually per line\nasync fn next_line_lossy<R: tokio::io::AsyncBufRead + Unpin>(r: &mut R) -> std::io::Result<Option<String>> {\n    let mut buf = Vec::new();\n    let n = r.read_until(b'\\n', &mut buf).await?;\n    Ok((n > 0).then(|| String::from_utf8_lossy(&buf).trim_end_matches(['\\n','\\r']).into_owned()))\n}","typeGuard":"fn is_utf8_bytes(b: &[u8]) -> bool { std::str::from_utf8(b).is_ok() }","tryCatchPattern":"match res.next_line().await {\n    Ok(Some(line)) => handle(line),\n    Ok(None) => break,\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {\n        // recover: switch to byte reads / from_utf8_lossy for the rest of the stream\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Never point Lines/read-line APIs at binary, compressed, or legacy-encoded sources.","Set the producer's encoding/locale to UTF-8 (LANG=C.UTF-8) for subprocesses.","Use read_until + String::from_utf8_lossy when the encoding is uncertain.","Sniff the first bytes of a stream (NUL byte / BOM / compression magic) before choosing a text reader."],"tags":["io","utf-8","tokio","async","encoding","invalid-data"],"backgroundTag":"invalid-utf8-data","analyzedSha":"7d0d729d8f03a0033d6752730d0fb5928962560e","analyzedAt":"2026-09-06T15:37:27.972Z","contentChangedAt":"2026-09-06T15:37:27.972Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}