{"record":{"id":"712de4660f0e566b","repo":"tokio-rs/tokio","slug":"stream-did-not-contain-valid-utf-8","errorCode":null,"errorMessage":"stream did not contain valid UTF-8","messagePattern":"stream did not contain valid UTF-8","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"tokio/src/io/util/read_line.rs","lineNumber":81,"sourceCode":"    match (io_res, utf8_res) {\n        (Ok(num_bytes), Ok(string)) => {\n            debug_assert_eq!(read, 0);\n            *output = string;\n            Poll::Ready(Ok(num_bytes))\n        }\n        (Err(io_err), Ok(string)) => {\n            *output = string;\n            if truncate_on_io_error {\n                let original_len = output.len() - read;\n                output.truncate(original_len);\n            }\n            Poll::Ready(Err(io_err))\n        }\n        (Ok(num_bytes), Err(utf8_err)) => {\n            debug_assert_eq!(read, 0);\n            put_back_original_data(output, utf8_err.into_bytes(), num_bytes);\n\n            Poll::Ready(Err(io::Error::new(\n                io::ErrorKind::InvalidData,\n                \"stream did not contain valid UTF-8\",\n            )))\n        }\n        (Err(io_err), Err(utf8_err)) => {\n            put_back_original_data(output, utf8_err.into_bytes(), read);\n\n            Poll::Ready(Err(io_err))\n        }\n    }\n}\n\npub(super) fn read_line_internal<R: AsyncBufRead + ?Sized>(\n    reader: Pin<&mut R>,\n    cx: &mut Context<'_>,\n    output: &mut String,\n    buf: &mut Vec<u8>,\n    read: &mut usize,","sourceCodeStart":63,"sourceCodeEnd":99,"githubUrl":"https://github.com/tokio-rs/tokio/blob/625954f365727668cb02d04172b34f1149637728/tokio/src/io/util/read_line.rs#L63-L99","documentation":"Returned by finish_string_read when the I/O read succeeded but converting the accumulated bytes to a String failed (Ok(num_bytes), Err(utf8_err)). Before erroring, tokio restores the original buffer state via put_back_original_data, then wraps the failure as io::ErrorKind::InvalidData. It applies to both read_line and read_to_string through this shared helper.","triggerScenarios":"Calling AsyncBufReadExt::read_line / read_until_string or AsyncReadExt::read_to_string on a byte stream that contains invalid UTF-8 sequences. The branch fires only when no I/O error occurred — pure UTF-8 invalidity.","commonSituations":"Reading a text protocol (HTTP headers, log lines) over a mislabeled binary stream; Latin-1/CP1252 data mistaken for UTF-8; truncated multibyte sequence split across reads; corrupted or partially-overwritten files; mojibake from upstream encoding mismatches.","solutions":["Switch to read_until(b'\\n', &mut Vec<u8>) and decode with String::from_utf8_lossy if lossy tolerance is acceptable.","Fix the source to emit UTF-8 (reconfigure the producer, transcode at ingestion, or specify the correct charset).","Validate with std::str::from_utf8 before constructing a String to surface the exact invalid byte offset.","For binary-safe protocols, use read (raw bytes) and parse explicitly instead of read_line."],"exampleFix":"// before\nlet mut line = String::new();\nreader.read_line(&mut line).await?; // InvalidData\n\n// after\nlet mut bytes = Vec::new();\nreader.read_until(b'\\n', &mut bytes).await?;\nlet line = String::from_utf8_lossy(&bytes).into_owned();","handlingStrategy":"validation","validationCode":"// Validate bytes before constructing a String:\nfn is_valid_utf8(bytes: &[u8]) -> bool {\n    std::str::from_utf8(bytes).is_ok()\n}","typeGuard":"fn is_invalid_data(e: &io::Error) -> bool {\n    e.kind() == io::ErrorKind::InvalidData\n}","tryCatchPattern":"match reader.read_line(&mut line).await {\n    Ok(_) => Ok(line),\n    Err(e) if e.kind() == io::ErrorKind::InvalidData => {\n        // decode lossily instead\n        let lossy = String::from_utf8_lossy(&raw_bytes).into_owned();\n        Ok(lossy)\n    }\n    Err(e) => Err(e.into()),\n}","preventionTips":["Use read_until + from_utf8_lossy for untrusted byte streams.","Confirm the upstream producer's charset and transcode at ingestion if it's not UTF-8.","Add an integration test with a known invalid byte to lock in error handling.","Avoid read_to_string on sockets whose encoding you don't control."],"tags":["io","utf-8","read-line","encoding","tokio"],"backgroundTag":null,"analyzedSha":"625954f365727668cb02d04172b34f1149637728","analyzedAt":"2026-08-11T17:46:45.378Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}