{"record":{"id":"945e1e76bc814b45","repo":"tokio-rs/tokio","slug":"unable-to-decode-input-as-utf8","errorCode":null,"errorMessage":"Unable to decode input as UTF8","messagePattern":"Unable to decode input as UTF8","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"tokio-util/src/codec/lines_codec.rs","lineNumber":97,"sourceCode":"    /// use tokio_util::codec::LinesCodec;\n    ///\n    /// let codec = LinesCodec::new();\n    /// assert_eq!(codec.max_length(), usize::MAX);\n    /// ```\n    /// ```\n    /// use tokio_util::codec::LinesCodec;\n    ///\n    /// let codec = LinesCodec::new_with_max_length(256);\n    /// assert_eq!(codec.max_length(), 256);\n    /// ```\n    pub fn max_length(&self) -> usize {\n        self.max_length\n    }\n}\n\nfn utf8(buf: &[u8]) -> Result<&str, io::Error> {\n    str::from_utf8(buf)\n        .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, \"Unable to decode input as UTF8\"))\n}\n\nfn without_carriage_return(s: &[u8]) -> &[u8] {\n    if let Some(&b'\\r') = s.last() {\n        &s[..s.len() - 1]\n    } else {\n        s\n    }\n}\n\nimpl Decoder for LinesCodec {\n    type Item = String;\n    type Error = LinesCodecError;\n\n    fn decode(&mut self, buf: &mut BytesMut) -> Result<Option<String>, LinesCodecError> {\n        loop {\n            // Determine how far into the buffer we'll search for a newline. If\n            // there's no max_length set, we'll read to the end of the buffer.","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/tokio-rs/tokio/blob/625954f365727668cb02d04172b34f1149637728/tokio-util/src/codec/lines_codec.rs#L79-L115","documentation":"Runtime error from the `utf8` helper in `LinesCodec` (lines_codec.rs:97). After splitting the stream on `\\n`, the line bytes failed `str::from_utf8`, so the codec returns `io::ErrorKind::InvalidData`. `LinesCodec` requires each emitted line to be valid UTF-8.","triggerScenarios":"A line (between newlines) contains bytes that are not valid UTF-8: binary data fed into a line-oriented codec, a legacy single-byte encoding (latin-1, CP1252), or a corrupt/truncated multibyte sequence.","commonSituations":"Feeding a binary protocol into `LinesCodec`; mixed text encodings; mojibake from a mis-decoded upstream; a non-UTF-8 log file; partial multibyte char split across reads (rare, since `\\n` is ASCII).","solutions":["Ensure the source is actually UTF-8 text before using `LinesCodec`.","Switch to `BytesCodec` or `LengthDelimitedCodec` if the data is binary or non-UTF-8.","Re-encode/transcode the stream to UTF-8 upstream of the codec.","Handle `InvalidData` and decide to skip the line, log, or close the connection."],"exampleFix":"// before: binary/non-utf8 source into LinesCodec fails\nlet framed = FramedRead::new(stream, LinesCodec::new());\n\n// after: use BytesCodec for non-UTF-8 data\nlet framed = FramedRead::new(stream, BytesCodec::new());\n// or transcode the source to UTF-8 first if it is text in another encoding","handlingStrategy":"try-catch","validationCode":"// If you control the producer, validate UTF-8 before framing\nlet line = std::str::from_utf8(raw).map_err(|_| io::Error::new(io::ErrorKind::InvalidData, \"non-utf8\"))?\n    .to_owned();","typeGuard":"fn is_utf8_error(e: &io::Error) -> bool {\n    e.kind() == io::ErrorKind::InvalidData && e.to_string().contains(\"UTF8\")\n}","tryCatchPattern":"match framed.next().await {\n    Some(Err(e)) if is_utf8_error(&e) => { /* skip or close */ continue; }\n    Some(Err(e)) => return Err(e.into()),\n    Some(Ok(line)) => handle(line),\n    None => break,\n}","preventionTips":["Only use `LinesCodec` for known-UTF-8 text streams.","Pick `BytesCodec` or `LengthDelimitedCodec` for binary or non-UTF-8 data.","Transcode legacy encodings to UTF-8 upstream of the codec."],"tags":["rust","tokio","tokio-util","codec","lines","utf8","encoding","runtime"],"backgroundTag":null,"analyzedSha":"625954f365727668cb02d04172b34f1149637728","analyzedAt":"2026-08-11T17:46:45.378Z","contentChangedAt":"2026-08-11T17:46:45.378Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}