tokio-rs/tokio · error · io::Error

Unable to decode input as UTF8

Error message

Unable to decode input as UTF8

What it means

Runtime error from the `utf8` helper in `LinesCodec` (lines_codec.rs:97). After splitting the stream on `\n`, the line bytes failed `str::from_utf8`, so the codec returns `io::ErrorKind::InvalidData`. `LinesCodec` requires each emitted line to be valid UTF-8.

Source

Thrown at tokio-util/src/codec/lines_codec.rs:97

    /// use tokio_util::codec::LinesCodec;
    ///
    /// let codec = LinesCodec::new();
    /// assert_eq!(codec.max_length(), usize::MAX);
    /// ```
    /// ```
    /// use tokio_util::codec::LinesCodec;
    ///
    /// let codec = LinesCodec::new_with_max_length(256);
    /// assert_eq!(codec.max_length(), 256);
    /// ```
    pub fn max_length(&self) -> usize {
        self.max_length
    }
}

fn utf8(buf: &[u8]) -> Result<&str, io::Error> {
    str::from_utf8(buf)
        .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "Unable to decode input as UTF8"))
}

fn without_carriage_return(s: &[u8]) -> &[u8] {
    if let Some(&b'\r') = s.last() {
        &s[..s.len() - 1]
    } else {
        s
    }
}

impl Decoder for LinesCodec {
    type Item = String;
    type Error = LinesCodecError;

    fn decode(&mut self, buf: &mut BytesMut) -> Result<Option<String>, LinesCodecError> {
        loop {
            // Determine how far into the buffer we'll search for a newline. If
            // there's no max_length set, we'll read to the end of the buffer.

View on GitHub (pinned to 625954f365)

Solutions

1. Ensure the source is actually UTF-8 text before using `LinesCodec`.
2. Switch to `BytesCodec` or `LengthDelimitedCodec` if the data is binary or non-UTF-8.
3. Re-encode/transcode the stream to UTF-8 upstream of the codec.
4. Handle `InvalidData` and decide to skip the line, log, or close the connection.

Example fix

// before: binary/non-utf8 source into LinesCodec fails
let framed = FramedRead::new(stream, LinesCodec::new());

// after: use BytesCodec for non-UTF-8 data
let framed = FramedRead::new(stream, BytesCodec::new());
// or transcode the source to UTF-8 first if it is text in another encoding

// If you control the producer, validate UTF-8 before framing
let line = std::str::from_utf8(raw).map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "non-utf8"))?
    .to_owned();

fn is_utf8_error(e: &io::Error) -> bool {
    e.kind() == io::ErrorKind::InvalidData && e.to_string().contains("UTF8")
}

match framed.next().await {
    Some(Err(e)) if is_utf8_error(&e) => { /* skip or close */ continue; }
    Some(Err(e)) => return Err(e.into()),
    Some(Ok(line)) => handle(line),
    None => break,
}

When it happens

Trigger: A line (between newlines) contains bytes that are not valid UTF-8: binary data fed into a line-oriented codec, a legacy single-byte encoding (latin-1, CP1252), or a corrupt/truncated multibyte sequence.

Common situations: Feeding a binary protocol into `LinesCodec`; mixed text encodings; mojibake from a mis-decoded upstream; a non-UTF-8 log file; partial multibyte char split across reads (rare, since `\n` is ASCII).

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• bytes remaining on stream
• failed to write frame to transport
• frame size too big
• provided length would overflow after adjustment
• failed to write entire datagram to socket

AI-assisted analysis of tokio-rs/tokio@625954f365 (2026-08-11). Data as JSON: /api/errors/945e1e76bc814b45. Report an issue: GitHub.