tokio-rs/tokio · error · io::Error (InvalidData)

utf-8 error (io::Error::new(io::ErrorKind::InvalidData, err)

Error message

utf-8 error (io::Error::new(io::ErrorKind::InvalidData, err))

What it means

tokio's Lines codec (AsyncBufReadExt::lines / Lines::next_line) splits the stream on newline bytes and requires each line to be valid UTF-8. When String::from_utf8 fails on the accumulated line bytes, it discards them and returns this io::Error with ErrorKind::InvalidData, wrapping the underlying Utf8Error as the source. The library throws it because Rust's String type cannot hold arbitrary bytes, so non-UTF-8 input is unrecoverable at this API level.

Source

Thrown at tokio/src/io/util/lines.rs:139

        let n = ready!(read_until_internal(me.reader, cx, b'\n', me.buf, &mut read))?;

        if n == 0 && me.buf.is_empty() {
            return Poll::Ready(Ok(None));
        }

        let mut bytes = mem::take(me.buf);

        if bytes.last() == Some(&b'\n') {
            bytes.pop();

            if bytes.last() == Some(&b'\r') {
                bytes.pop();
            }
        }

        match String::from_utf8(bytes) {
            Ok(line) => Poll::Ready(Ok(Some(line))),
            Err(err) => Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, err))),
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn assert_unpin() {
        crate::is_unpin::<Lines<()>>();
    }
}

View on GitHub (pinned to 7d0d729d8f)

Solutions

1. Read raw bytes instead of lines: use AsyncBufReadExt::read_until(b'\n', &mut buf) and decode each buffer yourself with String::from_utf8_lossy or an encoding_rs decoder for the actual source encoding.
2. Fix the data source to emit UTF-8: set the subprocess/producer's locale or encoding (e.g. LANG=C.UTF-8) or transcode the file.
3. If bytes only arrive split across reads but are valid overall, buffer with read_until across the whole record rather than relying on line decoding.
4. Detect and skip binary input up front (e.g. sniff for a NUL byte) so Lines is only used on verified text streams.

Example fix

// before: panics-less but errors on non-UTF-8 lines
let mut lines = reader.lines();
while let Some(line) = lines.next_line().await? { /* ... */ }

// after: tolerate arbitrary bytes per line
use tokio::io::AsyncBufReadExt;
let mut buf = Vec::new();
loop {
    buf.clear();
    let n = reader.read_until(b'\n', &mut buf).await?;
    if n == 0 { break; }
    let line = String::from_utf8_lossy(&buf).into_owned();
    // or: encoding_rs::WINDOWS_1252.decode(&buf) for legacy encodings
}

// Peek/probe before using Lines: ensure the stream is text and decode manually per line
async fn next_line_lossy<R: tokio::io::AsyncBufRead + Unpin>(r: &mut R) -> std::io::Result<Option<String>> {
    let mut buf = Vec::new();
    let n = r.read_until(b'\n', &mut buf).await?;
    Ok((n > 0).then(|| String::from_utf8_lossy(&buf).trim_end_matches(['\n','\r']).into_owned()))
}

fn is_utf8_bytes(b: &[u8]) -> bool { std::str::from_utf8(b).is_ok() }

match res.next_line().await {
    Ok(Some(line)) => handle(line),
    Ok(None) => break,
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
        // recover: switch to byte reads / from_utf8_lossy for the rest of the stream
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling next_line() (or polling Lines::poll_next_line) on a stream whose current line contains bytes that are not valid UTF-8 — e.g. reading binary data, a file in Latin-1/UTF-16/GBK encoding, compressed (gzip) bytes, or a split multi-byte character truncated at a read boundary that later resolves invalid. The error surfaces on the exact poll where the invalid byte sequence is completed.

Common situations: Piping a subprocess that emits binary or non-UTF-8 locale output (Windows cp1252), reading log files written in a legacy encoding, pointing the reader at a socket or file that is not line-oriented text (images, databases, gzipped data), or a peer sending corrupted/partially-transcoded bytes.

Related errors

• stream did not contain valid UTF-8 (io::Error::new(io::Error
• stream did not contain valid UTF-8
• Unable to decode input as UTF8
• early eof
• failed to write the buffered data

AI-assisted analysis of tokio-rs/tokio@7d0d729d8f (2026-09-06). Data as JSON: /api/errors/25b98259b566e4a0. Report an issue: GitHub.