tokio-rs/tokio · error · io::Error (InvalidData)

stream did not contain valid UTF-8 (io::Error::new(io::Error

Error message

stream did not contain valid UTF-8 (io::Error::new(io::ErrorKind::InvalidData, utf8_error))

What it means

AsyncBufReadExt::read_line and AsyncBufReadExt::read_to_string read the stream while decoding it as UTF-8 in place; if the UTF-8 check fails, finish_string_read puts the original bytes back into the buffer and returns io::Error::new(ErrorKind::InvalidData, utf8_error). tokio throws this because the API's output type is String, which can only represent valid UTF-8, so byte streams that violate UTF-8 are rejected as invalid data.

Source

Thrown at tokio/src/io/util/read_line.rs:82

        (Ok(num_bytes), Ok(string)) => {
            debug_assert_eq!(read, 0);
            *output = string;
            Poll::Ready(Ok(num_bytes))
        }
        (Err(io_err), Ok(string)) => {
            *output = string;
            if truncate_on_io_error {
                let original_len = output.len() - read;
                output.truncate(original_len);
            }
            Poll::Ready(Err(io_err))
        }
        (Ok(num_bytes), Err(utf8_err)) => {
            debug_assert_eq!(read, 0);
            let utf8_error = utf8_err.utf8_error();
            put_back_original_data(output, utf8_err.into_bytes(), num_bytes);

            Poll::Ready(Err(io::Error::new(io::ErrorKind::InvalidData, utf8_error)))
        }
        (Err(io_err), Err(utf8_err)) => {
            put_back_original_data(output, utf8_err.into_bytes(), read);

            Poll::Ready(Err(io_err))
        }
    }
}

fn read_line_internal<R: AsyncBufRead + ?Sized>(
    reader: Pin<&mut R>,
    cx: &mut Context<'_>,
    output: &mut String,
    buf: &mut Vec<u8>,
    read: &mut usize,
) -> Poll<io::Result<usize>> {
    let io_res = ready!(read_until_internal(reader, cx, b'\n', buf, read));
    let utf8_res = String::from_utf8(mem::take(buf));

View on GitHub (pinned to 7d0d729d8f)

Solutions

1. Switch to byte-based reads: AsyncReadExt::read_to_end into a Vec<u8> (or read_line's byte counterpart read_until), then decode with String::from_utf8_lossy or the actual source encoding via encoding_rs.
2. If the content is valid UTF-8 only as a whole but the error persists, verify the stream is not compressed; decompress (async-compression) before reading to a String.
3. Retranscode/fix the source data to UTF-8 (e.g. iconv the file, set the producer's encoding/locale).
4. Validate the encoding before committing to read_to_string: peek the first bytes and reject non-text content early.

Example fix

// before
let mut s = String::new();
reader.read_to_string(&mut s).await?; // InvalidData: stream did not contain valid UTF-8

// after
use tokio::io::AsyncReadExt;
let mut bytes = Vec::new();
reader.read_to_end(&mut bytes).await?;
let s = String::from_utf8_lossy(&bytes).into_owned();
// or strict: let s = String::from_utf8(bytes)?;

// Read to bytes first and validate before committing to String APIs
let mut bytes = Vec::new();
reader.read_to_end(&mut bytes).await?;
let text_ok = std::str::from_utf8(&bytes).is_ok();

fn as_utf8(b: &[u8]) -> Option<&str> { std::str::from_utf8(b).ok() }

match reader.read_to_string(&mut out).await {
    Ok(_) => use_text(&out),
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
        // fallback: read_to_end + String::from_utf8_lossy / encoding_rs decode
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling read_line(&mut String) or read_to_string(&mut String) (via read_line_internal / read_to_string_internal) on an AsyncBufRead source that delivers bytes invalid as UTF-8: binary files, non-UTF-8 encoded text (Latin-1, UTF-16, Shift-JIS), gzip/zstd compressed data, or corrupted network payloads. Notably, if the underlying read itself also fails the io error takes precedence, so this variant appears when the only problem is the encoding.

Common situations: Reading a config or log file saved in a legacy Windows or ISO-8859 encoding, ingesting a downloaded payload that is compressed or binary, terminal/pipe output from a process with a non-UTF-8 locale, or consumers migrating code that previously used byte-oriented reads to the String-based convenience APIs.

Related errors

• utf-8 error (io::Error::new(io::ErrorKind::InvalidData, err)
• stream did not contain valid UTF-8
• Unable to decode input as UTF8
• early eof
• failed to write the buffered data

AI-assisted analysis of tokio-rs/tokio@7d0d729d8f (2026-09-06). Data as JSON: /api/errors/0e159d23d139237b. Report an issue: GitHub.