tokio-rs/tokio · error · io::Error

stream did not contain valid UTF-8

Error message

stream did not contain valid UTF-8

What it means

Returned by finish_string_read when the I/O read succeeded but converting the accumulated bytes to a String failed (Ok(num_bytes), Err(utf8_err)). Before erroring, tokio restores the original buffer state via put_back_original_data, then wraps the failure as io::ErrorKind::InvalidData. It applies to both read_line and read_to_string through this shared helper.

Source

Thrown at tokio/src/io/util/read_line.rs:81

    match (io_res, utf8_res) {
        (Ok(num_bytes), Ok(string)) => {
            debug_assert_eq!(read, 0);
            *output = string;
            Poll::Ready(Ok(num_bytes))
        }
        (Err(io_err), Ok(string)) => {
            *output = string;
            if truncate_on_io_error {
                let original_len = output.len() - read;
                output.truncate(original_len);
            }
            Poll::Ready(Err(io_err))
        }
        (Ok(num_bytes), Err(utf8_err)) => {
            debug_assert_eq!(read, 0);
            put_back_original_data(output, utf8_err.into_bytes(), num_bytes);

            Poll::Ready(Err(io::Error::new(
                io::ErrorKind::InvalidData,
                "stream did not contain valid UTF-8",
            )))
        }
        (Err(io_err), Err(utf8_err)) => {
            put_back_original_data(output, utf8_err.into_bytes(), read);

            Poll::Ready(Err(io_err))
        }
    }
}

pub(super) fn read_line_internal<R: AsyncBufRead + ?Sized>(
    reader: Pin<&mut R>,
    cx: &mut Context<'_>,
    output: &mut String,
    buf: &mut Vec<u8>,
    read: &mut usize,

View on GitHub (pinned to 625954f365)

Solutions

1. Switch to read_until(b'\n', &mut Vec<u8>) and decode with String::from_utf8_lossy if lossy tolerance is acceptable.
2. Fix the source to emit UTF-8 (reconfigure the producer, transcode at ingestion, or specify the correct charset).
3. Validate with std::str::from_utf8 before constructing a String to surface the exact invalid byte offset.
4. For binary-safe protocols, use read (raw bytes) and parse explicitly instead of read_line.

Example fix

// before
let mut line = String::new();
reader.read_line(&mut line).await?; // InvalidData

// after
let mut bytes = Vec::new();
reader.read_until(b'\n', &mut bytes).await?;
let line = String::from_utf8_lossy(&bytes).into_owned();

// Validate bytes before constructing a String:
fn is_valid_utf8(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}

fn is_invalid_data(e: &io::Error) -> bool {
    e.kind() == io::ErrorKind::InvalidData
}

match reader.read_line(&mut line).await {
    Ok(_) => Ok(line),
    Err(e) if e.kind() == io::ErrorKind::InvalidData => {
        // decode lossily instead
        let lossy = String::from_utf8_lossy(&raw_bytes).into_owned();
        Ok(lossy)
    }
    Err(e) => Err(e.into()),
}

When it happens

Trigger: Calling AsyncBufReadExt::read_line / read_until_string or AsyncReadExt::read_to_string on a byte stream that contains invalid UTF-8 sequences. The branch fires only when no I/O error occurred — pure UTF-8 invalidity.

Common situations: Reading a text protocol (HTTP headers, log lines) over a mislabeled binary stream; Latin-1/CP1252 data mistaken for UTF-8; truncated multibyte sequence split across reads; corrupted or partially-overwritten files; mojibake from upstream encoding mismatches.

Related errors

• utf-8 error (io::Error::new(io::ErrorKind::InvalidData, err)
• stream did not contain valid UTF-8 (io::Error::new(io::Error
• Unable to decode input as UTF8
• early eof
• failed to write the buffered data

AI-assisted analysis of tokio-rs/tokio@625954f365 (2026-08-11). Data as JSON: /api/errors/712de4660f0e566b. Report an issue: GitHub.