atuinsh/atuin · error · std::io::Error

invalid utf-8 sequence of

Error message

invalid utf-8 sequence of {} bytes from index {}

What it means

`lines_utf8` streams byte lines from the tailed file and converts each to `String::from_utf8`. When a line contains bytes that are not valid UTF-8, the FromUtf8Error is wrapped in an io::Error with kind InvalidData carrying the message "invalid utf-8 sequence of N bytes from index I". The stream yields that error for the offending line instead of a string.

Solutions

1. Use `Positioned` byte-line streaming (the `lines` API) instead of `lines_utf8`, and decode with `String::from_utf8_lossy` to tolerate invalid bytes.
2. Ensure the file's encoding is UTF-8; reconfigure the producing tool (e.g. set LANG/LC_ALL to a UTF-8 locale).
3. If reading a still-growing file, wait for the writer to complete the line so multibyte sequences are not split.
4. Sanitize or transcode the file content before tailing (e.g. `iconv -f latin1 -t utf-8 log > log.utf8`).

Example fix

// before
let mut lines = source.lines_utf8();
while let Some(line) = lines.next().await {
    let line = line?; // fails on non-UTF-8 bytes
}

// after
let mut lines = source.lines(); // byte lines
while let Some(line) = lines.next().await {
    let bytes = line?;
    let text = String::from_utf8_lossy(&bytes); // lossy: U+FFFD replaces bad bytes
}

// quick pre-check on a prefix of the file
let mut buf = [0u8; 4096];
let n = file.read(&mut buf)?;
if std::str::from_utf8(&buf[..n]).is_err() {
    eprintln!("file does not look like UTF-8; use lossy decoding");
}

fn valid_utf8_line(bytes: &[u8]) -> Option<&str> {
    std::str::from_utf8(bytes).ok()
}

while let Some(line) = lines_utf8_stream.next().await {
    match line {
        Ok(text) => process(&text),
        Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
            eprintln!("skipping non-UTF-8 line: {e}");
        }
        Err(e) => return Err(e.into()),
    }
}

When it happens

Trigger: Tailing a file that contains binary data, a log with mixed encodings (e.g. Latin-1/GBK content), or a line truncated mid-multibyte-UTF-8-character at the file's end or at a partial read boundary.

Common situations: Tailing logs written by non-UTF-8-locale tools, logs containing raw binary blobs/hex dumps, or tailing a file while it is being written and a multibyte character is split across flushes.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed printing to stdout
• tail target is not a regular file

AI-assisted analysis of atuinsh/atuin@900007c480 (2026-09-19). Data as JSON: /api/errors/35594b592ce05b5c. Report an issue: GitHub.

Appendix: source

Thrown at crates/atuin-common/src/fs/tail.rs:646

                }
            }
        }
    }

    /// Stream the file as newline-delimited byte lines, with the trailing `\n` stripped.
    pub fn lines(self) -> impl Stream<Item = io::Result<Bytes>> + Send {
        line_bytes_stream(self.max_line_len, self.events())
    }

    /// Stream the file as UTF-8 line strings; invalid UTF-8 yields an error.
    pub fn lines_utf8(self) -> impl Stream<Item = io::Result<String>> + Send {
        let lines = self.lines();
        async_stream::stream! {
            futures::pin_mut!(lines);
            while let Some(line) = lines.next().await {
                yield line.and_then(|bytes| {
                    String::from_utf8(bytes.to_vec())
                        .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))
                });
            }
        }
    }

    /// Stream byte lines, each paired with its [`Positioned`] location for resumable follows.
    pub fn lines_positioned(self) -> impl Stream<Item = Positioned<io::Result<Bytes>>> + Send {
        let max = self.max_line_len;
        let events = self.events();
        async_stream::stream! {
            let mut acc = LineAccumulator::new(max);
            let mut epoch: u64 = 0;
            let mut epoch_start: Option<u64> = None;
            futures::pin_mut!(events);
            while let Some(event) = events.next().await {
                match event {
                    Event::Data { bytes, offset } => {
                        if epoch_start.is_none() {

View on GitHub (pinned to 900007c480)