{"record":{"id":"35594b592ce05b5c","repo":"atuinsh/atuin","slug":"invalid-utf-8-sequence-of-bytes-from-index","errorCode":null,"errorMessage":"invalid utf-8 sequence of {} bytes from index {}","messagePattern":"invalid utf-8 sequence of (.+?) bytes from index (.+?)","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"crates/atuin-common/src/fs/tail.rs","lineNumber":646,"sourceCode":"                }\n            }\n        }\n    }\n\n    /// Stream the file as newline-delimited byte lines, with the trailing `\\n` stripped.\n    pub fn lines(self) -> impl Stream<Item = io::Result<Bytes>> + Send {\n        line_bytes_stream(self.max_line_len, self.events())\n    }\n\n    /// Stream the file as UTF-8 line strings; invalid UTF-8 yields an error.\n    pub fn lines_utf8(self) -> impl Stream<Item = io::Result<String>> + Send {\n        let lines = self.lines();\n        async_stream::stream! {\n            futures::pin_mut!(lines);\n            while let Some(line) = lines.next().await {\n                yield line.and_then(|bytes| {\n                    String::from_utf8(bytes.to_vec())\n                        .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))\n                });\n            }\n        }\n    }\n\n    /// Stream byte lines, each paired with its [`Positioned`] location for resumable follows.\n    pub fn lines_positioned(self) -> impl Stream<Item = Positioned<io::Result<Bytes>>> + Send {\n        let max = self.max_line_len;\n        let events = self.events();\n        async_stream::stream! {\n            let mut acc = LineAccumulator::new(max);\n            let mut epoch: u64 = 0;\n            let mut epoch_start: Option<u64> = None;\n            futures::pin_mut!(events);\n            while let Some(event) = events.next().await {\n                match event {\n                    Event::Data { bytes, offset } => {\n                        if epoch_start.is_none() {","sourceCodeStart":628,"sourceCodeEnd":664,"githubUrl":"https://github.com/atuinsh/atuin/blob/900007c48045867e7f44b0f9bc55ad9dbe99ee0d/crates/atuin-common/src/fs/tail.rs#L628-L664","documentation":"`lines_utf8` streams byte lines from the tailed file and converts each to `String::from_utf8`. When a line contains bytes that are not valid UTF-8, the FromUtf8Error is wrapped in an io::Error with kind InvalidData carrying the message \"invalid utf-8 sequence of N bytes from index I\". The stream yields that error for the offending line instead of a string.","triggerScenarios":"Tailing a file that contains binary data, a log with mixed encodings (e.g. Latin-1/GBK content), or a line truncated mid-multibyte-UTF-8-character at the file's end or at a partial read boundary.","commonSituations":"Tailing logs written by non-UTF-8-locale tools, logs containing raw binary blobs/hex dumps, or tailing a file while it is being written and a multibyte character is split across flushes.","solutions":["Use `Positioned` byte-line streaming (the `lines` API) instead of `lines_utf8`, and decode with `String::from_utf8_lossy` to tolerate invalid bytes.","Ensure the file's encoding is UTF-8; reconfigure the producing tool (e.g. set LANG/LC_ALL to a UTF-8 locale).","If reading a still-growing file, wait for the writer to complete the line so multibyte sequences are not split.","Sanitize or transcode the file content before tailing (e.g. `iconv -f latin1 -t utf-8 log > log.utf8`)."],"exampleFix":"// before\nlet mut lines = source.lines_utf8();\nwhile let Some(line) = lines.next().await {\n    let line = line?; // fails on non-UTF-8 bytes\n}\n\n// after\nlet mut lines = source.lines(); // byte lines\nwhile let Some(line) = lines.next().await {\n    let bytes = line?;\n    let text = String::from_utf8_lossy(&bytes); // lossy: U+FFFD replaces bad bytes\n}","handlingStrategy":"fallback","validationCode":"// quick pre-check on a prefix of the file\nlet mut buf = [0u8; 4096];\nlet n = file.read(&mut buf)?;\nif std::str::from_utf8(&buf[..n]).is_err() {\n    eprintln!(\"file does not look like UTF-8; use lossy decoding\");\n}","typeGuard":"fn valid_utf8_line(bytes: &[u8]) -> Option<&str> {\n    std::str::from_utf8(bytes).ok()\n}","tryCatchPattern":"while let Some(line) = lines_utf8_stream.next().await {\n    match line {\n        Ok(text) => process(&text),\n        Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {\n            eprintln!(\"skipping non-UTF-8 line: {e}\");\n        }\n        Err(e) => return Err(e.into()),\n    }\n}","preventionTips":["Prefer byte-line APIs plus String::from_utf8_lossy over strict UTF-8 line streams for untrusted logs.","Ensure log producers run with UTF-8 locales (LC_ALL=C.UTF-8).","For files still being written, wait for line completion to avoid split multibyte sequences."],"tags":["encoding","utf-8","io","streaming"],"backgroundTag":"invalid-argument-format","analyzedSha":"900007c48045867e7f44b0f9bc55ad9dbe99ee0d","analyzedAt":"2026-09-19T12:52:51.491Z","contentChangedAt":"2026-09-19T12:52:51.491Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}