{"record":{"id":"b7e5afd43a71c0cd","repo":"pnpm/pnpm","slug":"error-utf-8-validation-error-from-string-from-utf8-on-the","errorCode":null,"errorMessage":"{error} (UTF-8 validation error from String::from_utf8 on the YAML document body)","messagePattern":"(.+?) \\(UTF-8 validation error from String::from_utf8 on the YAML document body\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pnpm/crates/lockfile/src/yaml_documents.rs","lineNumber":130,"sourceCode":"        }\n        if let Some(separator) = find_document_separator(&content, scan_from) {\n            return first_document(content, separator).map(Some);\n        }\n        // A separator may straddle the chunk boundary, so resume the\n        // scan far enough back to catch a partial match.\n        scan_from = content\n            .len()\n            .saturating_sub(YAML_DOCUMENT_SEPARATOR.len() - 1)\n            .max(YAML_DOCUMENT_START.len());\n    }\n}\n\n/// The document body between the start marker and the separator that closes\n/// it.\nfn first_document(mut content: Vec<u8>, separator: usize) -> io::Result<String> {\n    content.truncate(separator);\n    content.drain(..YAML_DOCUMENT_START.len());\n    String::from_utf8(content).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))\n}\n\n/// Strip a leading byte-order mark once enough bytes have arrived to tell\n/// whether there is one. Reports whether the caller can go on reading this\n/// chunk, or has to pull more bytes first.\nfn take_byte_order_mark(content: &mut Vec<u8>, pending: &mut bool) -> bool {\n    if !*pending {\n        return true;\n    }\n    if content.len() < BYTE_ORDER_MARK.len() {\n        return false;\n    }\n    *pending = false;\n    if content.starts_with(BYTE_ORDER_MARK) {\n        content.drain(..BYTE_ORDER_MARK.len());\n    }\n    true\n}","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/pnpm/pnpm/blob/49dbe22c32011a09be9e7a4315c380a0415a944f/pnpm/crates/lockfile/src/yaml_documents.rs#L112-L148","documentation":"This error wraps the failure of `String::from_utf8` when converting the YAML document body of a lockfile from bytes to a Rust `String`. The lockfile reader reads the document in chunks and must produce valid UTF-8 text before handing it to the YAML parser; if any byte sequence between the document start marker (`---`) and the closing separator is not valid UTF-8, the read fails with `io::ErrorKind::InvalidData`. It exists to give a clear, contextual failure instead of propagating a raw `FromUtf8Error`.","triggerScenarios":"Calling `read_first_yaml_document_in_chunks` on a `pnpm-lock.yaml` whose first document contains invalid UTF-8 bytes between `---` and the next document separator — e.g. a truncated multi-byte character from a partially written or corrupted file, a byte-order mark or stray high bytes embedded mid-document, or a file written in a non-UTF-8 encoding like Latin-1 or UTF-16.","commonSituations":"A lockfile corrupted by a crashed editor or interrupted write; a lockfile checked into git with CRLF plus non-UTF-8 bytes; a user hand-editing `pnpm-lock.yaml` in an editor that saved it as UTF-16 or with a BOM plus stray bytes; a disk/truncation issue leaving a partial multi-byte sequence at the end of the document body.","solutions":["Delete the corrupted `pnpm-lock.yaml` and regenerate it with `pnpm install` (pnpm will re-resolve from the manifest).","Recover the lockfile from version control (`git checkout -- pnpm-lock.yaml`) if it was valid before.","Check the file encoding (`file pnpm-lock.yaml`) and re-save it as UTF-8 without BOM if it was saved as UTF-16/Latin-1.","If corruption recurs across writes, rule out disk/filesystem issues or processes concurrently writing the lockfile.","Look for binary junk (BOM mid-file, NUL bytes) with `xxd pnpm-lock.yaml | head` and remove it."],"exampleFix":"// before: lockfile saved as UTF-16, from_utf8 fails\nString::from_utf8(content).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))\n\n// after: re-encode the lockfile as UTF-8 (or regenerate it)\n$ pnpm-lock.yaml is corrupt: rm pnpm-lock.yaml && pnpm install","handlingStrategy":"validation","validationCode":"// Before handing the lockfile to the reader, verify it is valid UTF-8:\nconst bytes = await fs.promises.readFile('pnpm-lock.yaml');\ntry {\n  new TextDecoder('utf-8', { fatal: true }).decode(bytes);\n} catch {\n  // corrupt / wrong encoding: regenerate the lockfile\n  await fs.promises.rm('pnpm-lock.yaml');\n}\n","typeGuard":"fn is_valid_utf8(bytes: &[u8]) -> bool {\n    std::str::from_utf8(bytes).is_ok()\n}\n","tryCatchPattern":"// Rust: match on the io error kind and fall back to regenerating the lockfile\nmatch read_first_yaml_document_in_chunks(&mut reader) {\n    Ok(doc) => parse(doc),\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {\n        // treat as missing lockfile: re-resolve and rewrite it\n    }\n    Err(e) => return Err(e),\n}\n","preventionTips":["Never hand-edit pnpm-lock.yaml in editors that may save as UTF-16 or Latin-1.","Ensure editors/git filters leave the lockfile byte-for-byte unchanged (no encoding hooks).","Treat sudden InvalidData errors as lockfile corruption: delete and regenerate with pnpm install.","Monitor disks for truncation issues if corruption recurs."],"tags":["lockfile","yaml","utf-8","encoding"],"backgroundTag":"file-read-failed","analyzedSha":"49dbe22c32011a09be9e7a4315c380a0415a944f","analyzedAt":"2026-09-15T20:33:00.202Z","contentChangedAt":"2026-09-15T20:33:00.202Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}