pnpm/pnpm · error
(UTF-8 validation error from String::from_utf8 on the YAML…
Error message
{error} (UTF-8 validation error from String::from_utf8 on the YAML document body) What it means
This error wraps the failure of `String::from_utf8` when converting the YAML document body of a lockfile from bytes to a Rust `String`. The lockfile reader reads the document in chunks and must produce valid UTF-8 text before handing it to the YAML parser; if any byte sequence between the document start marker (`---`) and the closing separator is not valid UTF-8, the read fails with `io::ErrorKind::InvalidData`. It exists to give a clear, contextual failure instead of propagating a raw `FromUtf8Error`.
Solutions
- Delete the corrupted `pnpm-lock.yaml` and regenerate it with `pnpm install` (pnpm will re-resolve from the manifest).
- Recover the lockfile from version control (`git checkout -- pnpm-lock.yaml`) if it was valid before.
- Check the file encoding (`file pnpm-lock.yaml`) and re-save it as UTF-8 without BOM if it was saved as UTF-16/Latin-1.
- If corruption recurs across writes, rule out disk/filesystem issues or processes concurrently writing the lockfile.
- Look for binary junk (BOM mid-file, NUL bytes) with `xxd pnpm-lock.yaml | head` and remove it.
Example fix
// before: lockfile saved as UTF-16, from_utf8 fails String::from_utf8(content).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error)) // after: re-encode the lockfile as UTF-8 (or regenerate it) $ pnpm-lock.yaml is corrupt: rm pnpm-lock.yaml && pnpm install
Defensive patterns
Strategy: validation
Validate before calling
// Before handing the lockfile to the reader, verify it is valid UTF-8:
const bytes = await fs.promises.readFile('pnpm-lock.yaml');
try {
new TextDecoder('utf-8', { fatal: true }).decode(bytes);
} catch {
// corrupt / wrong encoding: regenerate the lockfile
await fs.promises.rm('pnpm-lock.yaml');
}
Type guard
fn is_valid_utf8(bytes: &[u8]) -> bool {
std::str::from_utf8(bytes).is_ok()
}
Try / catch
// Rust: match on the io error kind and fall back to regenerating the lockfile
match read_first_yaml_document_in_chunks(&mut reader) {
Ok(doc) => parse(doc),
Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
// treat as missing lockfile: re-resolve and rewrite it
}
Err(e) => return Err(e),
}
Prevention
- Never hand-edit pnpm-lock.yaml in editors that may save as UTF-16 or Latin-1.
- Ensure editors/git filters leave the lockfile byte-for-byte unchanged (no encoding hooks).
- Treat sudden InvalidData errors as lockfile corruption: delete and regenerate with pnpm install.
- Monitor disks for truncation issues if corruption recurs.
When it happens
Trigger: Calling `read_first_yaml_document_in_chunks` on a `pnpm-lock.yaml` whose first document contains invalid UTF-8 bytes between `---` and the next document separator — e.g. a truncated multi-byte character from a partially written or corrupted file, a byte-order mark or stray high bytes embedded mid-document, or a file written in a non-UTF-8 encoding like Latin-1 or UTF-16.
Common situations: A lockfile corrupted by a crashed editor or interrupted write; a lockfile checked into git with CRLF plus non-UTF-8 bytes; a user hand-editing `pnpm-lock.yaml` in an editor that saved it as UTF-16 or with a BOM plus stray bytes; a disk/truncation issue leaving a partial multi-byte sequence at the end of the document body.
Understand the failure class
Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.
Related errors
- invalid utf-8 sequence of
- BROKEN_LOCKFILE
- Ignoring broken lockfile at
- INVALID_TARBALL_RESOLUTION
- LoadLockfileError
AI-assisted analysis of pnpm/pnpm@49dbe22c32 (2026-09-15).
Data as JSON: /api/errors/b7e5afd43a71c0cd.
Report an issue: GitHub.
Appendix: source
Thrown at pnpm/crates/lockfile/src/yaml_documents.rs:130
}
if let Some(separator) = find_document_separator(&content, scan_from) {
return first_document(content, separator).map(Some);
}
// A separator may straddle the chunk boundary, so resume the
// scan far enough back to catch a partial match.
scan_from = content
.len()
.saturating_sub(YAML_DOCUMENT_SEPARATOR.len() - 1)
.max(YAML_DOCUMENT_START.len());
}
}
/// The document body between the start marker and the separator that closes
/// it.
fn first_document(mut content: Vec<u8>, separator: usize) -> io::Result<String> {
content.truncate(separator);
content.drain(..YAML_DOCUMENT_START.len());
String::from_utf8(content).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))
}
/// Strip a leading byte-order mark once enough bytes have arrived to tell
/// whether there is one. Reports whether the caller can go on reading this
/// chunk, or has to pull more bytes first.
fn take_byte_order_mark(content: &mut Vec<u8>, pending: &mut bool) -> bool {
if !*pending {
return true;
}
if content.len() < BYTE_ORDER_MARK.len() {
return false;
}
*pending = false;
if content.starts_with(BYTE_ORDER_MARK) {
content.drain(..BYTE_ORDER_MARK.len());
}
true
}View on GitHub (pinned to 49dbe22c32)