pnpm/pnpm · error

(UTF-8 validation error from String::from_utf8 on the YAML…

Error message

{error} (UTF-8 validation error from String::from_utf8 on the YAML document body)

What it means

This error wraps the failure of `String::from_utf8` when converting the YAML document body of a lockfile from bytes to a Rust `String`. The lockfile reader reads the document in chunks and must produce valid UTF-8 text before handing it to the YAML parser; if any byte sequence between the document start marker (`---`) and the closing separator is not valid UTF-8, the read fails with `io::ErrorKind::InvalidData`. It exists to give a clear, contextual failure instead of propagating a raw `FromUtf8Error`.

Solutions

  1. Delete the corrupted `pnpm-lock.yaml` and regenerate it with `pnpm install` (pnpm will re-resolve from the manifest).
  2. Recover the lockfile from version control (`git checkout -- pnpm-lock.yaml`) if it was valid before.
  3. Check the file encoding (`file pnpm-lock.yaml`) and re-save it as UTF-8 without BOM if it was saved as UTF-16/Latin-1.
  4. If corruption recurs across writes, rule out disk/filesystem issues or processes concurrently writing the lockfile.
  5. Look for binary junk (BOM mid-file, NUL bytes) with `xxd pnpm-lock.yaml | head` and remove it.

Example fix

// before: lockfile saved as UTF-16, from_utf8 fails
String::from_utf8(content).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))

// after: re-encode the lockfile as UTF-8 (or regenerate it)
$ pnpm-lock.yaml is corrupt: rm pnpm-lock.yaml && pnpm install
Defensive patterns

Strategy: validation

Validate before calling

// Before handing the lockfile to the reader, verify it is valid UTF-8:
const bytes = await fs.promises.readFile('pnpm-lock.yaml');
try {
  new TextDecoder('utf-8', { fatal: true }).decode(bytes);
} catch {
  // corrupt / wrong encoding: regenerate the lockfile
  await fs.promises.rm('pnpm-lock.yaml');
}

Type guard

fn is_valid_utf8(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}

Try / catch

// Rust: match on the io error kind and fall back to regenerating the lockfile
match read_first_yaml_document_in_chunks(&mut reader) {
    Ok(doc) => parse(doc),
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
        // treat as missing lockfile: re-resolve and rewrite it
    }
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: Calling `read_first_yaml_document_in_chunks` on a `pnpm-lock.yaml` whose first document contains invalid UTF-8 bytes between `---` and the next document separator — e.g. a truncated multi-byte character from a partially written or corrupted file, a byte-order mark or stray high bytes embedded mid-document, or a file written in a non-UTF-8 encoding like Latin-1 or UTF-16.

Common situations: A lockfile corrupted by a crashed editor or interrupted write; a lockfile checked into git with CRLF plus non-UTF-8 bytes; a user hand-editing `pnpm-lock.yaml` in an editor that saved it as UTF-16 or with a BOM plus stray bytes; a disk/truncation issue leaving a partial multi-byte sequence at the end of the document body.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors


AI-assisted analysis of pnpm/pnpm@49dbe22c32 (2026-09-15). Data as JSON: /api/errors/b7e5afd43a71c0cd. Report an issue: GitHub.

Appendix: source

Thrown at pnpm/crates/lockfile/src/yaml_documents.rs:130

        }
        if let Some(separator) = find_document_separator(&content, scan_from) {
            return first_document(content, separator).map(Some);
        }
        // A separator may straddle the chunk boundary, so resume the
        // scan far enough back to catch a partial match.
        scan_from = content
            .len()
            .saturating_sub(YAML_DOCUMENT_SEPARATOR.len() - 1)
            .max(YAML_DOCUMENT_START.len());
    }
}

/// The document body between the start marker and the separator that closes
/// it.
fn first_document(mut content: Vec<u8>, separator: usize) -> io::Result<String> {
    content.truncate(separator);
    content.drain(..YAML_DOCUMENT_START.len());
    String::from_utf8(content).map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))
}

/// Strip a leading byte-order mark once enough bytes have arrived to tell
/// whether there is one. Reports whether the caller can go on reading this
/// chunk, or has to pull more bytes first.
fn take_byte_order_mark(content: &mut Vec<u8>, pending: &mut bool) -> bool {
    if !*pending {
        return true;
    }
    if content.len() < BYTE_ORDER_MARK.len() {
        return false;
    }
    *pending = false;
    if content.starts_with(BYTE_ORDER_MARK) {
        content.drain(..BYTE_ORDER_MARK.len());
    }
    true
}

View on GitHub (pinned to 49dbe22c32)