sxyazi/yazi · error

Invalid WTF-8 sequence

Error message

Invalid WTF-8 sequence

What it means

`from_wtf8` on the windows target converts raw WTF-8 bytes (the UTF-8 superset that can encode unpaired surrogates used in Windows paths) into a path string, but first validates the bytes with `super::valid_wtf8`. Invalid byte sequences — e.g. malformed surrogate encodings or bad UTF-8 — are rejected with this error rather than producing a corrupted path.

Solutions

1. Validate the byte source — re-encode from UTF-16 with the proper surrogate algorithm before conversion
2. Sanitize or replace unpaired surrogates with U+FFFD before calling from_wtf8
3. Check for truncation: ensure the buffer contains complete multi-byte sequences
4. If data comes from Win32 APIs, use the correct wide-to-WTF-8 conversion helpers

Example fix

// before
Wtf8Buf::from_wtf8(raw_bytes)?; // raw_bytes has a truncated 3-byte surrogate
// after
let fixed = String::from_utf8_lossy(&raw_bytes).into_owned(); // sanitize first
let path = Wtf8Buf::from_wtf8(fixed.into_bytes())?;

fn is_valid_wtf8(b: &[u8]) -> bool {
    std::str::from_utf8(b).is_ok() // fast path; full check lives in yazi_shim::wtf8::valid_wtf8
        || yazi_shim::wtf8::valid_wtf8(b)
}
if !is_valid_wtf8(&bytes) { sanitize(&mut bytes); }

match Wtf8Buf::from_wtf8(bytes) {
    Err(e) if e.to_string() == "Invalid WTF-8 sequence" => sanitize_and_retry(bytes),
    other => other,
}

When it happens

Trigger: Calling `FromWtf8::from_wtf8` with a byte buffer that fails `valid_wtf8` on Windows: bytes are neither valid UTF-8 nor valid WTF-8 (e.g. an encoded unpaired surrogate written incorrectly, or truncated multi-byte sequence).

Common situations: Receiving file names over IPC or from FFI buffers on Windows and constructing paths from raw bytes; data corrupted in transit; hand-crafting WTF-8 for testing unpaired-surrogate paths.

Related errors

• {e}
• Failed to get asset directory
• Failed to get cache directory
• Failed to get config directory
• Failed to get state directory

AI-assisted analysis of sxyazi/yazi@5f901b886b (2026-09-02). Data as JSON: /api/errors/a88237de58e3076d. Report an issue: GitHub.

Appendix: source

Thrown at yazi-shim/src/wtf8/wtf8.rs:41

// --- FromWtf8
pub trait FromWtf8 {
	fn from_wtf8(wtf8: &[u8]) -> Result<&Self>;
}

impl FromWtf8 for OsStr {
	fn from_wtf8(wtf8: &[u8]) -> Result<&Self> {
		#[cfg(unix)]
		{
			use std::os::unix::ffi::OsStrExt;
			Ok(Self::from_bytes(wtf8))
		}
		#[cfg(windows)]
		{
			if super::valid_wtf8(wtf8) {
				Ok(unsafe { Self::from_encoded_bytes_unchecked(wtf8) })
			} else {
				Err(anyhow::anyhow!("Invalid WTF-8 sequence"))
			}
		}
	}
}

impl FromWtf8 for std::path::Path {
	fn from_wtf8(wtf8: &[u8]) -> Result<&Self> { Ok(OsStr::from_wtf8(wtf8)?.as_ref()) }
}

// --- FromWtf8Vec
pub trait FromWtf8Vec {
	fn from_wtf8_vec(wtf8: Vec<u8>) -> Result<Self>
	where
		Self: Sized;
}

impl FromWtf8Vec for OsString {
	fn from_wtf8_vec(wtf8: Vec<u8>) -> Result<Self> {

View on GitHub (pinned to 5f901b886b)