janhq/jan · error · io::Error

<Utf8Error>

Error message

<Utf8Error>

What it means

Produced by `String::from_utf8(buf)` inside `read_gguf_string` when the bytes read for a GGUF string key or value are not valid UTF-8. The `Utf8Error` is wrapped into an `io::Error` of kind `InvalidData`. GGUF strings are spec-required to be UTF-8, so this indicates either corruption or a writer that emitted raw bytes.

Source

Thrown at src-tauri/plugins/tauri-plugin-llamacpp/src/gguf/helpers.rs:73

    let value_type_u32 = reader.read_u32::<LittleEndian>()?;
    let value_type = GgufValueType::try_from(value_type_u32)?;
    let value = read_gguf_value(reader, value_type)?;

    Ok((key, value))
}

fn read_gguf_string<R: Read + ReadBytesExt>(reader: &mut R) -> io::Result<String> {
    let len = reader.read_u64::<LittleEndian>()?;
    if len > (1024 * 1024) {
        return Err(io::Error::new(
            io::ErrorKind::InvalidData,
            format!("String length {} is unreasonably large", len),
        ));
    }
    let mut buf = vec![0u8; len as usize];
    reader.read_exact(&mut buf)?;
    String::from_utf8(buf).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))
}

fn read_gguf_value<R: Read + Seek + ReadBytesExt>(
    reader: &mut R,
    value_type: GgufValueType,
) -> io::Result<String> {
    match value_type {
        GgufValueType::Uint8 => Ok(reader.read_u8()?.to_string()),
        GgufValueType::Int8 => Ok(reader.read_i8()?.to_string()),
        GgufValueType::Uint16 => Ok(reader.read_u16::<LittleEndian>()?.to_string()),
        GgufValueType::Int16 => Ok(reader.read_i16::<LittleEndian>()?.to_string()),
        GgufValueType::Uint32 => Ok(reader.read_u32::<LittleEndian>()?.to_string()),
        GgufValueType::Int32 => Ok(reader.read_i32::<LittleEndian>()?.to_string()),
        GgufValueType::Float32 => Ok(reader.read_f32::<LittleEndian>()?.to_string()),
        GgufValueType::Bool => Ok((reader.read_u8()? != 0).to_string()),
        GgufValueType::String => read_gguf_string(reader),
        GgufValueType::Uint64 => Ok(reader.read_u64::<LittleEndian>()?.to_string()),
        GgufValueType::Int64 => Ok(reader.read_i64::<LittleEndian>()?.to_string()),

View on GitHub (pinned to fad3f12a14)

Solutions

1. Confirm stream alignment by re-reading the preceding value with the correct width from the GGUF spec.
2. Lossily inspect the bytes (`String::from_utf8_lossy`) during debugging to see whether they look like partial text (misalignment) or random bytes (corruption).
3. Verify the file checksum and re-download if corrupted.
4. If the value is genuinely non-text, report the bug to the GGUF producer; do not silently coerce.

Example fix

// before
String::from_utf8(buf).map_err(|e| io::Error::new(InvalidData, e))

// after - keep lossy variant for diagnostics but still fail loudly
String::from_utf8(buf).map_err(|e| {
    let lossy = String::from_utf8_lossy(&buf[..e.valid_up_to().saturating_add(1)]);
    io::Error::new(InvalidData,
        format!("invalid UTF-8 at byte {}: valid_prefix={:?}", e.valid_up_to(), lossy))
})

null

fn is_valid_metadata_key(s: &str) -> bool {
    s.chars().all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '_')
}

match read_gguf_string(reader) {
    Ok(s) => Ok(s),
    Err(e) if e.to_string().contains("Utf8Error") || e.kind() == InvalidData => {
        Err(CorruptFile("non-UTF-8 string in metadata".into()))
    }
    Err(e) => Err(e.into()),
}

When it happens

Trigger: The declared-length byte slice for a metadata key or string value contains invalid UTF-8 sequences (lone continuation bytes, truncated multibyte sequences). Most often the length was correct but the bytes are not textual — e.g. a binary blob was mis-tagged as a string, or the stream is misaligned and the reader is interpreting unrelated tensor/value bytes as a string.

Common situations: Stream misalignment after an earlier field was read with the wrong width; a quantized file whose metadata contains a non-text blob; corruption during download or disk write; an older GGUF writer that did not enforce UTF-8.

Related errors

• String length {} is unreasonably large
• Not a GGUF file
• Failed to read key for metadata entry {}: {}
• Array length {} is unreasonably large
• Unknown GGUF value type: {}

AI-assisted analysis of janhq/jan@fad3f12a14 (2026-08-12). Data as JSON: /api/errors/96a166b39300b3ad. Report an issue: GitHub.