zed-industries/zed · error

invalid utf-8 in semantic_token_rules

Error message

invalid utf-8 in semantic_token_rules

What it means

Panics because the embedded cpp/semantic_token_rules.json bytes are not valid UTF-8 during std::str::from_utf8. A vendored JSON file should always be UTF-8; failure means the embedded asset is corrupt or was stored with a BOM/binary encoding at packaging time.

Source

Thrown at crates/languages/src/cpp.rs:6

use settings::SemanticTokenRules;

pub(crate) fn semantic_token_rules() -> SemanticTokenRules {
    let content = grammars::get_file("cpp/semantic_token_rules.json")
        .expect("missing cpp/semantic_token_rules.json");
    let json = std::str::from_utf8(&content.data).expect("invalid utf-8 in semantic_token_rules");
    settings::parse_json_with_comments::<SemanticTokenRules>(json)
        .expect("failed to parse cpp semantic_token_rules.json")
}

#[cfg(test)]
mod tests {
    use gpui::{AppContext as _, BorrowAppContext, TestAppContext};
    use language::{AutoindentMode, Buffer};
    use settings::SettingsStore;
    use std::num::NonZeroU32;
    use unindent::Unindent;

    #[gpui::test]
    async fn test_cpp_autoindent_switch_case(cx: &mut TestAppContext) {
        cx.update(|cx| {
            let test_settings = SettingsStore::test(cx);
            cx.set_global(test_settings);
            cx.update_global::<SettingsStore, _>(|store, cx| {

View on GitHub (pinned to f4178619ac)

Solutions

1. Inspect the vendored cpp/semantic_token_rules.json for non-UTF-8 bytes and re-save it as UTF-8
2. Validate all embedded JSON assets with a build-time test asserting from_utf8 succeeds
3. Use from_utf8_lossy to degrade instead of panicking on a corrupt asset

When it happens

Trigger: Thrown at crates/languages/src/cpp.rs:6 when the library encounters an invalid state.

Common situations: See trigger scenarios.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

AI-assisted analysis of zed-industries/zed@f4178619ac (2026-08-20). Data as JSON: /api/errors/ed94db1d86dfcba6. Report an issue: GitHub.