{"record":{"id":"85c7860fcc46bfb4","repo":"valeriansaliou/sonic","slug":"unable-to-initialize-japanese-tokenizer","errorCode":null,"errorMessage":"unable to initialize japanese tokenizer","messagePattern":"unable to initialize japanese tokenizer","errorType":"panic","errorClass":"panic","httpStatus":null,"severity":"error","filePath":"core/src/lexer/token.rs","lineNumber":311,"sourceCode":"const TEXT_LANG_DETECT_PROCEED_OVER_CHARS: usize = 20;\nconst TEXT_LANG_DETECT_NGRAM_UNDER_CHARS: usize = 60;\n\n#[cfg(feature = \"tokenizer-chinese\")]\nstatic TOKENIZER_JIEBA: LazyLock<jieba_rs::Jieba> = LazyLock::new(jieba_rs::Jieba::new);\n\n#[cfg(feature = \"tokenizer-japanese\")]\nstatic TOKENIZER_LINDERA: LazyLock<lindera_tokenizer::tokenizer::Tokenizer> = LazyLock::new(|| {\n    lindera_tokenizer::tokenizer::Tokenizer::from_config(\n        lindera_tokenizer::tokenizer::TokenizerConfig {\n            dictionary: lindera_dictionary::DictionaryConfig {\n                kind: Some(lindera_dictionary::DictionaryKind::UniDic),\n                path: None,\n            },\n            user_dictionary: None,\n            mode: lindera_core::mode::Mode::Normal,\n        },\n    )\n    .expect(\"unable to initialize japanese tokenizer\")\n});\n\nimpl TokenLexerBuilder {\n    pub fn from<'a>(\n        mode: TokenLexerMode,\n        lang: Option<Lang>,\n        text: &'a str,\n        normalization_config: ConfigNormalization,\n        tokenization_config: ConfigTokenization,\n        stopwords_config: &'a ConfigStopwords,\n    ) -> Result<TokenLexer<'a>, ()> {\n        let locale = match lang {\n            // If user provided a language, use it.\n            Some(hinted_lang) => {\n                // Use hinted language (current lexer mode asks for a cleanup)\n                tracing::debug!(\n                    \"using hinted locale: {} from lexer text: {}\",\n                    hinted_lang,","sourceCodeStart":293,"sourceCodeEnd":329,"githubUrl":"https://github.com/valeriansaliou/sonic/blob/e6a72da6a532bc3f31d7d93ef1e0964a1a5d01b2/core/src/lexer/token.rs#L293-L329","documentation":"The token lexer lazily initializes a lindera Japanese tokenizer for the Japanese language and panics with this message if lindera cannot build the tokenizer (e.g. it cannot load or locate the embedded/external dictionary). Without the tokenizer, Japanese text cannot be segmented into tokens.","triggerScenarios":"Creating a TokenLexer with lang = Japanese (via TokenLexerBuilder) when the lindera dictionary (e.g. embedded IPADIC) is unavailable — missing dictionary feature flag, corrupted/unreachable dictionary path, or incompatible lindera version.","commonSituations":"Building without the lindera dictionary feature enabled; mismatched lindera crate versions between core and lexer; running in a stripped environment where the dictionary asset wasn't bundled.","solutions":["Enable the correct lindera dictionary feature (e.g. lindera = { features = [\"ipadic\"] }) in Cargo.toml","Verify lindera version compatibility between sonic-core's dependency tree and the dictionary crate","Rebuild the project so the dictionary assets are embedded, and confirm the binary runs in the same environment it was built for","If using an external dictionary path, confirm the path exists and is readable at runtime"],"exampleFix":"// before\nlindera = \"0.16\"            # dictionary feature missing\n// after\nlindera = { version = \"0.16\", features = [\"ipadic\"] }","handlingStrategy":"validation","validationCode":"// Cargo.toml — ensure dictionary feature before building\n[dependencies]\nlindera = { version = \"0.16\", features = [\"ipadic\"] }","typeGuard":"fn japanese_supported(lang: &Lang) -> bool {\n    matches!(lang, Lang::Japanese) && cfg!(feature = \"ipadic\")\n}","tryCatchPattern":"let lexer = std::panic::catch_unwind(|| {\n    TokenLexerBuilder::new(TokenLexerMode::Default, Some(Lang::Japanese)).build()\n});\nmatch lexer {\n    Ok(l) => l,\n    Err(_) => fallback_to_whitespace_tokenizer(),\n}","preventionTips":["Always enable the lindera dictionary feature when supporting Japanese","Pin lindera versions consistently across workspace crates","Smoke-test Japanese indexing at startup rather than lazily on first request","If using an external dictionary, verify its path exists before launching"],"tags":["rust","tokenizer","japanese","configuration"],"backgroundTag":"tokenizer-init-failed","analyzedSha":"e6a72da6a532bc3f31d7d93ef1e0964a1a5d01b2","analyzedAt":"2026-09-01T14:10:00.384Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T20:17:18.057Z"}