valeriansaliou/sonic · error · panic

unable to initialize japanese tokenizer

Error message

unable to initialize japanese tokenizer

What it means

The token lexer lazily initializes a lindera Japanese tokenizer for the Japanese language and panics with this message if lindera cannot build the tokenizer (e.g. it cannot load or locate the embedded/external dictionary). Without the tokenizer, Japanese text cannot be segmented into tokens.

Source

Thrown at core/src/lexer/token.rs:311

const TEXT_LANG_DETECT_PROCEED_OVER_CHARS: usize = 20;
const TEXT_LANG_DETECT_NGRAM_UNDER_CHARS: usize = 60;

#[cfg(feature = "tokenizer-chinese")]
static TOKENIZER_JIEBA: LazyLock<jieba_rs::Jieba> = LazyLock::new(jieba_rs::Jieba::new);

#[cfg(feature = "tokenizer-japanese")]
static TOKENIZER_LINDERA: LazyLock<lindera_tokenizer::tokenizer::Tokenizer> = LazyLock::new(|| {
    lindera_tokenizer::tokenizer::Tokenizer::from_config(
        lindera_tokenizer::tokenizer::TokenizerConfig {
            dictionary: lindera_dictionary::DictionaryConfig {
                kind: Some(lindera_dictionary::DictionaryKind::UniDic),
                path: None,
            },
            user_dictionary: None,
            mode: lindera_core::mode::Mode::Normal,
        },
    )
    .expect("unable to initialize japanese tokenizer")
});

impl TokenLexerBuilder {
    pub fn from<'a>(
        mode: TokenLexerMode,
        lang: Option<Lang>,
        text: &'a str,
        normalization_config: ConfigNormalization,
        tokenization_config: ConfigTokenization,
        stopwords_config: &'a ConfigStopwords,
    ) -> Result<TokenLexer<'a>, ()> {
        let locale = match lang {
            // If user provided a language, use it.
            Some(hinted_lang) => {
                // Use hinted language (current lexer mode asks for a cleanup)
                tracing::debug!(
                    "using hinted locale: {} from lexer text: {}",
                    hinted_lang,

View on GitHub (pinned to e6a72da6a5)

Solutions

1. Enable the correct lindera dictionary feature (e.g. lindera = { features = ["ipadic"] }) in Cargo.toml
2. Verify lindera version compatibility between sonic-core's dependency tree and the dictionary crate
3. Rebuild the project so the dictionary assets are embedded, and confirm the binary runs in the same environment it was built for
4. If using an external dictionary path, confirm the path exists and is readable at runtime

Example fix

// before
lindera = "0.16"            # dictionary feature missing
// after
lindera = { version = "0.16", features = ["ipadic"] }

// Cargo.toml — ensure dictionary feature before building
[dependencies]
lindera = { version = "0.16", features = ["ipadic"] }

fn japanese_supported(lang: &Lang) -> bool {
    matches!(lang, Lang::Japanese) && cfg!(feature = "ipadic")
}

let lexer = std::panic::catch_unwind(|| {
    TokenLexerBuilder::new(TokenLexerMode::Default, Some(Lang::Japanese)).build()
});
match lexer {
    Ok(l) => l,
    Err(_) => fallback_to_whitespace_tokenizer(),
}

When it happens

Trigger: Creating a TokenLexer with lang = Japanese (via TokenLexerBuilder) when the lindera dictionary (e.g. embedded IPADIC) is unavailable — missing dictionary feature flag, corrupted/unreachable dictionary path, or incompatible lindera version.

Common situations: Building without the lindera dictionary feature enabled; mismatched lindera crate versions between core and lexer; running in a stripped environment where the dictionary asset wasn't bundled.

Related errors

• error reading config
• syntax error in config
• invalid log level
• BrokenPipe
• InvalidData

AI-assisted analysis of valeriansaliou/sonic@e6a72da6a5 (2026-09-01). Data as JSON: /api/errors/85c7860fcc46bfb4. Report an issue: GitHub.