huggingface/tokenizers · error

Uninitialized Model

Error message

Uninitialized Model

What it means

The Node binding's Model wrapper holds Option<model>; get_vocab (and sibling methods) call .expect("Uninitialized Model") on it. If the wrapper was created with model: None — e.g. a model type instantiated empty instead of loaded from a vocab file — the expect panics and Node surfaces the panic as an error. The library throws because a vocabulary only exists once a concrete model (BPE/WordPiece/etc.) has been loaded.

Solutions

1. Initialize the model with its required data, e.g. BPE.from(vocab, merges) or load the whole tokenizer via tokenizer.fromJSON / from pretrained, before calling get_vocab.
2. Wrap the call in try/catch in JS; the panic surfaces as an exception you can handle and log.
3. Patch the binding (models.rs) to use self.model.as_ref().ok_or("Uninitialized Model")? with a proper napi Error instead of expect().

Example fix

// before
const bpe = new BPE(); // empty
bpe.getVocab(); // panics: Uninitialized Model

// after
const bpe = await BPE.from(vocabObject, mergesArray);
bpe.getVocab(); // { "hello": 0, ... }

// ensure model was created with vocab/merges before querying
if (!bpe || !bpe.getVocab) throw new Error('model not initialized; use BPE.from(vocab, merges) or tokenizer.fromJSON');

try { vocab = model.getVocab(); } catch (e) { if (String(e).includes('Uninitialized Model')) { model = await Tokenizer.fromJSON(config); vocab = model.model.getVocab(); } else throw e; }

When it happens

Trigger: Calling get_vocab()/get_vocab_size() on a model wrapper whose inner model was never initialized, e.g. `new BPE()` with no constructor args or a model created through a code path that leaves the Option empty.

Common situations: Building a tokenizer in JS by constructing model classes directly without passing vocab/merges; deserializing a tokenizer.json with the binding and inspecting the raw model object; version mismatches between JS wrapper and native library where constructors changed.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Uninitialized Encoding
• likelihood is NAN. Input sentence may be too long.
• Helper
• NormalizedString bad split
• AddedVocabulary bad split

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/90799d432be18b32. Report an issue: GitHub.

Appendix: source

Thrown at bindings/node/src/models.rs:126

        direction,
      ),
      None => pretokenized.tokenize(|normalized| guard.tokenize(normalized.get())),
    }
  }

  fn token_to_id(&self, token: &str) -> Option<u32> {
    self.model.as_ref()?.read().unwrap().token_to_id(token)
  }

  fn id_to_token(&self, id: u32) -> Option<String> {
    self.model.as_ref()?.read().unwrap().id_to_token(id)
  }

  fn get_vocab(&self) -> HashMap<String, u32> {
    self
      .model
      .as_ref()
      .expect("Uninitialized Model")
      .read()
      .unwrap()
      .get_vocab()
  }

  fn get_vocab_size(&self) -> usize {
    self
      .model
      .as_ref()
      .expect("Uninitialized Model")
      .read()
      .unwrap()
      .get_vocab_size()
  }

  fn save(&self, folder: &Path, name: Option<&str>) -> tk::Result<Vec<PathBuf>> {
    self
      .model

View on GitHub (pinned to 6cfd9d385c)