huggingface/tokenizers · error

Uninitialized Encoding

Error message

Uninitialized Encoding

What it means

The Node binding wraps the Rust Encoding in an Option that is only populated after a tokenizer `encode` call or explicit construction. `get_length` unwraps it with `.expect("Uninitialized Encoding")`, so calling this accessor on an Encoding object whose inner value was never set panics with 'Uninitialized Encoding'.

Solutions

1. Only call get_length on encodings produced by `tokenizer.encode(...)` or `tokenizer.encodeBatch(...)`.
2. If constructing Encoding manually (binding-internal code), pass a valid Rust Encoding to the constructor instead of leaving the Option None.
3. Wrap the call in try/catch in JS, since expect() panics surface as a thrown N-API error.
4. Check for null/undefined or an 'initialized' flag on the object before accessing encoding methods.

Example fix

// before
const enc = new Encoding();
console.log(enc.getLength()); // panic: Uninitialized Encoding
// after
const enc = tokenizer.encode("hello world");
console.log(enc.length); // valid: 2

function encodingIsUsable(enc) {
  return enc != null && typeof enc.getLength === "function" && enc._initialized !== false;
}
if (encodingIsUsable(enc)) console.log(enc.getLength());

function isInitializedEncoding(obj) {
  return obj instanceof Encoding && obj._encoding != null;
}

let length;
try {
  length = enc.getLength();
} catch (err) {
  if (String(err.message).includes("Uninitialized Encoding")) {
    length = 0;
  } else {
    throw err;
  }
}

When it happens

Trigger: Creating an empty Encoding object via the N-API constructor (e.g. `new encoding.Encoding()` in the bindings' own test/util code, or an Encoding returned in an uninitialized state) and then calling `getLength()` before it has been assigned a real encoding.

Common situations: Binding-internal tests or wrapper code that instantiates Encoding directly; deserialization/interop paths that hand back an Encoding placeholder without running encode; holding a reference to an encoding whose underlying value failed to initialize.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• likelihood is NAN. Input sentence may be too long.
• Uninitialized Model
• NormalizedString bad split
• Error in offsets
• Helper

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/4c31158224fb0421. Report an issue: GitHub.

Appendix: source

Thrown at bindings/node/src/encoding.rs:88

      JsTruncationStrategy::OnlyFirst => tokenizers::TruncationStrategy::OnlyFirst,
      JsTruncationStrategy::OnlySecond => tokenizers::TruncationStrategy::OnlySecond,
    }
  }
}

#[napi]
impl JsEncoding {
  #[napi(constructor)]
  pub fn new() -> Self {
    Self { encoding: None }
  }

  #[napi]
  pub fn get_length(&self) -> u32 {
    self
      .encoding
      .as_ref()
      .expect("Uninitialized Encoding")
      .get_ids()
      .len() as u32
  }

  #[napi]
  pub fn get_n_sequences(&self) -> u32 {
    self
      .encoding
      .as_ref()
      .expect("Uninitialized Encoding")
      .n_sequences() as u32
  }

  #[napi]
  pub fn get_ids(&self) -> Vec<u32> {
    self
      .encoding
      .as_ref()

View on GitHub (pinned to 6cfd9d385c)