huggingface/tokenizers · error · Error

Error in offsets

Error message

Error in offsets

What it means

The Node binding's `slice` helper extracts a substring of a normalized string via the Rust `get_range_of` range helper. `get_range_of` returns None when the requested character range is invalid for the string (out of bounds or inverted range), and the binding converts that None into this generic N-API error 'Error in offsets'.

Solutions

1. Clamp and validate indices before calling: ensure 0 <= begin_index <= end_index <= string length (in characters, not bytes).
2. Swap begin/end if inverted, and skip the call when the range is empty or the string is empty.
3. Verify offsets come from the same (unmodified) string/encoding the slice is applied to, and that they are character offsets.

Example fix

// before
const text = tokenizer.decode(ids);
const piece = slice(text, start, end); // Error in offsets when end > len
// after
const len = [...text].length;
const b = Math.max(0, Math.min(start ?? 0, len));
const e = Math.max(b, Math.min(end ?? len, len));
const piece = b < e ? slice(text, b, e) : "";

function validRange(text, begin, end) {
  const len = [...text].length;
  const b = Math.max(0, Math.min(begin ?? 0, len));
  const e = Math.max(0, Math.min(end ?? len, len));
  return b <= e ? [b, e] : null;
}
const r = validRange(text, start, end);
if (!r) throw new RangeError("invalid offsets");
const piece = slice(text, r[0], r[1]);

function isSafeOffset(n) {
  return Number.isInteger(n) && n >= 0;
}

try {
  const piece = slice(text, begin, end);
} catch (err) {
  if (err.message === "Error in offsets") {
    piece = ""; // or clamp indices and retry
  } else {
    throw err;
  }
}

When it happens

Trigger: Calling `OffsetReferential`/`slice` (the util used by offset-reference lookups) with begin_index/end_index such that begin > end, either index beyond the string's character length, or a negative index beyond bounds — e.g. slice(s, 10, 5) or slice(s, 0, 999) on a 3-char string.

Common situations: Computing offsets from token span indices without clamping to the string length; passing byte offsets instead of character offsets on multibyte text; inverted start/end from sorting mistakes; stale offsets after the text was modified.

Related errors

• Uninitialized Encoding
• NormalizedString bad split

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/eda161ee1760b05f. Report an issue: GitHub.

Appendix: source

Thrown at bindings/node/src/utils.rs:26

#[napi]
pub fn slice(s: String, begin_index: Option<i32>, end_index: Option<i32>) -> Result<String> {
  let len = s.chars().count();

  let get_index = |x: i32| -> usize {
    if x >= 0 {
      x as usize
    } else {
      (len as i32 + x) as usize
    }
  };

  let begin_index = get_index(begin_index.unwrap_or(0));
  let end_index = get_index(end_index.unwrap_or(len as i32));

  if let Some(slice) = tk::tokenizer::normalizer::get_range_of(&s, begin_index..end_index) {
    Ok(slice.to_string())
  } else {
    Err(Error::new(
      Status::GenericFailure,
      "Error in offsets".to_string(),
    ))
  }
}

#[napi]
pub fn merge_encodings(
  encodings: Vec<&JsEncoding>,
  growing_offsets: Option<bool>,
) -> Result<JsEncoding> {
  let growing_offsets = growing_offsets.unwrap_or(false);

  let encodings: Vec<_> = encodings
    .into_iter()
    .map(|enc| enc.encoding.to_owned().unwrap())
    .collect();

View on GitHub (pinned to 6cfd9d385c)