{"record":{"id":"eda161ee1760b05f","repo":"huggingface/tokenizers","slug":"error-in-offsets","errorCode":null,"errorMessage":"Error in offsets","messagePattern":"Error in offsets","errorType":"error_code","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"bindings/node/src/utils.rs","lineNumber":26,"sourceCode":"#[napi]\npub fn slice(s: String, begin_index: Option<i32>, end_index: Option<i32>) -> Result<String> {\n  let len = s.chars().count();\n\n  let get_index = |x: i32| -> usize {\n    if x >= 0 {\n      x as usize\n    } else {\n      (len as i32 + x) as usize\n    }\n  };\n\n  let begin_index = get_index(begin_index.unwrap_or(0));\n  let end_index = get_index(end_index.unwrap_or(len as i32));\n\n  if let Some(slice) = tk::tokenizer::normalizer::get_range_of(&s, begin_index..end_index) {\n    Ok(slice.to_string())\n  } else {\n    Err(Error::new(\n      Status::GenericFailure,\n      \"Error in offsets\".to_string(),\n    ))\n  }\n}\n\n#[napi]\npub fn merge_encodings(\n  encodings: Vec<&JsEncoding>,\n  growing_offsets: Option<bool>,\n) -> Result<JsEncoding> {\n  let growing_offsets = growing_offsets.unwrap_or(false);\n\n  let encodings: Vec<_> = encodings\n    .into_iter()\n    .map(|enc| enc.encoding.to_owned().unwrap())\n    .collect();\n","sourceCodeStart":8,"sourceCodeEnd":44,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/node/src/utils.rs#L8-L44","documentation":"The Node binding's `slice` helper extracts a substring of a normalized string via the Rust `get_range_of` range helper. `get_range_of` returns None when the requested character range is invalid for the string (out of bounds or inverted range), and the binding converts that None into this generic N-API error 'Error in offsets'.","triggerScenarios":"Calling `OffsetReferential`/`slice` (the util used by offset-reference lookups) with begin_index/end_index such that begin > end, either index beyond the string's character length, or a negative index beyond bounds — e.g. slice(s, 10, 5) or slice(s, 0, 999) on a 3-char string.","commonSituations":"Computing offsets from token span indices without clamping to the string length; passing byte offsets instead of character offsets on multibyte text; inverted start/end from sorting mistakes; stale offsets after the text was modified.","solutions":["Clamp and validate indices before calling: ensure 0 <= begin_index <= end_index <= string length (in characters, not bytes).","Swap begin/end if inverted, and skip the call when the range is empty or the string is empty.","Verify offsets come from the same (unmodified) string/encoding the slice is applied to, and that they are character offsets."],"exampleFix":"// before\nconst text = tokenizer.decode(ids);\nconst piece = slice(text, start, end); // Error in offsets when end > len\n// after\nconst len = [...text].length;\nconst b = Math.max(0, Math.min(start ?? 0, len));\nconst e = Math.max(b, Math.min(end ?? len, len));\nconst piece = b < e ? slice(text, b, e) : \"\";","handlingStrategy":"validation","validationCode":"function validRange(text, begin, end) {\n  const len = [...text].length;\n  const b = Math.max(0, Math.min(begin ?? 0, len));\n  const e = Math.max(0, Math.min(end ?? len, len));\n  return b <= e ? [b, e] : null;\n}\nconst r = validRange(text, start, end);\nif (!r) throw new RangeError(\"invalid offsets\");\nconst piece = slice(text, r[0], r[1]);","typeGuard":"function isSafeOffset(n) {\n  return Number.isInteger(n) && n >= 0;\n}","tryCatchPattern":"try {\n  const piece = slice(text, begin, end);\n} catch (err) {\n  if (err.message === \"Error in offsets\") {\n    piece = \"\"; // or clamp indices and retry\n  } else {\n    throw err;\n  }\n}","preventionTips":["Clamp begin/end to [0, character length] and ensure begin <= end before slicing.","Use character (not byte) offsets, especially with multibyte text.","Derive offsets from the same encoding/string instance you slice; never reuse stale offsets."],"tags":["node","offsets","range-error","argument-out-of-range"],"backgroundTag":"argument-out-of-range","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}