diegosouzapw/OmniRoute · error · Error

js-tiktoken not available: cannot create encoder for ${encod

Error message

js-tiktoken not available: cannot create encoder for ${encoding}

What it means

Error "js-tiktoken not available: cannot create encoder for ${encoding}" thrown in diegosouzapw/OmniRoute.

Source

Thrown at src/shared/utils/tiktokenCounter.ts:77

    model.includes("codex")
  );
}

export function resolveTokenizerEncoding(context?: TokenizerContext): TokenizerEncoding {
  return isCodexTokenizerContext(context) ? "o200k_base" : "cl100k_base";
}

function getEncoder(encoding: TokenizerEncoding): Tiktoken {
  const cached = encoders.get(encoding);
  if (cached) return cached;
  let tiktoken: { getEncoding: (name: string) => Tiktoken } | null = null;
  try {
    tiktoken ??= _require("js-tiktoken") as { getEncoding: (name: string) => Tiktoken };
    const created = tiktoken.getEncoding(encoding);
    encoders.set(encoding, created);
    return created;
  } catch {
    throw new Error(`js-tiktoken not available: cannot create encoder for ${encoding}`);
  }
}

/**
 * Exact token count for a string using the selected offline tokenizer.
 * Existing callers retain cl100k_base; Codex callers may pass provider/model context
 * to use o200k_base.
 * Defensive: never throws in a counting path — falls back to a char heuristic.
 * Oversized inputs (over 50k chars) and base64 image data URIs are never
 * tokenized: the encoder is near-quadratic on large strings and would block the
 * event loop (worker wedge regression).
 */
export function countTextTokens(text: string, context?: TokenizerContext): number {
  if (!text || typeof text !== "string") return 0;
  const stripped = stripBase64DataUris(text);
  if (stripped.length > MAX_EXACT_TOKEN_COUNT_CHARS) {
    return Math.ceil(stripped.length / 4);
  }

View on GitHub (pinned to a179ffed5b)

When it happens

Trigger: Thrown at src/shared/utils/tiktokenCounter.ts:77 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of diegosouzapw/OmniRoute@a179ffed5b (2026-08-25). Data as JSON: /api/errors/f02cd5b1f0d06a33. Report an issue: GitHub.