microsoft/vscode · error · CopilotPromptLoadFailure

Could not load tokenizer

Error message

Could not load tokenizer

What it means

Error "Could not load tokenizer" thrown in microsoft/vscode.

Source

Thrown at extensions/copilot/src/extension/completions-core/vscode-node/prompt/src/tokenization/tokenizer.ts:164

	 */
	takeLastLinesTokens(text: string, n: number): string;
}

export class TTokenizer implements Tokenizer {
	constructor(private readonly _tokenizer: TikTokenizer) { }

	static async create(encoder: TokenizerName): Promise<TTokenizer> {
		try {
			const tokenizer = createTokenizer(
				parseTikTokenBinary(locateFile(`${encoder}.tiktoken`)),
				getSpecialTokensByEncoder(encoder),
				getRegexByEncoder(encoder),
				32768
			);
			return new TTokenizer(tokenizer);
		} catch (e: unknown) {
			if (e instanceof Error) {
				throw new CopilotPromptLoadFailure(`Could not load tokenizer`, e);
			}
			throw e;
		}
	}

	tokenize(text: string): number[] {
		return this._tokenizer.encode(text);
	}

	detokenize(tokens: number[]): string {
		return this._tokenizer.decode(tokens);
	}

	tokenLength(text: string): number {
		return this.tokenize(text).length;
	}

	tokenizeStrings(text: string): string[] {

View on GitHub (pinned to a94b963a32)

When it happens

Trigger: Thrown at extensions/copilot/src/extension/completions-core/vscode-node/prompt/src/tokenization/tokenizer.ts:164 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of microsoft/vscode@a94b963a32 (2026-08-12). Data as JSON: /api/errors/7e3ec51b04474928. Report an issue: GitHub.