What causes "Tokenizer is closed; call reset() before reusing." in can1357/oh-my-pi?

Call reset() on the Tokenizer before feeding new data after it was closed.

can1357/oh-my-pi · error · Error

Tokenizer is closed; call reset() before reusing.

Error message

Tokenizer is closed; call reset() before reusing.

What it means

Tokenizer accumulates chunked text and emits line tokens; after end() is called it is closed (this.#closed = true) because it has flushed its final state. Any subsequent feed() on a closed Tokenizer throws this error. Tokenizers are single-pass: close, then reset() to clear state and reuse the instance.

Source

Thrown at packages/hashline/src/tokenizer.ts:578

		line.startsWith(HL_PUT_KEYWORD, lead) ||
		line.startsWith(HL_CUT_KEYWORD, lead) ||
		line.startsWith(HL_REM_KEYWORD, lead) ||
		line.startsWith(HL_MOVE_KEYWORD, lead);
	if (isHunkLead) {
		const hunk = tryParseHunkHeader(line);
		if (hunk !== null) return { kind: "op-block", lineNum, target: hunk.target, hadColon: hunk.hadColon };
	}
	if (firstCode === CHAR_PAYLOAD_REPLACE) return { kind: "payload-literal", lineNum, text: line.slice(1) };
	return { kind: "raw", lineNum, text: line };
}

export class Tokenizer {
	#buffer = "";
	#nextLineNum = 1;
	#closed = false;

	feed(chunk: string): Token[] {
		if (this.#closed) throw new Error("Tokenizer is closed; call reset() before reusing.");
		if (chunk.length === 0) return [];
		this.#buffer = this.#buffer ? this.#buffer + chunk : chunk;
		return this.#drainCompleteLines();
	}

	end(): Token[] {
		if (this.#closed) return [];
		this.#closed = true;
		const buf = this.#buffer;
		this.#buffer = "";
		if (buf.length === 0) return [];
		let stop = buf.length;
		if (buf.charCodeAt(stop - 1) === CHAR_CARRIAGE_RETURN) stop--;
		return [classifyLine(buf.slice(0, stop), this.#nextLineNum++)];
	}

	reset(): void {
		this.#buffer = "";

View on GitHub (pinned to 9690622007)

Solutions

Call reset() on the Tokenizer before feeding new data after it was closed.
Better: create a fresh Tokenizer instance for each new input stream.
Restructure code so feed()/end() are called exactly once per input, e.g. wrap the tokenize step in a function that constructs a new instance.

Example fix

// before
tokenizer.end();
tokenizer.feed(moreData); // throws
// after
tokenizer.reset();
tokenizer.feed(moreData); // ok

Defensive patterns

Strategy: try-catch

Validate before calling

// guard before reuse
if (tokenizerIsClosed) tokenizer.reset(); // track closed state yourself or always reset between streams

Type guard

// Tokenizer has no public closed getter; enforce by construction:
// only feed a tokenizer you have not called end() on (track in a WeakSet)
const ended = new WeakSet<object>();
function isTokenizerOpen(t: object): boolean {
	return !ended.has(t);
}

Try / catch

try {
	tokens.push(...tokenizer.feed(chunk));
} catch (err) {
	if (err instanceof Error && err.message.includes("Tokenizer is closed")) {
		tokenizer.reset();
		tokens.push(...tokenizer.feed(chunk));
	} else throw err;
}

Prevention

Create a new Tokenizer per input stream instead of reusing instances.
Always pair end() with reset() if the instance will be reused.
Never cache Tokenizer instances in module-level maps across requests.
Wrap feed/end in a small helper that owns lifecycle.

When it happens

Trigger: Calling feed() (or end() again after a successful drain path that closes) after end() has already been invoked, without calling reset(). Typically from reusing a cached Tokenizer across two streams/files.

Common situations: Reusing a Tokenizer object for a second file; a retry path that re-feeds data after the stream already ended; long-lived tokenizers stored in a module-level map across requests.

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/7b5890c79a622bb9. Report an issue: GitHub.