{"record":{"id":"7b5890c79a622bb9","repo":"can1357/oh-my-pi","slug":"tokenizer-is-closed-call-reset-before-reusing","errorCode":null,"errorMessage":"Tokenizer is closed; call reset() before reusing.","messagePattern":"Tokenizer is closed; call reset\\(\\) before reusing\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/hashline/src/tokenizer.ts","lineNumber":578,"sourceCode":"\t\tline.startsWith(HL_PUT_KEYWORD, lead) ||\n\t\tline.startsWith(HL_CUT_KEYWORD, lead) ||\n\t\tline.startsWith(HL_REM_KEYWORD, lead) ||\n\t\tline.startsWith(HL_MOVE_KEYWORD, lead);\n\tif (isHunkLead) {\n\t\tconst hunk = tryParseHunkHeader(line);\n\t\tif (hunk !== null) return { kind: \"op-block\", lineNum, target: hunk.target, hadColon: hunk.hadColon };\n\t}\n\tif (firstCode === CHAR_PAYLOAD_REPLACE) return { kind: \"payload-literal\", lineNum, text: line.slice(1) };\n\treturn { kind: \"raw\", lineNum, text: line };\n}\n\nexport class Tokenizer {\n\t#buffer = \"\";\n\t#nextLineNum = 1;\n\t#closed = false;\n\n\tfeed(chunk: string): Token[] {\n\t\tif (this.#closed) throw new Error(\"Tokenizer is closed; call reset() before reusing.\");\n\t\tif (chunk.length === 0) return [];\n\t\tthis.#buffer = this.#buffer ? this.#buffer + chunk : chunk;\n\t\treturn this.#drainCompleteLines();\n\t}\n\n\tend(): Token[] {\n\t\tif (this.#closed) return [];\n\t\tthis.#closed = true;\n\t\tconst buf = this.#buffer;\n\t\tthis.#buffer = \"\";\n\t\tif (buf.length === 0) return [];\n\t\tlet stop = buf.length;\n\t\tif (buf.charCodeAt(stop - 1) === CHAR_CARRIAGE_RETURN) stop--;\n\t\treturn [classifyLine(buf.slice(0, stop), this.#nextLineNum++)];\n\t}\n\n\treset(): void {\n\t\tthis.#buffer = \"\";","sourceCodeStart":560,"sourceCodeEnd":596,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/hashline/src/tokenizer.ts#L560-L596","documentation":"Tokenizer accumulates chunked text and emits line tokens; after end() is called it is closed (this.#closed = true) because it has flushed its final state. Any subsequent feed() on a closed Tokenizer throws this error. Tokenizers are single-pass: close, then reset() to clear state and reuse the instance.","triggerScenarios":"Calling feed() (or end() again after a successful drain path that closes) after end() has already been invoked, without calling reset(). Typically from reusing a cached Tokenizer across two streams/files.","commonSituations":"Reusing a Tokenizer object for a second file; a retry path that re-feeds data after the stream already ended; long-lived tokenizers stored in a module-level map across requests.","solutions":["Call reset() on the Tokenizer before feeding new data after it was closed.","Better: create a fresh Tokenizer instance for each new input stream.","Restructure code so feed()/end() are called exactly once per input, e.g. wrap the tokenize step in a function that constructs a new instance."],"exampleFix":"// before\ntokenizer.end();\ntokenizer.feed(moreData); // throws\n// after\ntokenizer.reset();\ntokenizer.feed(moreData); // ok","handlingStrategy":"try-catch","validationCode":"// guard before reuse\nif (tokenizerIsClosed) tokenizer.reset(); // track closed state yourself or always reset between streams","typeGuard":"// Tokenizer has no public closed getter; enforce by construction:\n// only feed a tokenizer you have not called end() on (track in a WeakSet)\nconst ended = new WeakSet<object>();\nfunction isTokenizerOpen(t: object): boolean {\n\treturn !ended.has(t);\n}","tryCatchPattern":"try {\n\ttokens.push(...tokenizer.feed(chunk));\n} catch (err) {\n\tif (err instanceof Error && err.message.includes(\"Tokenizer is closed\")) {\n\t\ttokenizer.reset();\n\t\ttokens.push(...tokenizer.feed(chunk));\n\t} else throw err;\n}","preventionTips":["Create a new Tokenizer per input stream instead of reusing instances.","Always pair end() with reset() if the instance will be reused.","Never cache Tokenizer instances in module-level maps across requests.","Wrap feed/end in a small helper that owns lifecycle."],"tags":["state","lifecycle","streaming"],"backgroundTag":"object-in-invalid-state","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}