can1357/oh-my-pi · error · SyntaxError

Unexpected token at position ${this.#i}

Error message

Unexpected token at position ${this.#i}

What it means

This SyntaxError is thrown by the streaming JSON parser's #keyword routine when it encounters an unrecognized atomic token at the current position. Unlike incomplete-token conditions (which return INCOMPLETE so the caller can roll back), this is a definitive syntax failure — the input contains a character that cannot start any JSON value, keyword, or recoverable bareword. The parser is designed for incremental/streaming parsing, so 'position' refers to the character offset within the chunk being parsed.

Source

Thrown at packages/utils/src/json-parse.ts:504

	#keyword(allowBareword: boolean): unknown {
		const s = this.#s;
		const i = this.#i;
		for (const [word, value] of KEYWORDS) {
			// Require a non-identifier boundary so `Truex` / `nullish` are not misread
			// as the keyword followed by junk.
			if (s.startsWith(word, i) && !isIdentChar(s.charCodeAt(i + word.length))) {
				this.#i += word.length;
				return value;
			}
		}
		if (this.#partial) {
			// Incomplete / unrecognized atomic token at the streaming edge — signal the
			// caller to roll back to the last valid prefix instead of committing junk.
			this.#i = this.#n;
			return INCOMPLETE;
		}
		if (allowBareword) return this.#bareword();
		throw new SyntaxError(`Unexpected token at position ${this.#i}`);
	}

	/**
	 * Strict-mode recovery of an unquoted string value, e.g.
	 * `{"paths": packages/foo/*}`: consume until `,` / `}` / `]` / newline and
	 * trim trailing whitespace. Recovery still throws — so a final parse never
	 * accepts a half-formed or non-finite argument — when the token:
	 * - hits end-of-input before a delimiter (truncated value);
	 * - contains a `"`, `{`, `[`, or a key-like `:` — this parser accepts
	 *   unquoted keys, so a missed comma (`{"a": foo "b": 1}`, `{a: foo b: 1}`)
	 *   would otherwise silently swallow the following field. A colon followed
	 *   by `/` or `\` stays literal so URL and Windows-path values recover;
	 * - is a non-finite atom ({@link NON_RECOVERABLE_BAREWORDS}).
	 */
	#bareword(): string {
		const s = this.#s;
		const start = this.#i;
		let i = start;

View on GitHub (pinned to 9690622007)

Solutions

1. Inspect the input at the reported position and remove/fix the offending character (print the substring around `position`).
2. Ensure the text is valid JSON before parsing (validate with JSON.parse or a linter).
3. If input may arrive incrementally, make sure the parser's INCOMPLETE return is handled by rolling back to the last valid prefix instead of treating it as fatal.
4. If the value was meant to be an unquoted string (e.g. {"paths": packages/foo/*}), quote it properly or rely on strict-mode recovery only for supported bareword shapes.
5. Strip BOMs, log prefixes, or markdown fences before parsing.

Example fix

// before
parse('{"a": ?}'); // SyntaxError: Unexpected token at position 6
// after
parse('{"a": null}'); // OK — or handle INCOMPLETE and wait for more input in streaming mode

// quick sanity check before parsing
typeof input === "string" || throw new TypeError("input must be a string");
const first = input.trimStart()[0];
if (first !== undefined && !'{["'.includes(first) && !/[\-0-9tfn]/.test(first)) {
  throw new SyntaxError(`Input does not start with a JSON token: ${JSON.stringify(first)}`);
}

function looksLikeJsonText(s: unknown): s is string {
  return typeof s === "string" && /^[\s]*[\[{"\-0-9tfn]/.test(s);
}

try {
  return parser.feed(chunk);
} catch (err) {
  if (err instanceof SyntaxError && err.message.startsWith("Unexpected token at position")) {
    const pos = Number(err.message.match(/position (\d+)/)?.[1]);
    logger.warn("JSON syntax error", { pos, context: chunk.slice(Math.max(0, pos - 20), pos + 20) });
    return rollBackToLastValidPrefix();
  }
  throw err;
}

When it happens

Trigger: Calling the parser with input whose next token starts with a character that is not a quote, digit, '{', '[', a recognized keyword letter, or a valid bareword start while parsing a value (via #value). E.g. feed '{"a": ?}' or '{"a": +x}' — '?'/'+' cannot begin a token, so #keyword throws immediately instead of returning INCOMPLETE.

Common situations: Truncated LLM/CLI output containing stray characters, hand-edited JSON with typos (unquoted punctuation, stray commas followed by garbage), binary bytes or BOM/preamble before JSON, or users trying to parse pseudo-JSON like {'a': undefined} or JSON5 syntax the parser does not support.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• V2 compaction stream parse failed: ${err instanceof Error ?
• Unexpected end of JSON input
• Replacement text is not valid UTF-8: {err}
• Validation failed for tool "${toolCall.name}": Tool call arg
• Gemini Files API ${context} response is not valid JSON

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/3dc4306573ba049c. Report an issue: GitHub.