can1357/oh-my-pi · error

Failed to parse new stories

Error message

Failed to parse new stories

What it means

Thrown when the body fetched from `newstories.json` cannot be parsed as a JSON number array by `tryParseJson<number[]>` while handling /newest. The HTTP fetch succeeded (ok:true) but the decoded content is not the expected JSON shape.

Source

Thrown at packages/coding-agent/src/web/scrapers/hackernews.ts:164

		if (itemId) {
			const item = await fetchItem(parseInt(itemId, 10), timeout, signal);
			if (!item) throw new Error(`Failed to fetch item ${itemId}`);

			content = await renderStory(item, timeout, 0, signal);
			notes.push(`Fetched HN item ${itemId} with top-level comments (depth 2)`);
		} else if (parsed.pathname === "/" || parsed.pathname === "/news") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/topstories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch top stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse top stories");
			content = await renderListing(ids, timeout, "Hacker News - Top Stories", signal);
			notes.push("Fetched top 20 stories from HN front page");
		} else if (parsed.pathname === "/newest") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/newstories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch new stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse new stories");
			content = await renderListing(ids, timeout, "Hacker News - New Stories", signal);
			notes.push("Fetched top 20 new stories");
		} else if (parsed.pathname === "/best") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/beststories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch best stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse best stories");
			content = await renderListing(ids, timeout, "Hacker News - Best Stories", signal);
			notes.push("Fetched top 20 best stories");
		} else {
			return null;
		}

		return buildResult(content, { url, method: "hackernews", fetchedAt, notes });
	} catch (err) {
		const errorMsg = err instanceof Error ? err.message : String(err);
		notes.push(`Error: ${errorMsg}`);
		return buildResult(`# Error fetching Hacker News content\n\n${errorMsg}`, {

View on GitHub (pinned to 9690622007)

Solutions

1. Log/inspect the raw body (first bytes) to see what was actually returned.
2. Check for proxies or captive portals intercepting HTTPS with 200 responses.
3. Retry; transient truncation is the usual cause.
4. Fall back to Algolia's search_by_date API for new stories.
5. Verify Accept-Encoding handling — a proxy that ignores the identity encoding request can return bytes the decoder mangles.

Example fix

// before
const ids = tryParseJson<number[]>(raw);
if (!ids) throw new Error("Failed to parse new stories");
// after
const ids = tryParseJson<number[]>(raw);
if (!ids) throw new Error(`newstories.json non-JSON body (${raw.length} bytes), head: ${raw.slice(0, 120)}`);

const ids = tryParseJson<number[]>(raw);
if (!Array.isArray(ids) || ids.some(id => typeof id !== "number")) {
  throw new Error(`unexpected newstories payload: ${String(raw).slice(0, 120)}`);
}

function isIdArray(v: unknown): v is number[] {
  return Array.isArray(v) && v.every(id => typeof id === "number");
}

try {
  const ids = tryParseJson<number[]>(raw);
  if (!ids) throw new Error(`non-JSON body: ${raw.slice(0, 120)}`);
} catch (err) {
  logger.warn("newstories.json body not JSON — inspect for proxy/challenge page", { head: raw.slice(0, 200) });
}

When it happens

Trigger: GET to https://hacker-news.firebaseio.com/v0/newstories.json returned ok:true but body is HTML (challenge/portal page), empty, garbled by charset issues, or truncated at maxBytes before the JSON array closes.

Common situations: Captive portal or corporate proxy returning a 200 HTML page; CDN bot-challenge interstitial; truncated body; encoding mismatch.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Failed to parse top stories
• Failed to parse best stories
• Replacement text is not valid UTF-8: {err}
• V2 compaction stream parse failed: ${err instanceof Error ?
• xAI device-code response returned invalid JSON: ${error inst

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/e45907568f31245b. Report an issue: GitHub.