can1357/oh-my-pi · error

Failed to parse top stories

Error message

Failed to parse top stories

What it means

Thrown when the body fetched from `topstories.json` cannot be parsed as a JSON array of story ids by `tryParseJson<number[]>`. Unlike fetch failure, this means a response was received but its content is not the expected shape — typically an error page, HTML interstitial, or truncated body (e.g. cut at maxBytes, or a bot-block challenge page served with 200).

Source

Thrown at packages/coding-agent/src/web/scrapers/hackernews.ts:157

	const notes: string[] = [];
	let content = "";
	const fetchedAt = new Date().toISOString();

	try {
		const itemId = parsed.searchParams.get("id");

		if (itemId) {
			const item = await fetchItem(parseInt(itemId, 10), timeout, signal);
			if (!item) throw new Error(`Failed to fetch item ${itemId}`);

			content = await renderStory(item, timeout, 0, signal);
			notes.push(`Fetched HN item ${itemId} with top-level comments (depth 2)`);
		} else if (parsed.pathname === "/" || parsed.pathname === "/news") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/topstories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch top stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse top stories");
			content = await renderListing(ids, timeout, "Hacker News - Top Stories", signal);
			notes.push("Fetched top 20 stories from HN front page");
		} else if (parsed.pathname === "/newest") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/newstories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch new stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse new stories");
			content = await renderListing(ids, timeout, "Hacker News - New Stories", signal);
			notes.push("Fetched top 20 new stories");
		} else if (parsed.pathname === "/best") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/beststories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch best stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse best stories");
			content = await renderListing(ids, timeout, "Hacker News - Best Stories", signal);
			notes.push("Fetched top 20 best stories");
		} else {
			return null;

View on GitHub (pinned to 9690622007)

Solutions

1. Inspect the raw body returned in the error result notes to see what actually came back (HTML? empty? truncated JSON?).
2. Check for a proxy/captive portal intercepting HTTPS traffic with a 200 HTML page.
3. Retry — a mid-stream truncation is usually transient.
4. Raise the loadPage maxBytes option if the listing payload legitimately exceeds 50 MB is unlikely here; instead check network MTU/proxy buffering issues.
5. Fall back to the Algolia HN API which returns self-describing JSON.

Example fix

// before: assume parse failure is transient
const ids = tryParseJson<number[]>(raw);
if (!ids) throw new Error("Failed to parse top stories");
// after: fail with diagnostic context
const ids = tryParseJson<number[]>(raw);
if (!ids) throw new Error(`topstories.json returned non-JSON body (${raw.length} bytes): ${raw.slice(0, 120)}`);

const ids = tryParseJson<number[]>(raw);
if (!Array.isArray(ids) || ids.some(id => typeof id !== "number")) {
  throw new Error(`unexpected topstories payload: ${String(raw).slice(0, 120)}`);
}

function isIdArray(v: unknown): v is number[] {
  return Array.isArray(v) && v.every(id => typeof id === "number");
}

try {
  const ids = tryParseJson<number[]>(raw);
  if (!ids) throw new Error(`non-JSON body: ${raw.slice(0, 120)}`);
} catch (err) {
  logger.warn("topstories.json body not JSON — likely proxy/challenge page", { head: raw.slice(0, 200) });
}

When it happens

Trigger: GET to https://hacker-news.firebaseio.com/v0/topstories.json returned ok:true but the decoded text is not valid JSON or not an array — HTML challenge page, empty body, compressed bytes misdecoded, or maxBytes truncation mid-payload.

Common situations: Captive portal or proxy injecting an HTML login page with 200 OK; CDN serving a Cloudflare challenge; body truncated by maxBytes limit; charset/encoding mismatch garbling the payload.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Failed to parse new stories
• Failed to parse best stories
• Replacement text is not valid UTF-8: {err}
• V2 compaction stream parse failed: ${err instanceof Error ?
• xAI device-code response returned invalid JSON: ${error inst

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/669ed9b41e37939d. Report an issue: GitHub.