What causes "Failed to parse best stories" in can1357/oh-my-pi?

Inspect the raw body in diagnostics to identify what was actually returned.

Failed to parse best stories

Error message

Failed to parse best stories

What it means

Thrown when the body fetched from `beststories.json` cannot be parsed as a JSON number array by `tryParseJson<number[]>` while handling /best. The fetch reported ok:true but the decoded content is not the expected JSON — typically an injected HTML page or truncated payload.

Source

Thrown at packages/coding-agent/src/web/scrapers/hackernews.ts:171

		} else if (parsed.pathname === "/" || parsed.pathname === "/news") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/topstories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch top stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse top stories");
			content = await renderListing(ids, timeout, "Hacker News - Top Stories", signal);
			notes.push("Fetched top 20 stories from HN front page");
		} else if (parsed.pathname === "/newest") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/newstories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch new stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse new stories");
			content = await renderListing(ids, timeout, "Hacker News - New Stories", signal);
			notes.push("Fetched top 20 new stories");
		} else if (parsed.pathname === "/best") {
			const { content: raw, ok } = await loadPage(`${API_BASE}/beststories.json`, { timeout, signal });
			if (!ok) throw new Error("Failed to fetch best stories");
			const ids = tryParseJson<number[]>(raw);
			if (!ids) throw new Error("Failed to parse best stories");
			content = await renderListing(ids, timeout, "Hacker News - Best Stories", signal);
			notes.push("Fetched top 20 best stories");
		} else {
			return null;
		}

		return buildResult(content, { url, method: "hackernews", fetchedAt, notes });
	} catch (err) {
		const errorMsg = err instanceof Error ? err.message : String(err);
		notes.push(`Error: ${errorMsg}`);
		return buildResult(`# Error fetching Hacker News content\n\n${errorMsg}`, {
			url,
			method: "hackernews",
			fetchedAt,
			notes,
		});
	}
};

View on GitHub (pinned to 9690622007)

Solutions

Inspect the raw body in diagnostics to identify what was actually returned.
Check for proxies/captive portals intercepting HTTPS with 200 responses.
Retry; transient truncation is most common.
Fall back to Algolia HN API.
Confirm no middleware strips or rewrites the response body.

Example fix

// before
const ids = tryParseJson<number[]>(raw);
if (!ids) throw new Error("Failed to parse best stories");
// after
const ids = tryParseJson<number[]>(raw);
if (!ids) throw new Error(`beststories.json non-JSON body (${raw.length} bytes), head: ${raw.slice(0, 120)}`);

Defensive patterns

Strategy: validation

Validate before calling

const ids = tryParseJson<number[]>(raw);
if (!Array.isArray(ids) || ids.some(id => typeof id !== "number")) {
  throw new Error(`unexpected beststories payload: ${String(raw).slice(0, 120)}`);
}

Type guard

function isIdArray(v: unknown): v is number[] {
  return Array.isArray(v) && v.every(id => typeof id === "number");
}

Try / catch

try {
  const ids = tryParseJson<number[]>(raw);
  if (!ids) throw new Error(`non-JSON body: ${raw.slice(0, 120)}`);
} catch (err) {
  logger.warn("beststories.json body not JSON — inspect for proxy/challenge page", { head: raw.slice(0, 200) });
}

Prevention

Log the body head on parse failure to identify HTML injection.
Check proxy middleware that rewrites response bodies.
Keep identity content-encoding to avoid decode corruption.
Compare content-length vs received bytes to catch truncation.
Fall back to Algolia if Firebase keeps returning malformed bodies.

When it happens

Trigger: GET to https://hacker-news.firebaseio.com/v0/beststories.json returned ok:true but body is not valid JSON — challenge page, captive portal, empty body, or truncation at maxBytes mid-array.

Common situations: Corporate proxy returning 200 HTML; Cloudflare/CDN interstitial; truncated body; charset mismatch garbling bytes.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/426f9fe62cc70311. Report an issue: GitHub.