{"record":{"id":"669ed9b41e37939d","repo":"can1357/oh-my-pi","slug":"failed-to-parse-top-stories","errorCode":null,"errorMessage":"Failed to parse top stories","messagePattern":"Failed to parse top stories","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/coding-agent/src/web/scrapers/hackernews.ts","lineNumber":157,"sourceCode":"\n\tconst notes: string[] = [];\n\tlet content = \"\";\n\tconst fetchedAt = new Date().toISOString();\n\n\ttry {\n\t\tconst itemId = parsed.searchParams.get(\"id\");\n\n\t\tif (itemId) {\n\t\t\tconst item = await fetchItem(parseInt(itemId, 10), timeout, signal);\n\t\t\tif (!item) throw new Error(`Failed to fetch item ${itemId}`);\n\n\t\t\tcontent = await renderStory(item, timeout, 0, signal);\n\t\t\tnotes.push(`Fetched HN item ${itemId} with top-level comments (depth 2)`);\n\t\t} else if (parsed.pathname === \"/\" || parsed.pathname === \"/news\") {\n\t\t\tconst { content: raw, ok } = await loadPage(`${API_BASE}/topstories.json`, { timeout, signal });\n\t\t\tif (!ok) throw new Error(\"Failed to fetch top stories\");\n\t\t\tconst ids = tryParseJson<number[]>(raw);\n\t\t\tif (!ids) throw new Error(\"Failed to parse top stories\");\n\t\t\tcontent = await renderListing(ids, timeout, \"Hacker News - Top Stories\", signal);\n\t\t\tnotes.push(\"Fetched top 20 stories from HN front page\");\n\t\t} else if (parsed.pathname === \"/newest\") {\n\t\t\tconst { content: raw, ok } = await loadPage(`${API_BASE}/newstories.json`, { timeout, signal });\n\t\t\tif (!ok) throw new Error(\"Failed to fetch new stories\");\n\t\t\tconst ids = tryParseJson<number[]>(raw);\n\t\t\tif (!ids) throw new Error(\"Failed to parse new stories\");\n\t\t\tcontent = await renderListing(ids, timeout, \"Hacker News - New Stories\", signal);\n\t\t\tnotes.push(\"Fetched top 20 new stories\");\n\t\t} else if (parsed.pathname === \"/best\") {\n\t\t\tconst { content: raw, ok } = await loadPage(`${API_BASE}/beststories.json`, { timeout, signal });\n\t\t\tif (!ok) throw new Error(\"Failed to fetch best stories\");\n\t\t\tconst ids = tryParseJson<number[]>(raw);\n\t\t\tif (!ids) throw new Error(\"Failed to parse best stories\");\n\t\t\tcontent = await renderListing(ids, timeout, \"Hacker News - Best Stories\", signal);\n\t\t\tnotes.push(\"Fetched top 20 best stories\");\n\t\t} else {\n\t\t\treturn null;","sourceCodeStart":139,"sourceCodeEnd":175,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/coding-agent/src/web/scrapers/hackernews.ts#L139-L175","documentation":"Thrown when the body fetched from `topstories.json` cannot be parsed as a JSON array of story ids by `tryParseJson<number[]>`. Unlike fetch failure, this means a response was received but its content is not the expected shape — typically an error page, HTML interstitial, or truncated body (e.g. cut at maxBytes, or a bot-block challenge page served with 200).","triggerScenarios":"GET to https://hacker-news.firebaseio.com/v0/topstories.json returned ok:true but the decoded text is not valid JSON or not an array — HTML challenge page, empty body, compressed bytes misdecoded, or maxBytes truncation mid-payload.","commonSituations":"Captive portal or proxy injecting an HTML login page with 200 OK; CDN serving a Cloudflare challenge; body truncated by maxBytes limit; charset/encoding mismatch garbling the payload.","solutions":["Inspect the raw body returned in the error result notes to see what actually came back (HTML? empty? truncated JSON?).","Check for a proxy/captive portal intercepting HTTPS traffic with a 200 HTML page.","Retry — a mid-stream truncation is usually transient.","Raise the loadPage maxBytes option if the listing payload legitimately exceeds 50 MB is unlikely here; instead check network MTU/proxy buffering issues.","Fall back to the Algolia HN API which returns self-describing JSON."],"exampleFix":"// before: assume parse failure is transient\nconst ids = tryParseJson<number[]>(raw);\nif (!ids) throw new Error(\"Failed to parse top stories\");\n// after: fail with diagnostic context\nconst ids = tryParseJson<number[]>(raw);\nif (!ids) throw new Error(`topstories.json returned non-JSON body (${raw.length} bytes): ${raw.slice(0, 120)}`);","handlingStrategy":"validation","validationCode":"const ids = tryParseJson<number[]>(raw);\nif (!Array.isArray(ids) || ids.some(id => typeof id !== \"number\")) {\n  throw new Error(`unexpected topstories payload: ${String(raw).slice(0, 120)}`);\n}","typeGuard":"function isIdArray(v: unknown): v is number[] {\n  return Array.isArray(v) && v.every(id => typeof id === \"number\");\n}","tryCatchPattern":"try {\n  const ids = tryParseJson<number[]>(raw);\n  if (!ids) throw new Error(`non-JSON body: ${raw.slice(0, 120)}`);\n} catch (err) {\n  logger.warn(\"topstories.json body not JSON — likely proxy/challenge page\", { head: raw.slice(0, 200) });\n}","preventionTips":["Log the first bytes of any unparseable body — HTML reveals proxy/portal interception instantly.","Check corporate proxies/captive portals when in managed networks.","Keep Accept-Encoding: identity so decompression mismatches can't corrupt bodies.","Fall back to Algolia when Firebase payloads are malformed repeatedly.","Compare content-length vs received bytes to catch truncation."],"tags":["json","parsing","hacker-news","proxy"],"backgroundTag":"invalid-json-response","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}