{"record":{"id":"e45907568f31245b","repo":"can1357/oh-my-pi","slug":"failed-to-parse-new-stories","errorCode":null,"errorMessage":"Failed to parse new stories","messagePattern":"Failed to parse new stories","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/coding-agent/src/web/scrapers/hackernews.ts","lineNumber":164,"sourceCode":"\n\t\tif (itemId) {\n\t\t\tconst item = await fetchItem(parseInt(itemId, 10), timeout, signal);\n\t\t\tif (!item) throw new Error(`Failed to fetch item ${itemId}`);\n\n\t\t\tcontent = await renderStory(item, timeout, 0, signal);\n\t\t\tnotes.push(`Fetched HN item ${itemId} with top-level comments (depth 2)`);\n\t\t} else if (parsed.pathname === \"/\" || parsed.pathname === \"/news\") {\n\t\t\tconst { content: raw, ok } = await loadPage(`${API_BASE}/topstories.json`, { timeout, signal });\n\t\t\tif (!ok) throw new Error(\"Failed to fetch top stories\");\n\t\t\tconst ids = tryParseJson<number[]>(raw);\n\t\t\tif (!ids) throw new Error(\"Failed to parse top stories\");\n\t\t\tcontent = await renderListing(ids, timeout, \"Hacker News - Top Stories\", signal);\n\t\t\tnotes.push(\"Fetched top 20 stories from HN front page\");\n\t\t} else if (parsed.pathname === \"/newest\") {\n\t\t\tconst { content: raw, ok } = await loadPage(`${API_BASE}/newstories.json`, { timeout, signal });\n\t\t\tif (!ok) throw new Error(\"Failed to fetch new stories\");\n\t\t\tconst ids = tryParseJson<number[]>(raw);\n\t\t\tif (!ids) throw new Error(\"Failed to parse new stories\");\n\t\t\tcontent = await renderListing(ids, timeout, \"Hacker News - New Stories\", signal);\n\t\t\tnotes.push(\"Fetched top 20 new stories\");\n\t\t} else if (parsed.pathname === \"/best\") {\n\t\t\tconst { content: raw, ok } = await loadPage(`${API_BASE}/beststories.json`, { timeout, signal });\n\t\t\tif (!ok) throw new Error(\"Failed to fetch best stories\");\n\t\t\tconst ids = tryParseJson<number[]>(raw);\n\t\t\tif (!ids) throw new Error(\"Failed to parse best stories\");\n\t\t\tcontent = await renderListing(ids, timeout, \"Hacker News - Best Stories\", signal);\n\t\t\tnotes.push(\"Fetched top 20 best stories\");\n\t\t} else {\n\t\t\treturn null;\n\t\t}\n\n\t\treturn buildResult(content, { url, method: \"hackernews\", fetchedAt, notes });\n\t} catch (err) {\n\t\tconst errorMsg = err instanceof Error ? err.message : String(err);\n\t\tnotes.push(`Error: ${errorMsg}`);\n\t\treturn buildResult(`# Error fetching Hacker News content\\n\\n${errorMsg}`, {","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/coding-agent/src/web/scrapers/hackernews.ts#L146-L182","documentation":"Thrown when the body fetched from `newstories.json` cannot be parsed as a JSON number array by `tryParseJson<number[]>` while handling /newest. The HTTP fetch succeeded (ok:true) but the decoded content is not the expected JSON shape.","triggerScenarios":"GET to https://hacker-news.firebaseio.com/v0/newstories.json returned ok:true but body is HTML (challenge/portal page), empty, garbled by charset issues, or truncated at maxBytes before the JSON array closes.","commonSituations":"Captive portal or corporate proxy returning a 200 HTML page; CDN bot-challenge interstitial; truncated body; encoding mismatch.","solutions":["Log/inspect the raw body (first bytes) to see what was actually returned.","Check for proxies or captive portals intercepting HTTPS with 200 responses.","Retry; transient truncation is the usual cause.","Fall back to Algolia's search_by_date API for new stories.","Verify Accept-Encoding handling — a proxy that ignores the identity encoding request can return bytes the decoder mangles."],"exampleFix":"// before\nconst ids = tryParseJson<number[]>(raw);\nif (!ids) throw new Error(\"Failed to parse new stories\");\n// after\nconst ids = tryParseJson<number[]>(raw);\nif (!ids) throw new Error(`newstories.json non-JSON body (${raw.length} bytes), head: ${raw.slice(0, 120)}`);","handlingStrategy":"validation","validationCode":"const ids = tryParseJson<number[]>(raw);\nif (!Array.isArray(ids) || ids.some(id => typeof id !== \"number\")) {\n  throw new Error(`unexpected newstories payload: ${String(raw).slice(0, 120)}`);\n}","typeGuard":"function isIdArray(v: unknown): v is number[] {\n  return Array.isArray(v) && v.every(id => typeof id === \"number\");\n}","tryCatchPattern":"try {\n  const ids = tryParseJson<number[]>(raw);\n  if (!ids) throw new Error(`non-JSON body: ${raw.slice(0, 120)}`);\n} catch (err) {\n  logger.warn(\"newstories.json body not JSON — inspect for proxy/challenge page\", { head: raw.slice(0, 200) });\n}","preventionTips":["Log the unparseable body's head to detect injected HTML pages.","Watch for captive portals on managed/corporate networks.","Ensure Accept-Encoding: identity survives any proxy in the path.","Compare content-length vs received bytes for truncation.","Fall back to Algolia when Firebase payloads are consistently malformed."],"tags":["json","parsing","hacker-news","proxy"],"backgroundTag":"invalid-json-response","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}