{"record":{"id":"4bef37709fdccb61","repo":"koala73/worldmonitor","slug":"http-response-status","errorCode":null,"errorMessage":"HTTP ${response.status}","messagePattern":"HTTP \\$\\{response\\.status\\}","errorType":"exception","errorClass":"Error","httpStatus":502,"severity":"error","filePath":"api/fwdstart.js","lineNumber":25,"sourceCode":"\n// The archive URL is fixed, so every CDN miss was re-scraping the same page.\n// Cache the parsed items — not the rendered RSS, whose lastBuildDate must stay\n// current — for the same window the response already advertises.\nconst CACHE_KEY = 'fwdstart:archive-items:v1';\nconst CACHE_TTL_SECONDS = 1800;\n\n/** Fetch the archive page and extract post items. Throws on upstream failure. */\nasync function scrapeArchiveItems() {\n  const response = await fetch('https://www.fwdstart.me/archive', {\n    headers: {\n      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',\n      'Accept': 'text/html,application/xhtml+xml',\n    },\n    signal: AbortSignal.timeout(15000),\n  });\n\n  if (!response.ok) {\n    throw new Error(`HTTP ${response.status}`);\n  }\n\n  const html = await response.text();\n  const items = [];\n  const seenUrls = new Set();\n\n  // Split by embla__slide to get each post block\n  const slideBlocks = html.split('embla__slide');\n\n  for (const block of slideBlocks) {\n    // Extract URL\n    const urlMatch = block.match(/href=\"(\\/p\\/[^\"]+)\"/);\n    if (!urlMatch) continue;\n\n    const url = `https://www.fwdstart.me${urlMatch[1]}`;\n    if (seenUrls.has(url)) continue;\n    seenUrls.add(url);\n","sourceCodeStart":7,"sourceCodeEnd":43,"githubUrl":"https://github.com/koala73/worldmonitor/blob/ffec79ac339946fd2d24e85845da5755dcaa534b/api/fwdstart.js#L7-L43","documentation":"Thrown by scrapeArchiveItems in api/fwdstart.js when the upstream GET to https://www.fwdstart.me/archive returns a non-2xx status. The thrown message is the literal HTTP status code (e.g. 'HTTP 503'); the outer handler catches it, logs to Sentry via captureSilentError, and returns a 502 'Failed to fetch FwdStart archive' to the caller.","triggerScenarios":"A request to /api/fwdstart (or /api/fwdstart?cache miss) where the upstream fwdstart.me archive page responded with 4xx/5xx within the 15s AbortSignal timeout. A Redis cache miss forces the live scrape, so the error only surfaces when the cached items are absent or expired (TTL 1800s).","commonSituations":"fwdstart.me is temporarily down or rate-limiting the scraper; Cloudflare challenge/WAF blocking the scraper's User-Agent; the upstream changed host and 404s; transient 5xx on the origin.","solutions":["Retry the request after a short wait — upstream blips usually clear within minutes and the cache TTL will hide them once one scrape succeeds.","Check https://www.fwdstart.me/archive directly in a browser to see if the upstream is up or returning a challenge.","If the upstream User-Agent is being challenged, coordinate a scraper UA change with the site owner rather than evading the WAF.","Verify the Vercel Edge function can egress to fwdstart.me (no egress-proxy regression)."],"exampleFix":"// before — upstream returns 503, scrapeArchiveItems throws `HTTP 503`\n//   -> handler returns 502 { error: 'Failed to fetch FwdStart archive' }\n// after — retry once on a transient non-OK status before giving up\nasync function scrapeArchiveItems() {\n  for (let attempt = 0; attempt < 2; attempt++) {\n    const response = await fetch('https://www.fwdstart.me/archive', { /* ... */ });\n    if (response.ok) return parseArchive(await response.text());\n    if (response.status < 500 && response.status !== 429) break;\n    await new Promise(r => setTimeout(r, 500 * (attempt + 1)));\n  }\n  throw new Error('FwdStart archive unavailable');\n}","handlingStrategy":"retry","validationCode":"async function archiveReachable(): Promise<boolean> {\n  try {\n    const r = await fetch('https://www.fwdstart.me/archive', {\n      method: 'HEAD',\n      signal: AbortSignal.timeout(5000),\n    });\n    return r.ok;\n  } catch { return false; }\n}","typeGuard":null,"tryCatchPattern":"// Inside the handler: a Redis cache hit avoids the scrape entirely. On miss,\n// retry the scrape once before returning 502.\nasync function scrapeWithRetry(): Promise<ArchiveItem[]> {\n  let lastErr: unknown;\n  for (let attempt = 0; attempt < 2; attempt++) {\n    try { return await scrapeArchiveItems(); }\n    catch (err) { lastErr = err; await new Promise(r => setTimeout(r, 500)); }\n  }\n  throw lastErr;\n}","preventionTips":["Keep the Redis cache warm (TTL 1800s) so a cache miss — the only path that scrapes — is rare.","Add a single retry-on-5xx around scrapeArchiveItems to absorb upstream blips before the 502.","Monitor Sentry for the route 'api/fwdstart' / step 'scrape' tags; a burst indicates an upstream incident."],"tags":["scraper","network","upstream","rss","retryable"],"backgroundTag":null,"analyzedSha":"ffec79ac339946fd2d24e85845da5755dcaa534b","analyzedAt":"2026-08-12T11:24:56.012Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}