koala73/worldmonitor · error · Error

HTTP ${response.status}

Error message

HTTP ${response.status}

What it means

Thrown by scrapeArchiveItems in api/fwdstart.js when the upstream GET to https://www.fwdstart.me/archive returns a non-2xx status. The thrown message is the literal HTTP status code (e.g. 'HTTP 503'); the outer handler catches it, logs to Sentry via captureSilentError, and returns a 502 'Failed to fetch FwdStart archive' to the caller.

Source

Thrown at api/fwdstart.js:25

// The archive URL is fixed, so every CDN miss was re-scraping the same page.
// Cache the parsed items — not the rendered RSS, whose lastBuildDate must stay
// current — for the same window the response already advertises.
const CACHE_KEY = 'fwdstart:archive-items:v1';
const CACHE_TTL_SECONDS = 1800;

/** Fetch the archive page and extract post items. Throws on upstream failure. */
async function scrapeArchiveItems() {
  const response = await fetch('https://www.fwdstart.me/archive', {
    headers: {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
      'Accept': 'text/html,application/xhtml+xml',
    },
    signal: AbortSignal.timeout(15000),
  });

  if (!response.ok) {
    throw new Error(`HTTP ${response.status}`);
  }

  const html = await response.text();
  const items = [];
  const seenUrls = new Set();

  // Split by embla__slide to get each post block
  const slideBlocks = html.split('embla__slide');

  for (const block of slideBlocks) {
    // Extract URL
    const urlMatch = block.match(/href="(\/p\/[^"]+)"/);
    if (!urlMatch) continue;

    const url = `https://www.fwdstart.me${urlMatch[1]}`;
    if (seenUrls.has(url)) continue;
    seenUrls.add(url);

View on GitHub (pinned to ffec79ac33)

Solutions

1. Retry the request after a short wait — upstream blips usually clear within minutes and the cache TTL will hide them once one scrape succeeds.
2. Check https://www.fwdstart.me/archive directly in a browser to see if the upstream is up or returning a challenge.
3. If the upstream User-Agent is being challenged, coordinate a scraper UA change with the site owner rather than evading the WAF.
4. Verify the Vercel Edge function can egress to fwdstart.me (no egress-proxy regression).

Example fix

// before — upstream returns 503, scrapeArchiveItems throws `HTTP 503`
//   -> handler returns 502 { error: 'Failed to fetch FwdStart archive' }
// after — retry once on a transient non-OK status before giving up
async function scrapeArchiveItems() {
  for (let attempt = 0; attempt < 2; attempt++) {
    const response = await fetch('https://www.fwdstart.me/archive', { /* ... */ });
    if (response.ok) return parseArchive(await response.text());
    if (response.status < 500 && response.status !== 429) break;
    await new Promise(r => setTimeout(r, 500 * (attempt + 1)));
  }
  throw new Error('FwdStart archive unavailable');
}

async function archiveReachable(): Promise<boolean> {
  try {
    const r = await fetch('https://www.fwdstart.me/archive', {
      method: 'HEAD',
      signal: AbortSignal.timeout(5000),
    });
    return r.ok;
  } catch { return false; }
}

// Inside the handler: a Redis cache hit avoids the scrape entirely. On miss,
// retry the scrape once before returning 502.
async function scrapeWithRetry(): Promise<ArchiveItem[]> {
  let lastErr: unknown;
  for (let attempt = 0; attempt < 2; attempt++) {
    try { return await scrapeArchiveItems(); }
    catch (err) { lastErr = err; await new Promise(r => setTimeout(r, 500)); }
  }
  throw lastErr;
}

When it happens

Trigger: A request to /api/fwdstart (or /api/fwdstart?cache miss) where the upstream fwdstart.me archive page responded with 4xx/5xx within the 15s AbortSignal timeout. A Redis cache miss forces the live scrape, so the error only surfaces when the cached items are absent or expired (TTL 1800s).

Common situations: fwdstart.me is temporarily down or rate-limiting the scraper; Cloudflare challenge/WAF blocking the scraper's User-Agent; the upstream changed host and 404s; transient 5xx on the origin.

Related errors

• Webhook URL DNS resolution failed: ${message}
• Redis request failed
• Redis snapshot lock retry failed
• serverUrl DNS resolution failed: ${message}
• No result found in SSE response

AI-assisted analysis of koala73/worldmonitor@ffec79ac33 (2026-08-12). Data as JSON: /api/errors/4bef37709fdccb61. Report an issue: GitHub.