{"record":{"id":"87ec0a28e8f605e4","repo":"koala73/worldmonitor","slug":"firecrawl-scrape-failed-http-resp-status","errorCode":null,"errorMessage":"Firecrawl scrape failed: HTTP ${resp.status}","messagePattern":"Firecrawl scrape failed: HTTP (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"consumer-prices-core/src/acquisition/firecrawl.ts","lineNumber":75,"sourceCode":"      'User-Agent': 'worldmonitor-consumer-prices/1.0',\n    };\n  }\n\n  async fetch(url: string, opts: FetchOptions = {}): Promise<FetchResult> {\n    const resp = await fetch(`${this.baseUrl}/scrape`, {\n      method: 'POST',\n      headers: this.headers(),\n      body: JSON.stringify({\n        url,\n        formats: ['html', 'markdown'],\n        waitFor: opts.waitForSelector ? 2000 : 0,\n        timeout: opts.timeout ?? 30_000,\n        headers: opts.headers,\n      }),\n      signal: AbortSignal.timeout((opts.timeout ?? 30_000) + 5_000),\n    });\n\n    if (!resp.ok) throw new Error(`Firecrawl scrape failed: HTTP ${resp.status}`);\n\n    const data = (await resp.json()) as FirecrawlScrapeResponse;\n    if (!data.success || !data.data) {\n      throw new Error(`Firecrawl error: ${data.error ?? 'unknown'}`);\n    }\n\n    return {\n      url,\n      html: data.data.html ?? '',\n      markdown: data.data.markdown ?? '',\n      statusCode: 200,\n      provider: this.name,\n      fetchedAt: new Date(),\n      metadata: data.data.metadata,\n    };\n  }\n\n  async search(query: string, opts: SearchOptions = {}): Promise<SearchResult[]> {","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/koala73/worldmonitor/blob/eeab0a219fce0f02a00603b532dbae9041b934ac/consumer-prices-core/src/acquisition/firecrawl.ts#L57-L93","documentation":"FirecrawlProvider.fetch POSTs to /v1/scrape and throws on any non-2xx before parsing the body. The common causes map to status codes: 401 bad or missing API key in headers(), 402 credits exhausted, and timeout-class statuses when a heavy page exceeds the (timeout + 5s) abort budget.","triggerScenarios":"Invalid or unset Firecrawl API key; out-of-credit account; slow SPA storefront exceeding opts.timeout (default 30s) plus the 5s abort margin; a malformed target URL rejected by the API.","commonSituations":"Trial keys expiring mid-scrape; quota drained by a large batch; JavaScript-heavy product pages needing more than 30s to render.","solutions":["Verify the Firecrawl API key and that credits remain (dashboard billing)","Raise opts.timeout and consider waitForSelector for render-heavy pages","Confirm the URL is well-formed and publicly reachable","On persistent 5xx, check the Firecrawl status page and retry later"],"exampleFix":"// before\nawait firecrawl.fetch(url);                       // HTTP 408/504 on a heavy storefront\n\n// after — give the render a real budget\nawait firecrawl.fetch(url, { timeout: 60_000, waitForSelector: '.price' });","handlingStrategy":"try-catch","validationCode":"// Preflight key and give heavy pages a real budget\nif (!process.env.FIRECRAWL_API_KEY) throw new Error('FIRECRAWL_API_KEY not set');\nconst opts = isHeavyStorefront(url) ? { timeout: 60_000, waitForSelector: '.price' } : {};","typeGuard":null,"tryCatchPattern":"try {\n  return await firecrawl.fetch(url, opts);\n} catch (err) {\n  const m = /HTTP (\\d+)/.exec(err.message);\n  const status = m ? Number(m[1]) : 0;\n  if (status === 401 || status === 402) throw new ConfigError(err.message);   // key/credits\n  if (status === 408 || status >= 500) return retryWithBackoff(() => firecrawl.fetch(url, { ...opts, timeout: (opts.timeout ?? 30_000) * 2 }));\n  throw err;\n}","preventionTips":["Monitor credit balance before large scrape batches","Size opts.timeout to the page class (SPA storefronts need more than 30s)","Track per-domain scrape failure rates to catch blocks early"],"tags":["firecrawl","http","scraping","api-errors"],"backgroundTag":"http-request-failed","analyzedSha":"eeab0a219fce0f02a00603b532dbae9041b934ac","analyzedAt":"2026-08-21T16:51:25.751Z","schemaVersion":2},"datasetVersion":"2026-08-23T11:17:13.642Z"}