{"record":{"id":"0f2897f8639b34e6","repo":"koala73/worldmonitor","slug":"firecrawl-extract-failed-http-resp-status","errorCode":null,"errorMessage":"Firecrawl extract failed: HTTP ${resp.status}","messagePattern":"Firecrawl extract failed: HTTP (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"consumer-prices-core/src/acquisition/firecrawl.ts","lineNumber":157,"sourceCode":"\n    const resp = await fetch(`${this.baseUrl}/scrape`, {\n      method: 'POST',\n      headers: this.headers(),\n      body: JSON.stringify({\n        url,\n        // markdown rides along in the same render so the caller can verify an\n        // extracted price actually appears on the page (price-evidence.ts).\n        formats: ['extract', 'markdown'],\n        extract: { schema: jsonSchema, ...(schema.prompt ? { prompt: schema.prompt } : {}) },\n        timeout: opts.timeout ?? 30_000,\n        // Late-hydrating storefronts capture as a breadcrumb shell without a\n        // settle delay; the abort deadline below must absorb it too.\n        ...(opts.waitFor ? { waitFor: opts.waitFor } : {}),\n      }),\n      signal: AbortSignal.timeout(extractAbortMs(opts.timeout) + (opts.waitFor ?? 0)),\n    });\n\n    if (!resp.ok) throw new Error(`Firecrawl extract failed: HTTP ${resp.status}`);\n\n    const data = (await resp.json()) as FirecrawlExtractResponse;\n    // Throw ONLY for a provider-side failure (quota exhausted, rate limited,\n    // bad request) — those are transport conditions the caller's cooldown\n    // should count. A successful call that simply found nothing to extract is\n    // a PAGE-level outcome: return empty so the caller records `missing-price`\n    // and moves to the next candidate URL without accruing an outage streak.\n    // Conflating the two lets two ordinary no-product pages disable Firecrawl\n    // for the rest of the scrape, on every retailer, not just opted-in ones.\n    if (!data.success) {\n      throw new Error(`Firecrawl extract error: ${data.error ?? 'unknown'}`);\n    }\n\n    return {\n      url,\n      data: (data.data?.extract ?? {}) as T,\n      provider: this.name,\n      fetchedAt: new Date(),","sourceCodeStart":139,"sourceCodeEnd":175,"githubUrl":"https://github.com/koala73/worldmonitor/blob/eeab0a219fce0f02a00603b532dbae9041b934ac/consumer-prices-core/src/acquisition/firecrawl.ts#L139-L175","documentation":"FirecrawlProvider.extract POSTs to /v1/scrape with formats ['extract','markdown'] plus a JSON schema, and throws on any non-2xx with the status. The abort budget deliberately adds opts.waitFor on top of extractAbortMs(timeout) because late-hydrating storefronts need the settle delay — so slow extractions surface here as timeout-class statuses, distinct from the in-band `Firecrawl extract error` reserved for provider-side failures (an extract that simply finds nothing returns empty instead).","triggerScenarios":"408/504 when extraction exceeds the timeout+waitFor abort budget on slow storefronts; 401/402 key or credit problems; 400 for a schema the extract validator rejects; provider 5xx incidents.","commonSituations":"Late-hydrating SPA product pages where the price appears seconds after load; complex extraction schemas; quota exhausted mid-run.","solutions":["Raise opts.timeout and set opts.waitFor so the budget covers storefront hydration","Verify key and credits on 401/402","Simplify the extract schema if the validator rejects it (400)","Classify correctly: this is a transport failure for cooldown counting; a page with nothing to extract is not an error"],"exampleFix":"// before — budget too small for a late-hydrating storefront\nawait firecrawl.extract(url, schema, { timeout: 15_000 });   // HTTP 408/504\n\n// after — extend both knobs; the abort deadline absorbs waitFor\nawait firecrawl.extract(url, schema, { timeout: 45_000, waitFor: 3_000 });","handlingStrategy":"retry","validationCode":"// Budget the call up front: abort deadline = extractAbortMs(timeout) + waitFor\nconst opts = { timeout: slowStorefront(url) ? 45_000 : 30_000, waitFor: slowStorefront(url) ? 3_000 : 0 };","typeGuard":null,"tryCatchPattern":"try {\n  return await firecrawl.extract(url, schema, opts);\n} catch (err) {\n  const m = /HTTP (\\d+)/.exec(err.message);\n  const status = m ? Number(m[1]) : 0;\n  if (status === 408 || status === 504) {\n    return firecrawl.extract(url, schema, { ...opts, timeout: (opts.timeout ?? 30_000) * 2, waitFor: (opts.waitFor ?? 0) + 2000 });\n  }\n  throw err;   // 401/402/400 are not timeout-budget issues\n}","preventionTips":["Set opts.waitFor for late-hydrating storefronts so the render settles inside the budget","Scale timeout to page weight instead of using the 30s default everywhere","Remember the contract: transport failures throw here; a page with nothing to extract returns empty and must not count against provider cooldowns"],"tags":["firecrawl","extraction","http","timeout"],"backgroundTag":"http-request-failed","analyzedSha":"eeab0a219fce0f02a00603b532dbae9041b934ac","analyzedAt":"2026-08-21T16:51:25.751Z","schemaVersion":2},"datasetVersion":"2026-08-23T11:17:13.642Z"}