{"record":{"id":"311f8b539ee88617","repo":"koala73/worldmonitor","slug":"firecrawl-extract-error-data-error-unknown","errorCode":null,"errorMessage":"Firecrawl extract error: ${data.error ?? 'unknown'}","messagePattern":"Firecrawl extract error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"consumer-prices-core/src/acquisition/firecrawl.ts","lineNumber":168,"sourceCode":"        // Late-hydrating storefronts capture as a breadcrumb shell without a\n        // settle delay; the abort deadline below must absorb it too.\n        ...(opts.waitFor ? { waitFor: opts.waitFor } : {}),\n      }),\n      signal: AbortSignal.timeout(extractAbortMs(opts.timeout) + (opts.waitFor ?? 0)),\n    });\n\n    if (!resp.ok) throw new Error(`Firecrawl extract failed: HTTP ${resp.status}`);\n\n    const data = (await resp.json()) as FirecrawlExtractResponse;\n    // Throw ONLY for a provider-side failure (quota exhausted, rate limited,\n    // bad request) — those are transport conditions the caller's cooldown\n    // should count. A successful call that simply found nothing to extract is\n    // a PAGE-level outcome: return empty so the caller records `missing-price`\n    // and moves to the next candidate URL without accruing an outage streak.\n    // Conflating the two lets two ordinary no-product pages disable Firecrawl\n    // for the rest of the scrape, on every retailer, not just opted-in ones.\n    if (!data.success) {\n      throw new Error(`Firecrawl extract error: ${data.error ?? 'unknown'}`);\n    }\n\n    return {\n      url,\n      data: (data.data?.extract ?? {}) as T,\n      provider: this.name,\n      fetchedAt: new Date(),\n      ...(typeof data.data?.markdown === 'string' && data.data.markdown.trim()\n        ? { pageContent: data.data.markdown }\n        : {}),\n    };\n  }\n\n  async validate(): Promise<boolean> {\n    try {\n      const resp = await fetch(`${this.baseUrl}/scrape`, {\n        method: 'POST',\n        headers: this.headers(),","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/koala73/worldmonitor/blob/eeab0a219fce0f02a00603b532dbae9041b934ac/consumer-prices-core/src/acquisition/firecrawl.ts#L150-L186","documentation":"FirecrawlProvider.extract() throws this when POST https://api.firecrawl.dev/v1/scrape (formats: ['extract','markdown']) returns HTTP 200 but the body carries success:false — a provider-side failure such as rate limiting, exhausted quota, or a rejected request. Per the code comment this is deliberately a transport condition: the caller's ProviderCooldownGate counts it toward the per-scrape Firecrawl cooldown. A page that simply has nothing to extract is NOT this error; that path returns success:true with an empty extract so the caller records 'missing-price' without penalizing the provider.","triggerScenarios":"Calling extract() with a valid key while the Firecrawl plan is out of credits or rate-limited (the error string arrives in-band with HTTP 200); sending an extract payload Firecrawl rejects after acceptance; Firecrawl internal errors surfaced as success:false instead of a non-2xx status.","commonSituations":"A scrape run works for hours, then every Firecrawl extract fails after a quota or billing change; two consecutive provider-side errors open the adapter's firecrawlGate cooldown so the whole retailer scrape starts failing fast; a schema change (nullable-field encoding) that Firecrawl rejects in-band.","solutions":["Read the interpolated data.error text — 'rate limit' means back off, 'credits/quota' means billing, a schema complaint means fix the ExtractSchema","Verify the Firecrawl plan's remaining credits and rate limits in the dashboard","If rate-limited, wait out the cooldown window — the gate half-opens and probes automatically; do not force calls inside it","Confirm the schema encoding: Firecrawl accepts JSON Schema type:[T,'null'] unions for nullable fields (Exa must use anyOf instead — see the #6182 warning in firecrawl.ts)","If Firecrawl outages are routine for this retailer, set searchConfig.extractionFallback = 'exa' so the cooldown abort is bypassed"],"exampleFix":"// before — treats an empty extract as a provider failure\nconst res = await firecrawl.extract(url, schema);\nif (!Object.keys(res.data).length) throw new Error('provider broke');\n\n// after — empty extract is a page-level outcome; only !success throws\ntry {\n  const res = await firecrawl.extract(url, schema);\n  if (!Object.keys(res.data).length) recordOutcome('missing-price'); // keep going, no outage streak\n} catch (err) {\n  gate.recordFailure(); // provider-side failure: let the cooldown count it\n}","handlingStrategy":"retry","validationCode":"// before the scrape run: fail fast on a missing key\nif (!process.env.FIRECRAWL_API_KEY) {\n  throw new Error('FIRECRAWL_API_KEY not set — every Firecrawl extract will throw');\n}","typeGuard":"function isFirecrawlExtractError(err: unknown): boolean {\n  return err instanceof Error && err.message.startsWith('Firecrawl extract error:');\n}","tryCatchPattern":"try {\n  const res = await provider.extract(url, schema);\n  if (!Object.keys(res.data).length) recordOutcome('missing-price'); // page-level, NOT an error\n} catch (err) {\n  if (isFirecrawlExtractError(err)) {\n    // provider-side failure: let the cooldown gate absorb it, retry on the half-open probe\n    return await retryLater(() => provider.extract(url, schema));\n  }\n  throw err;\n}","preventionTips":["Treat success:true with an empty extract as 'missing-price', never as a provider failure — conflating them opens the cooldown on ordinary no-product pages","Monitor the data.error strings: quota and rate-limit messages arrive in-band with HTTP 200","Keep ExtractSchema encodings provider-specific (Firecrawl: type union; Exa: anyOf) and re-test live before unifying"],"tags":["firecrawl","web-scraping","api-response","rate-limit","quota"],"backgroundTag":"api-error-response","analyzedSha":"eeab0a219fce0f02a00603b532dbae9041b934ac","analyzedAt":"2026-08-21T16:51:25.751Z","schemaVersion":2},"datasetVersion":"2026-08-23T13:39:53.451Z"}