{"record":{"id":"65bda32426dbc2ba","repo":"koala73/worldmonitor","slug":"firecrawl-error-data-error-unknown","errorCode":null,"errorMessage":"Firecrawl error: ${data.error ?? 'unknown'}","messagePattern":"Firecrawl error: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"consumer-prices-core/src/acquisition/firecrawl.ts","lineNumber":79,"sourceCode":"  async fetch(url: string, opts: FetchOptions = {}): Promise<FetchResult> {\n    const resp = await fetch(`${this.baseUrl}/scrape`, {\n      method: 'POST',\n      headers: this.headers(),\n      body: JSON.stringify({\n        url,\n        formats: ['html', 'markdown'],\n        waitFor: opts.waitForSelector ? 2000 : 0,\n        timeout: opts.timeout ?? 30_000,\n        headers: opts.headers,\n      }),\n      signal: AbortSignal.timeout((opts.timeout ?? 30_000) + 5_000),\n    });\n\n    if (!resp.ok) throw new Error(`Firecrawl scrape failed: HTTP ${resp.status}`);\n\n    const data = (await resp.json()) as FirecrawlScrapeResponse;\n    if (!data.success || !data.data) {\n      throw new Error(`Firecrawl error: ${data.error ?? 'unknown'}`);\n    }\n\n    return {\n      url,\n      html: data.data.html ?? '',\n      markdown: data.data.markdown ?? '',\n      statusCode: 200,\n      provider: this.name,\n      fetchedAt: new Date(),\n      metadata: data.data.metadata,\n    };\n  }\n\n  async search(query: string, opts: SearchOptions = {}): Promise<SearchResult[]> {\n    const resp = await fetch(`${this.baseUrl}/search`, {\n      method: 'POST',\n      headers: this.headers(),\n      body: JSON.stringify({","sourceCodeStart":61,"sourceCodeEnd":97,"githubUrl":"https://github.com/koala73/worldmonitor/blob/eeab0a219fce0f02a00603b532dbae9041b934ac/consumer-prices-core/src/acquisition/firecrawl.ts#L61-L97","documentation":"The scrape HTTP call succeeded (2xx) but the FirecrawlScrapeResponse body reports success:false or lacks data — a provider-level failure delivered in-band. The embedded data.error names the cause ('unknown' when absent), typically a failed render, an upstream block, or a provider-internal error.","triggerScenarios":"Target site blocks Firecrawl (bot protection, JS challenge, 403 at origin); the page render times out inside Firecrawl while the HTTP envelope still returns 200; a provider incident producing error bodies; quota-related in-band errors.","commonSituations":"Hardened retailer sites during price scraping; regional blocks; intermittent render failures on specific templates.","solutions":["Read the embedded data.error string — it names the provider-side cause","Retry with waitForSelector and/or a higher timeout for render-dependent pages","Route persistently blocked domains to a different provider (Exa) or the relay path","If data.error mentions quota/credits, top up the account"],"exampleFix":"// before\nif (!data.success || !data.data) {\n  throw new Error(`Firecrawl error: ${data.error ?? 'unknown'}`);\n}\n\n// after — keep the provider error, but make it classifiable for cooldown logic\nif (!data.success || !data.data) {\n  const err = new Error(`Firecrawl error: ${data.error ?? 'unknown'}`) as Error & { providerError?: string };\n  err.providerError = data.error ?? 'unknown';\n  throw err;\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"interface FirecrawlScrapeResponse { success?: boolean; data?: unknown; error?: string }\nfunction isProviderFailure(body: FirecrawlScrapeResponse | undefined): body is { success: false; error?: string } {\n  return !body?.success || body?.data === undefined;\n}","tryCatchPattern":"try {\n  return await firecrawl.fetch(url);\n} catch (err) {\n  if (err instanceof Error && err.message.startsWith('Firecrawl error:')) {\n    // in-band provider failure: consult data.error for the cause; render issues are retryable with waitFor\n    if (/block|403|forbidden/i.test(err.message)) return routeToAlternativeProvider(url);\n    return retryOnce(() => firecrawl.fetch(url, { waitForSelector: 'body' }));\n  }\n  throw err;\n}","preventionTips":["Distinguish in-band provider errors (HTTP 200, success:false) from transport errors — they need different handling","Capture data.error strings in logs; they carry the actionable cause","Keep an alternative provider wired for domains Firecrawl cannot render"],"tags":["firecrawl","scraping","api-response","bot-protection"],"backgroundTag":"api-error-response-body","analyzedSha":"eeab0a219fce0f02a00603b532dbae9041b934ac","analyzedAt":"2026-08-21T16:51:25.751Z","schemaVersion":2},"datasetVersion":"2026-08-23T13:39:53.451Z"}