{"record":{"id":"e28679167d1f798a","repo":"koala73/worldmonitor","slug":"firecrawl-search-failed-http-resp-status","errorCode":null,"errorMessage":"Firecrawl search failed: HTTP ${resp.status}","messagePattern":"Firecrawl search failed: HTTP (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"consumer-prices-core/src/acquisition/firecrawl.ts","lineNumber":105,"sourceCode":"      provider: this.name,\n      fetchedAt: new Date(),\n      metadata: data.data.metadata,\n    };\n  }\n\n  async search(query: string, opts: SearchOptions = {}): Promise<SearchResult[]> {\n    const resp = await fetch(`${this.baseUrl}/search`, {\n      method: 'POST',\n      headers: this.headers(),\n      body: JSON.stringify({\n        query,\n        limit: opts.numResults ?? 10,\n        includeDomains: opts.includeDomains,\n        scrapeOptions: { formats: ['markdown'] },\n      }),\n    });\n\n    if (!resp.ok) throw new Error(`Firecrawl search failed: HTTP ${resp.status}`);\n\n    const data = (await resp.json()) as FirecrawlSearchResponse;\n    return (data.data ?? []).map((r) => ({\n      url: r.url,\n      title: r.title,\n      text: r.description ?? r.markdown,\n    }));\n  }\n\n  async extract<T = Record<string, unknown>>(\n    url: string,\n    schema: ExtractSchema,\n    opts: FetchOptions = {},\n  ): Promise<ExtractResult<T>> {\n    // Nullable is encoded DIFFERENTLY here than in ExaProvider.extract, and the\n    // divergence is deliberate — do not \"unify\" these without re-testing both\n    // providers live. Firecrawl accepts the JSON Schema `type: [T,'null']`\n    // union (verified: HTTP 200, extract returned). Exa's /contents validator","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/koala73/worldmonitor/blob/eeab0a219fce0f02a00603b532dbae9041b934ac/consumer-prices-core/src/acquisition/firecrawl.ts#L87-L123","documentation":"FirecrawlProvider.search POSTs to /v1/search with the query, limit (default 10) and optional includeDomains; any non-2xx throws with the HTTP status. Typical codes: 401 auth failure, 402 credits exhausted, 400 for invalid parameters such as malformed includeDomains entries, and 5xx provider incidents.","triggerScenarios":"Missing or wrong API key; exhausted credits after search-heavy discovery loops; malformed query options (empty query, invalid domain filters); provider 5xx.","commonSituations":"Discovery phases firing many searches and draining quota; new options added without checking the current Firecrawl search API contract.","solutions":["Check the API key and credit balance first — 401/402 cover most cases","Validate query options (non-empty query, well-formed includeDomains) against the current search API","Retry with backoff only on 5xx; treat 4xx as a caller-side fix","Batch discovery queries and cache results to conserve credits"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Validate search inputs before spending a credit\nif (!query.trim()) throw new Error('query must be non-empty');\nif (includeDomains?.some((d) => !/^[a-z0-9.-]+\\.[a-z]{2,}$/i.test(d))) {\n  throw new Error('includeDomains entries must be bare hostnames');\n}","typeGuard":null,"tryCatchPattern":"try {\n  return await firecrawl.search(query, opts);\n} catch (err) {\n  const m = /HTTP (\\d+)/.exec(err.message);\n  const status = m ? Number(m[1]) : 0;\n  if (status >= 500 || status === 429) return backoffRetry(() => firecrawl.search(query, opts));\n  throw err;   // 400/401/402 are caller-side: fix options or credentials\n}","preventionTips":["Check credits before search-heavy discovery loops","Cache search results per query to avoid repeat spend","Keep option names aligned with the current Firecrawl search API"],"tags":["firecrawl","search","http","api-errors"],"backgroundTag":"http-request-failed","analyzedSha":"eeab0a219fce0f02a00603b532dbae9041b934ac","analyzedAt":"2026-08-21T16:51:25.751Z","schemaVersion":2},"datasetVersion":"2026-08-23T11:17:13.642Z"}