{"record":{"id":"d5f3421936120da4","repo":"firecrawl/open-lovable","slug":"firecrawl-api-error-error","errorCode":null,"errorMessage":"Firecrawl API error: ${error}","messagePattern":"Firecrawl API error: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"app/api/scrape-url-enhanced/route.ts","lineNumber":66,"sourceCode":"        timeout: 30000,\n        blockAds: true,\n        maxAge: 3600000, // Use cached data if less than 1 hour old (500% faster!)\n        actions: [\n          {\n            type: 'wait',\n            milliseconds: 2000\n          },\n          {\n            type: 'screenshot',\n            fullPage: false // Just visible viewport for performance\n          }\n        ]\n      })\n    });\n    \n    if (!firecrawlResponse.ok) {\n      const error = await firecrawlResponse.text();\n      throw new Error(`Firecrawl API error: ${error}`);\n    }\n    \n    const data = await firecrawlResponse.json();\n    \n    if (!data.success || !data.data) {\n      throw new Error('Failed to scrape content');\n    }\n    \n    const { markdown, metadata, screenshot, actions } = data.data;\n    // html available but not used in current implementation\n    \n    // Get screenshot from either direct field or actions result\n    const screenshotUrl = screenshot || actions?.screenshots?.[0] || null;\n    \n    // Sanitize the markdown content\n    const sanitizedMarkdown = sanitizeQuotes(markdown || '');\n    \n    // Extract structured data from the response","sourceCodeStart":48,"sourceCodeEnd":84,"githubUrl":"https://github.com/firecrawl/open-lovable/blob/69bd93bae7a9c97ef989eb70aabe6797fb3dac89/app/api/scrape-url-enhanced/route.ts#L48-L84","documentation":"This error is thrown by the POST handler in app/api/scrape-url-enhanced/route.ts:66 when the raw fetch call to https://api.firecrawl.dev/v1/scrape returns a non-2xx HTTP status. The developer reads the response body as text and prefixes it with 'Firecrawl API error:' so the upstream API's error payload (e.g. 401 unauthorized, 402 payment required, 429 rate limit, 5xx) is surfaced verbatim. It is a generic wrapper for any Firecrawl HTTP-level failure, as opposed to the 2xx-but-unsuccessful case handled on the next lines.","triggerScenarios":"Any POST to /api/scrape-url-enhanced where api.firecrawl.dev/v1/scrape responds with a non-ok status: invalid or missing FIRECRAWL_API_KEY (401), exhausted credits/plan limit (402/429), malformed request body (400), a URL Firecrawl refuses to scrape (blocked/paywalled site), or Firecrawl-side outage/timeout exceeding the 30000ms timeout option.","commonSituations":"Deployed app where FIRECRAWL_API_KEY env var was never set or rotated/revoked; free-tier credit exhaustion after heavy scraping; scraping sites that block bots (403) or require JS/captcha; passing internal/localhost URLs which Firecrawl rejects; Firecrawl incidents or rate limiting under concurrent load.","solutions":["Log firecrawlResponse.status alongside the body text so the actual upstream cause (401 vs 402 vs 429 vs 5xx) is visible, then fix that specific cause first.","Verify FIRECRAWL_API_KEY is set in the deployment environment and is valid — test with curl -H 'Authorization: Bearer $KEY' https://api.firecrawl.dev/v1/scrape.","Check your Firecrawl dashboard for credit/plan exhaustion or rate limits; upgrade the plan or add backoff/retry on 429.","Retry transient failures (429/5xx) with exponential backoff; for slow sites consider raising the timeout/waitFor options or dropping the extra screenshot actions.","Return a structured error to the client (status code plus sanitized message) instead of leaking the raw upstream response body."],"exampleFix":"// before\nif (!firecrawlResponse.ok) {\n  const error = await firecrawlResponse.text();\n  throw new Error(`Firecrawl API error: ${error}`);\n}\n// after\nif (!firecrawlResponse.ok) {\n  const error = await firecrawlResponse.text();\n  if (firecrawlResponse.status === 429 || firecrawlResponse.status >= 500) {\n    // retry with backoff, e.g. via p-retry\n  }\n  console.error(`Firecrawl scrape failed (${firecrawlResponse.status}):`, error);\n  return NextResponse.json({ success: false, error: `Scrape failed (${firecrawlResponse.status})` }, { status: firecrawlResponse.status === 401 ? 500 : 502 });\n}","handlingStrategy":"try-catch","validationCode":"if (!process.env.FIRECRAWL_API_KEY) {\n  throw new Error('FIRECRAWL_API_KEY is not configured');\n}\ntry { new URL(url); } catch { throw new Error(`Invalid URL: ${url}`); }","typeGuard":"function isFirecrawlOk(res: Response) { return res.ok; }","tryCatchPattern":"try {\n  const res = await fetch('https://api.firecrawl.dev/v1/scrape', { ... });\n  if (!res.ok) {\n    const body = await res.text();\n    if (res.status === 429 || res.status >= 500) throw new RetryableError(`Firecrawl ${res.status}: ${body}`);\n    throw new Error(`Firecrawl ${res.status}: ${body.slice(0, 300)}`);\n  }\n} catch (e) {\n  if (e instanceof RetryableError) { /* backoff + retry */ }\n  return NextResponse.json({ success: false, error: 'Scrape unavailable' }, { status: 502 });\n}","preventionTips":["Always check both res.ok and res.status before parsing the body, and log the status with the body text.","Validate the URL is public and well-formed before calling Firecrawl (reject localhost/private IPs).","Monitor Firecrawl credit usage and set alerts before quota exhaustion.","Implement exponential backoff retry for 429/5xx responses only.","Never surface the raw upstream body to end users; map it to a friendly message."],"tags":["firecrawl","http-error","api","network"],"backgroundTag":"upstream-api-http-error","analyzedSha":"69bd93bae7a9c97ef989eb70aabe6797fb3dac89","analyzedAt":"2026-08-28T22:20:32.339Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}