{"record":{"id":"8ca68be36108db5e","repo":"firecrawl/open-lovable","slug":"failed-to-scrape-content","errorCode":null,"errorMessage":"Failed to scrape content","messagePattern":"Failed to scrape content","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"app/api/scrape-url-enhanced/route.ts","lineNumber":72,"sourceCode":"            milliseconds: 2000\n          },\n          {\n            type: 'screenshot',\n            fullPage: false // Just visible viewport for performance\n          }\n        ]\n      })\n    });\n    \n    if (!firecrawlResponse.ok) {\n      const error = await firecrawlResponse.text();\n      throw new Error(`Firecrawl API error: ${error}`);\n    }\n    \n    const data = await firecrawlResponse.json();\n    \n    if (!data.success || !data.data) {\n      throw new Error('Failed to scrape content');\n    }\n    \n    const { markdown, metadata, screenshot, actions } = data.data;\n    // html available but not used in current implementation\n    \n    // Get screenshot from either direct field or actions result\n    const screenshotUrl = screenshot || actions?.screenshots?.[0] || null;\n    \n    // Sanitize the markdown content\n    const sanitizedMarkdown = sanitizeQuotes(markdown || '');\n    \n    // Extract structured data from the response\n    const title = metadata?.title || '';\n    const description = metadata?.description || '';\n    \n    // Format content for AI\n    const formattedContent = `\nTitle: ${sanitizeQuotes(title)}","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/firecrawl/open-lovable/blob/69bd93bae7a9c97ef989eb70aabe6797fb3dac89/app/api/scrape-url-enhanced/route.ts#L54-L90","documentation":"This error is thrown at app/api/scrape-url-enhanced/route.ts:72 when the Firecrawl /v1/scrape call returned HTTP 200 but its JSON body either has success=false or omits the data object. Firecrawl signals scrape failure inside a 200 response via the success flag, so this guard catches 'the API answered fine but did not produce scraped content'. It means the scraping job itself failed (or the response shape changed) even though authentication and the request were accepted.","triggerScenarios":"A POST to /api/scrape-url-enhanced where Firecrawl returns {success:false} without data: the target page could not be loaded or timed out despite waitFor:3000 and the wait action, the URL redirected to an error page, the site blocked the crawler after initial connection, or Firecrawl returned an undocumented/changed response shape that no longer includes data.","commonSituations":"Scraping single-page apps behind heavy JS where the scrape job times out; URLs that resolve to 404/403 pages server-side; paywalled or anti-bot-protected sites; a Firecrawl API version bump changing the response envelope so success/data no longer appear as expected.","solutions":["Log the full response JSON before throwing so you can see any Firecrawl-provided error field explaining why success was false.","Inspect the target URL in the Firecrawl playground to see if it is scrapeable at all (blocks, login walls, heavy JS).","Tune scrape options: raise timeout/waitFor, remove the extra actions, or add onlyMainContent/headers to improve success on difficult pages.","Add a retry for transient scrape failures and a fallback message to the client instead of a generic 500.","Pin/verify the Firecrawl API version (v1) and validate the response against the documented schema to catch shape changes early."],"exampleFix":"// before\nif (!data.success || !data.data) {\n  throw new Error('Failed to scrape content');\n}\n// after\nif (!data.success || !data.data) {\n  console.error('Firecrawl scrape unsuccessful for', url, JSON.stringify(data).slice(0, 500));\n  return NextResponse.json({ success: false, error: data.error || 'The page could not be scraped (it may block crawlers or require login)' }, { status: 502 });\n}","handlingStrategy":"type-guard","validationCode":"const FIRECRAWL_API_KEY = process.env.FIRECRAWL_API_KEY;\nif (!FIRECRAWL_API_KEY) throw new Error('FIRECRAWL_API_KEY is not set');\nlet parsed: URL; try { parsed = new URL(url); } catch { throw new Error('Invalid URL'); }","typeGuard":"interface FirecrawlScrapeSuccess { success: true; data: { markdown?: string; metadata?: any; screenshot?: string; actions?: { screenshots?: string[] } } }\nfunction isSuccessfulScrape(x: any): x is FirecrawlScrapeSuccess {\n  return !!x && typeof x === 'object' && x.success === true && x.data != null && typeof x.data === 'object';\n}","tryCatchPattern":"const data = await firecrawlResponse.json().catch(() => null);\nif (!isSuccessfulScrape(data)) {\n  console.error('Firecrawl unsuccessful:', JSON.stringify(data)?.slice(0, 500));\n  return NextResponse.json({ success: false, error: 'Page could not be scraped' }, { status: 502 });\n}","preventionTips":["Type-guard the full response envelope (success flag AND data object) instead of throwing a generic message.","Check the URL in the Firecrawl playground first for sites known to block scrapers.","Tune timeout/waitFor per target site; heavy SPAs need longer waits.","Retry transient scrape failures; give the user a meaningful 'page not scrapeable' message on permanent failures.","Pin the Firecrawl API version and add a schema assertion test for the response shape."],"tags":["firecrawl","api","scraping","response-shape"],"backgroundTag":"scrape-failed-response","analyzedSha":"69bd93bae7a9c97ef989eb70aabe6797fb3dac89","analyzedAt":"2026-08-28T22:20:32.339Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}