{"record":{"id":"1112aec3e8e61402","repo":"firecrawl/open-lovable","slug":"failed-to-scrape-website","errorCode":null,"errorMessage":"Failed to scrape website","messagePattern":"Failed to scrape website","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"app/api/scrape-website/route.ts","lineNumber":54,"sourceCode":"      });\n    }\n    \n    const app = new FirecrawlApp({ apiKey });\n    \n    // Scrape the website using the latest SDK patterns\n    // Include screenshot if requested in formats\n    const scrapeResult = await app.scrape(url, {\n      formats: formats,\n      onlyMainContent: options.onlyMainContent !== false, // Default to true for cleaner content\n      waitFor: options.waitFor || 2000, // Wait for dynamic content\n      timeout: options.timeout || 30000,\n      ...options // Pass through any additional options\n    });\n    \n    // Handle the response according to the latest SDK structure\n    const result = scrapeResult as any;\n    if (result.success === false) {\n      throw new Error(result.error || \"Failed to scrape website\");\n    }\n    \n    // The SDK may return data directly or nested\n    const data = result.data || result;\n    \n    return NextResponse.json({\n      success: true,\n      data: {\n        title: data?.metadata?.title || \"Untitled\",\n        content: data?.markdown || data?.html || \"\",\n        description: data?.metadata?.description || \"\",\n        markdown: data?.markdown || \"\",\n        html: data?.html || \"\",\n        metadata: data?.metadata || {},\n        screenshot: data?.screenshot || null,\n        links: data?.links || [],\n        // Include raw data for flexibility\n        raw: data","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/firecrawl/open-lovable/blob/69bd93bae7a9c97ef989eb70aabe6797fb3dac89/app/api/scrape-website/route.ts#L36-L72","documentation":"This error is thrown in app/api/scrape-website/route.ts:54 when the @mendable/firecrawl-js SDK's scrape() resolves with an object whose success property is explicitly false. The Firecrawl SDK mirrors the API contract: a scrape that fails server-side still resolves (does not reject) with {success:false, error:'...'}. The route forwards result.error if present and only falls back to the generic 'Failed to scrape website' message when the SDK supplied no error string.","triggerScenarios":"Calling POST /api/scrape-website where app.scrape(url, {...}) returns success===false with no error message: the target URL is unreachable or DNS-fails, the page blocks scraping, the scrape exceeds timeout:30000, or an SDK/API version mismatch yields an unexpected result shape (hence the `as any` cast).","commonSituations":"Scraping dead domains or mistyped URLs; sites with aggressive anti-bot protection; slow pages exceeding the 30s timeout; using an older @mendable/firecrawl-js version whose scrape() signature/response differs from the docs, so success/error fields are missing and the fallback message fires.","solutions":["Log the entire scrapeResult object to see whether the SDK actually returned an error string that was lost or a differently-named field (e.g. error.message).","Upgrade @mendable/firecrawl-js to the latest version and align the call with current docs (many versions changed scrape() argument order and response envelope).","Check the URL resolves publicly and is not blocked; test it directly against the Firecrawl API/ playground.","Raise timeout/waitFor or trim formats for slow or heavy pages, and add retry logic for transient failures.","Return result.error through to the client response instead of masking it with the generic message."],"exampleFix":"// before\nconst result = scrapeResult as any;\nif (result.success === false) {\n  throw new Error(result.error || \"Failed to scrape website\");\n}\n// after\nconst result = scrapeResult as any;\nif (result.success === false) {\n  console.error('Firecrawl scrape failed:', JSON.stringify(result).slice(0, 500));\n  throw new Error(result?.error || result?.data?.error || `Firecrawl reported failure for ${url}`);\n}","handlingStrategy":"type-guard","validationCode":"const apiKey = process.env.FIRECRAWL_API_KEY;\nif (!apiKey) throw new Error('FIRECRAWL_API_KEY is not set');\nnew URL(url); // throws on malformed url","typeGuard":"interface FirecrawlResult { success: boolean; error?: string; data?: any }\nfunction isScrapeFailure(r: any): r is { success: false; error?: string } {\n  return !!r && typeof r === 'object' && r.success === false;\n}\nfunction hasScrapeData(r: any): boolean {\n  return !!r && (r.data != null || r.markdown != null || r.html != null);\n}","tryCatchPattern":"try {\n  const result = (await app.scrape(url, opts)) as any;\n  if (result?.success === false) throw new Error(result.error || 'Firecrawl returned failure without details');\n  if (!result?.data && !result?.markdown) throw new Error('Unexpected Firecrawl response shape: ' + JSON.stringify(result).slice(0, 200));\n} catch (e) {\n  console.error('scrape-website failed:', e);\n  return NextResponse.json({ error: (e as Error).message }, { status: 502 });\n}","preventionTips":["Keep @mendable/firecrawl-js updated; SDK response shapes changed across versions and the `as any` cast hides drift.","Log the raw SDK result whenever success===false so result.error details are not lost.","Pre-validate URLs (reachable, public, not localhost) before calling the SDK.","Test scrape calls against the live Firecrawl API when upgrading the SDK.","Pass result.error through to the client response rather than the generic fallback string."],"tags":["firecrawl","sdk","scraping","api"],"backgroundTag":"scrape-failed-response","analyzedSha":"69bd93bae7a9c97ef989eb70aabe6797fb3dac89","analyzedAt":"2026-08-28T22:20:32.339Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}