firecrawl/open-lovable · error · Error

Failed to scrape website

Error message

Failed to scrape website

What it means

This error is thrown in app/api/scrape-website/route.ts:54 when the @mendable/firecrawl-js SDK's scrape() resolves with an object whose success property is explicitly false. The Firecrawl SDK mirrors the API contract: a scrape that fails server-side still resolves (does not reject) with {success:false, error:'...'}. The route forwards result.error if present and only falls back to the generic 'Failed to scrape website' message when the SDK supplied no error string.

Source

Thrown at app/api/scrape-website/route.ts:54

      });
    }
    
    const app = new FirecrawlApp({ apiKey });
    
    // Scrape the website using the latest SDK patterns
    // Include screenshot if requested in formats
    const scrapeResult = await app.scrape(url, {
      formats: formats,
      onlyMainContent: options.onlyMainContent !== false, // Default to true for cleaner content
      waitFor: options.waitFor || 2000, // Wait for dynamic content
      timeout: options.timeout || 30000,
      ...options // Pass through any additional options
    });
    
    // Handle the response according to the latest SDK structure
    const result = scrapeResult as any;
    if (result.success === false) {
      throw new Error(result.error || "Failed to scrape website");
    }
    
    // The SDK may return data directly or nested
    const data = result.data || result;
    
    return NextResponse.json({
      success: true,
      data: {
        title: data?.metadata?.title || "Untitled",
        content: data?.markdown || data?.html || "",
        description: data?.metadata?.description || "",
        markdown: data?.markdown || "",
        html: data?.html || "",
        metadata: data?.metadata || {},
        screenshot: data?.screenshot || null,
        links: data?.links || [],
        // Include raw data for flexibility
        raw: data

View on GitHub (pinned to 69bd93bae7)

Solutions

1. Log the entire scrapeResult object to see whether the SDK actually returned an error string that was lost or a differently-named field (e.g. error.message).
2. Upgrade @mendable/firecrawl-js to the latest version and align the call with current docs (many versions changed scrape() argument order and response envelope).
3. Check the URL resolves publicly and is not blocked; test it directly against the Firecrawl API/ playground.
4. Raise timeout/waitFor or trim formats for slow or heavy pages, and add retry logic for transient failures.
5. Return result.error through to the client response instead of masking it with the generic message.

Example fix

// before
const result = scrapeResult as any;
if (result.success === false) {
  throw new Error(result.error || "Failed to scrape website");
}
// after
const result = scrapeResult as any;
if (result.success === false) {
  console.error('Firecrawl scrape failed:', JSON.stringify(result).slice(0, 500));
  throw new Error(result?.error || result?.data?.error || `Firecrawl reported failure for ${url}`);
}

const apiKey = process.env.FIRECRAWL_API_KEY;
if (!apiKey) throw new Error('FIRECRAWL_API_KEY is not set');
new URL(url); // throws on malformed url

interface FirecrawlResult { success: boolean; error?: string; data?: any }
function isScrapeFailure(r: any): r is { success: false; error?: string } {
  return !!r && typeof r === 'object' && r.success === false;
}
function hasScrapeData(r: any): boolean {
  return !!r && (r.data != null || r.markdown != null || r.html != null);
}

try {
  const result = (await app.scrape(url, opts)) as any;
  if (result?.success === false) throw new Error(result.error || 'Firecrawl returned failure without details');
  if (!result?.data && !result?.markdown) throw new Error('Unexpected Firecrawl response shape: ' + JSON.stringify(result).slice(0, 200));
} catch (e) {
  console.error('scrape-website failed:', e);
  return NextResponse.json({ error: (e as Error).message }, { status: 502 });
}

When it happens

Trigger: Calling POST /api/scrape-website where app.scrape(url, {...}) returns success===false with no error message: the target URL is unreachable or DNS-fails, the page blocks scraping, the scrape exceeds timeout:30000, or an SDK/API version mismatch yields an unexpected result shape (hence the `as any` cast).

Common situations: Scraping dead domains or mistyped URLs; sites with aggressive anti-bot protection; slow pages exceeding the 30s timeout; using an older @mendable/firecrawl-js version whose scrape() signature/response differs from the docs, so success/error fields are missing and the fallback message fires.

Related errors

• Failed to scrape content
• Firecrawl API error: ${error}
• Firecrawl API key not configured
• Firecrawl API returned ${firecrawlResponse.status}
• No branding data in Firecrawl response

AI-assisted analysis of firecrawl/open-lovable@69bd93bae7 (2026-08-28). Data as JSON: /api/errors/1112aec3e8e61402. Report an issue: GitHub.