firecrawl/open-lovable · error · Error

Failed to scrape content

Error message

Failed to scrape content

What it means

This error is thrown at app/api/scrape-url-enhanced/route.ts:72 when the Firecrawl /v1/scrape call returned HTTP 200 but its JSON body either has success=false or omits the data object. Firecrawl signals scrape failure inside a 200 response via the success flag, so this guard catches 'the API answered fine but did not produce scraped content'. It means the scraping job itself failed (or the response shape changed) even though authentication and the request were accepted.

Source

Thrown at app/api/scrape-url-enhanced/route.ts:72

            milliseconds: 2000
          },
          {
            type: 'screenshot',
            fullPage: false // Just visible viewport for performance
          }
        ]
      })
    });
    
    if (!firecrawlResponse.ok) {
      const error = await firecrawlResponse.text();
      throw new Error(`Firecrawl API error: ${error}`);
    }
    
    const data = await firecrawlResponse.json();
    
    if (!data.success || !data.data) {
      throw new Error('Failed to scrape content');
    }
    
    const { markdown, metadata, screenshot, actions } = data.data;
    // html available but not used in current implementation
    
    // Get screenshot from either direct field or actions result
    const screenshotUrl = screenshot || actions?.screenshots?.[0] || null;
    
    // Sanitize the markdown content
    const sanitizedMarkdown = sanitizeQuotes(markdown || '');
    
    // Extract structured data from the response
    const title = metadata?.title || '';
    const description = metadata?.description || '';
    
    // Format content for AI
    const formattedContent = `
Title: ${sanitizeQuotes(title)}

View on GitHub (pinned to 69bd93bae7)

Solutions

1. Log the full response JSON before throwing so you can see any Firecrawl-provided error field explaining why success was false.
2. Inspect the target URL in the Firecrawl playground to see if it is scrapeable at all (blocks, login walls, heavy JS).
3. Tune scrape options: raise timeout/waitFor, remove the extra actions, or add onlyMainContent/headers to improve success on difficult pages.
4. Add a retry for transient scrape failures and a fallback message to the client instead of a generic 500.
5. Pin/verify the Firecrawl API version (v1) and validate the response against the documented schema to catch shape changes early.

Example fix

// before
if (!data.success || !data.data) {
  throw new Error('Failed to scrape content');
}
// after
if (!data.success || !data.data) {
  console.error('Firecrawl scrape unsuccessful for', url, JSON.stringify(data).slice(0, 500));
  return NextResponse.json({ success: false, error: data.error || 'The page could not be scraped (it may block crawlers or require login)' }, { status: 502 });
}

const FIRECRAWL_API_KEY = process.env.FIRECRAWL_API_KEY;
if (!FIRECRAWL_API_KEY) throw new Error('FIRECRAWL_API_KEY is not set');
let parsed: URL; try { parsed = new URL(url); } catch { throw new Error('Invalid URL'); }

interface FirecrawlScrapeSuccess { success: true; data: { markdown?: string; metadata?: any; screenshot?: string; actions?: { screenshots?: string[] } } }
function isSuccessfulScrape(x: any): x is FirecrawlScrapeSuccess {
  return !!x && typeof x === 'object' && x.success === true && x.data != null && typeof x.data === 'object';
}

const data = await firecrawlResponse.json().catch(() => null);
if (!isSuccessfulScrape(data)) {
  console.error('Firecrawl unsuccessful:', JSON.stringify(data)?.slice(0, 500));
  return NextResponse.json({ success: false, error: 'Page could not be scraped' }, { status: 502 });
}

When it happens

Trigger: A POST to /api/scrape-url-enhanced where Firecrawl returns {success:false} without data: the target page could not be loaded or timed out despite waitFor:3000 and the wait action, the URL redirected to an error page, the site blocked the crawler after initial connection, or Firecrawl returned an undocumented/changed response shape that no longer includes data.

Common situations: Scraping single-page apps behind heavy JS where the scrape job times out; URLs that resolve to 404/403 pages server-side; paywalled or anti-bot-protected sites; a Firecrawl API version bump changing the response envelope so success/data no longer appear as expected.

Related errors

• Failed to scrape website
• No branding data in Firecrawl response
• Screenshot not available in response - check console for ful
• Firecrawl API error: ${error}
• Firecrawl API key not configured

AI-assisted analysis of firecrawl/open-lovable@69bd93bae7 (2026-08-28). Data as JSON: /api/errors/8ca68be36108db5e. Report an issue: GitHub.