firecrawl/open-lovable · error

${scrapeData.error || 'Failed to scrape website'}

Error message

${scrapeData.error || 'Failed to scrape website'}

What it means

Application-level scraping failure: the scrape endpoint returned HTTP 200 but its JSON body has success === false, so scrapeData.error (or the fallback text) is thrown. The server handled the request but could not produce usable ScrapeData and reported the reason in the payload.

Source

Thrown at app/generation/page.tsx:2777

          };
          sessionStorage.removeItem('siteMarkdown'); // Clear after use
          addChatMessage('Using cached content from search results...', 'system');
        } else {
          // Perform fresh scraping
          const scrapeResponse = await fetch('/api/scrape-url-enhanced', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ url })
          });
          
          if (!scrapeResponse.ok) {
            throw new Error('Failed to scrape website');
          }
          
          scrapeData = await scrapeResponse.json() as ScrapeData;
          
          if (!scrapeData.success) {
            throw new Error(scrapeData.error || 'Failed to scrape website');
          }
        }
        }

        setUrlStatus(brandExtensionMode ? ['Brand styles extracted!', 'Building your component...'] : ['Website scraped successfully!', 'Generating React app...']);

        // Clear preparing design state and switch to generation tab
        setIsPreparingDesign(false);
        setIsScreenshotLoaded(false); // Reset loaded state
        setUrlScreenshot(null); // Clear screenshot when starting generation
        setTargetUrl(''); // Clear target URL

        // Update loading stage to planning
        setLoadingStage('planning');

        // Brief pause before switching to generation tab
        setTimeout(() => {
          setLoadingStage('generating');

View on GitHub (pinned to 69bd93bae7)

Solutions

1. Read scrapeData.error — it is included in the thrown message
2. Confirm the URL renders real content server-side (view-source or curl the page)
3. Try the site's plain homepage or a simpler page instead of a deep SPA route
4. Retry once; some sites return transient challenge pages that pass on a second fetch
5. If content is client-rendered only, use a scraping path with JS rendering or supply content manually

Example fix

// before
if (!scrapeData.success) {
  throw new Error(scrapeData.error || 'Failed to scrape website');
}
// after
if (!scrapeData?.success || !scrapeData?.content) {
  throw new Error(scrapeData?.error || 'Scrape returned no usable content');
}

const data = await scrapeResponse.json() as ScrapeData;
if (!data || data.success !== true || !data.content) {
  throw new Error(data?.error || 'Scrape returned no usable content');
}

function hasScrapeContent(d: unknown): d is ScrapeData & { success: true; content: { text: string } } {
  return typeof d === 'object' && d !== null && (d as any).success === true &&
    typeof (d as any).content === 'object' && (d as any).content !== null &&
    typeof (d as any).content.text === 'string' && (d as any).content.text.length > 0;
}

try {
  const data = await scrapeResponse.json() as ScrapeData;
  if (!hasScrapeContent(data)) throw new Error(data?.error || 'Failed to scrape website');
  scrapeData = data;
} catch (err: any) {
  addChatMessage(`Scrape failed: ${err.message}`, 'system');
  // let the user retry or supply a different URL
}

When it happens

Trigger: scrapeResponse.json() yields { success: false, error: '...' }: the fetcher got a soft error page (200 with error HTML), extracted no meaningful content, or the page's structure defeated the parser.

Common situations: URL serves a 200 soft-404 or consent-wall page with no real content; SPA whose initial HTML has no body text; page too large or too slow so the extractor truncated and failed; site is non-HTML (PDF, image) so extraction returns no text; extraction heuristics fail on unusual markup.

Related errors

• ${brandGuidelines.error || 'Failed to extract brand styles'}
• Failed to scrape content
• Failed to scrape website
• Unknown sandbox provider: ${selectedProvider}. Supported pro
• ${data.error}

AI-assisted analysis of firecrawl/open-lovable@69bd93bae7 (2026-08-28). Data as JSON: /api/errors/546527dc1201cbe0. Report an issue: GitHub.