firecrawl/open-lovable · error

Failed to scrape website

Error message

Failed to scrape website

What it means

Thrown when the website-scraping endpoint returns a non-2xx HTTP status in normal (clone) mode. The client posted { url } and scrapeResponse.ok was false, so no ScrapeData could be produced for the site-clone generation flow. Like error 44, the message is intentionally coarse; the status code and body hold the specifics.

Source

Thrown at app/generation/page.tsx:2771

          // Use the pre-scraped content
          scrapeData = {
            success: true,
            content: storedMarkdown,
            title: new URL(url).hostname,
            source: 'search-result'
          };
          sessionStorage.removeItem('siteMarkdown'); // Clear after use
          addChatMessage('Using cached content from search results...', 'system');
        } else {
          // Perform fresh scraping
          const scrapeResponse = await fetch('/api/scrape-url-enhanced', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ url })
          });
          
          if (!scrapeResponse.ok) {
            throw new Error('Failed to scrape website');
          }
          
          scrapeData = await scrapeResponse.json() as ScrapeData;
          
          if (!scrapeData.success) {
            throw new Error(scrapeData.error || 'Failed to scrape website');
          }
        }
        }

        setUrlStatus(brandExtensionMode ? ['Brand styles extracted!', 'Building your component...'] : ['Website scraped successfully!', 'Generating React app...']);

        // Clear preparing design state and switch to generation tab
        setIsPreparingDesign(false);
        setIsScreenshotLoaded(false); // Reset loaded state
        setUrlScreenshot(null); // Clear screenshot when starting generation
        setTargetUrl(''); // Clear target URL

View on GitHub (pinned to 69bd93bae7)

Solutions

1. Check the scrape request's status code and body in the Network tab for the real cause
2. Validate the URL (protocol + reachable host) before submitting — e.g. `new URL(url)` and a HEAD request
3. If the site blocks scraping, try a different publicly accessible page or mirror of the content
4. Check server logs / scraper service health; timeouts and DNS failures show up there
5. Retry once for transient 5xx or network blips

Example fix

// before
if (!scrapeResponse.ok) {
  throw new Error('Failed to scrape website');
}
// after
if (!scrapeResponse.ok) {
  let detail = '';
  try { detail = await scrapeResponse.text(); } catch {}
  throw new Error(`Failed to scrape website (${scrapeResponse.status}): ${detail || scrapeResponse.statusText}`);
}

function isScrapableUrl(url: string): boolean {
  try { const u = new URL(url); return (u.protocol === 'http:' || u.protocol === 'https:') && !!u.hostname; }
  catch { return false; }
}
if (!isScrapableUrl(url)) throw new Error('Enter a valid http(s) website URL');

interface ScrapeData { success: boolean; error?: string; content?: unknown }
function isScrapeOk(d: unknown): d is ScrapeData & { success: true; content: unknown } {
  return typeof d === 'object' && d !== null && (d as ScrapeData).success === true;
}

try {
  const res = await fetch(SCRAPE_URL, { method: 'POST', body: JSON.stringify({ url }) });
  if (!res.ok) throw new Error(`Scrape HTTP ${res.status}`);
  scrapeData = await res.json();
  if (!isScrapeOk(scrapeData)) throw new Error(scrapeData?.error || 'Failed to scrape website');
} catch (err: any) {
  addChatMessage(`Scraping failed: ${err.message}. Check the URL and try again.`, 'system');
}

When it happens

Trigger: fetch to the scrape route with { url } resolves with scrapeResponse.ok === false: target site refused or blocked the server-side fetch (403/robots/anti-bot), DNS failure for a bad domain, route timeout on slow pages, or the route 500s while parsing the page.

Common situations: User enters an unreachable or misspelled domain; target site blocks datacenter IPs or requires JS rendering; URL lacks a scheme causing server-side fetch failure; scraping route rate-limited by the target host; scraping service/route not deployed in the current environment.

Related errors

• Failed to extract brand styles
• Firecrawl API returned ${firecrawlResponse.status}
• Failed to apply code: ${response.statusText}
• HTTP error! status: ${response.status}
• Failed to generate code

AI-assisted analysis of firecrawl/open-lovable@69bd93bae7 (2026-08-28). Data as JSON: /api/errors/a594844d052818f3. Report an issue: GitHub.