{"record":{"id":"a594844d052818f3","repo":"firecrawl/open-lovable","slug":"failed-to-scrape-website-a59484","errorCode":null,"errorMessage":"Failed to scrape website","messagePattern":"Failed to scrape website","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"app/generation/page.tsx","lineNumber":2771,"sourceCode":"          // Use the pre-scraped content\n          scrapeData = {\n            success: true,\n            content: storedMarkdown,\n            title: new URL(url).hostname,\n            source: 'search-result'\n          };\n          sessionStorage.removeItem('siteMarkdown'); // Clear after use\n          addChatMessage('Using cached content from search results...', 'system');\n        } else {\n          // Perform fresh scraping\n          const scrapeResponse = await fetch('/api/scrape-url-enhanced', {\n            method: 'POST',\n            headers: { 'Content-Type': 'application/json' },\n            body: JSON.stringify({ url })\n          });\n          \n          if (!scrapeResponse.ok) {\n            throw new Error('Failed to scrape website');\n          }\n          \n          scrapeData = await scrapeResponse.json() as ScrapeData;\n          \n          if (!scrapeData.success) {\n            throw new Error(scrapeData.error || 'Failed to scrape website');\n          }\n        }\n        }\n\n        setUrlStatus(brandExtensionMode ? ['Brand styles extracted!', 'Building your component...'] : ['Website scraped successfully!', 'Generating React app...']);\n\n        // Clear preparing design state and switch to generation tab\n        setIsPreparingDesign(false);\n        setIsScreenshotLoaded(false); // Reset loaded state\n        setUrlScreenshot(null); // Clear screenshot when starting generation\n        setTargetUrl(''); // Clear target URL\n","sourceCodeStart":2753,"sourceCodeEnd":2789,"githubUrl":"https://github.com/firecrawl/open-lovable/blob/69bd93bae7a9c97ef989eb70aabe6797fb3dac89/app/generation/page.tsx#L2753-L2789","documentation":"Thrown when the website-scraping endpoint returns a non-2xx HTTP status in normal (clone) mode. The client posted { url } and scrapeResponse.ok was false, so no ScrapeData could be produced for the site-clone generation flow. Like error 44, the message is intentionally coarse; the status code and body hold the specifics.","triggerScenarios":"fetch to the scrape route with { url } resolves with scrapeResponse.ok === false: target site refused or blocked the server-side fetch (403/robots/anti-bot), DNS failure for a bad domain, route timeout on slow pages, or the route 500s while parsing the page.","commonSituations":"User enters an unreachable or misspelled domain; target site blocks datacenter IPs or requires JS rendering; URL lacks a scheme causing server-side fetch failure; scraping route rate-limited by the target host; scraping service/route not deployed in the current environment.","solutions":["Check the scrape request's status code and body in the Network tab for the real cause","Validate the URL (protocol + reachable host) before submitting — e.g. `new URL(url)` and a HEAD request","If the site blocks scraping, try a different publicly accessible page or mirror of the content","Check server logs / scraper service health; timeouts and DNS failures show up there","Retry once for transient 5xx or network blips"],"exampleFix":"// before\nif (!scrapeResponse.ok) {\n  throw new Error('Failed to scrape website');\n}\n// after\nif (!scrapeResponse.ok) {\n  let detail = '';\n  try { detail = await scrapeResponse.text(); } catch {}\n  throw new Error(`Failed to scrape website (${scrapeResponse.status}): ${detail || scrapeResponse.statusText}`);\n}","handlingStrategy":"validation","validationCode":"function isScrapableUrl(url: string): boolean {\n  try { const u = new URL(url); return (u.protocol === 'http:' || u.protocol === 'https:') && !!u.hostname; }\n  catch { return false; }\n}\nif (!isScrapableUrl(url)) throw new Error('Enter a valid http(s) website URL');","typeGuard":"interface ScrapeData { success: boolean; error?: string; content?: unknown }\nfunction isScrapeOk(d: unknown): d is ScrapeData & { success: true; content: unknown } {\n  return typeof d === 'object' && d !== null && (d as ScrapeData).success === true;\n}","tryCatchPattern":"try {\n  const res = await fetch(SCRAPE_URL, { method: 'POST', body: JSON.stringify({ url }) });\n  if (!res.ok) throw new Error(`Scrape HTTP ${res.status}`);\n  scrapeData = await res.json();\n  if (!isScrapeOk(scrapeData)) throw new Error(scrapeData?.error || 'Failed to scrape website');\n} catch (err: any) {\n  addChatMessage(`Scraping failed: ${err.message}. Check the URL and try again.`, 'system');\n}","preventionTips":["Validate the URL (protocol + host) before calling the scrape endpoint","Warn users when the target likely blocks bots (Cloudflare, login walls)","Read the scrape response body/status for real diagnostics, not a generic message","Add one retry for transient 5xx/timeout responses","Monitor the scraping service for rate-limiting or DNS failures"],"tags":["network","scraping","http-status"],"backgroundTag":"scrape-request-blocked","analyzedSha":"69bd93bae7a9c97ef989eb70aabe6797fb3dac89","analyzedAt":"2026-08-28T22:20:32.339Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}