{"record":{"id":"546527dc1201cbe0","repo":"firecrawl/open-lovable","slug":"scrapedata-error-failed-to-scrape-website","errorCode":null,"errorMessage":"${scrapeData.error || 'Failed to scrape website'}","messagePattern":"\\$\\{scrapeData\\.error \\|\\| 'Failed to scrape website'\\}","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"app/generation/page.tsx","lineNumber":2777,"sourceCode":"          };\n          sessionStorage.removeItem('siteMarkdown'); // Clear after use\n          addChatMessage('Using cached content from search results...', 'system');\n        } else {\n          // Perform fresh scraping\n          const scrapeResponse = await fetch('/api/scrape-url-enhanced', {\n            method: 'POST',\n            headers: { 'Content-Type': 'application/json' },\n            body: JSON.stringify({ url })\n          });\n          \n          if (!scrapeResponse.ok) {\n            throw new Error('Failed to scrape website');\n          }\n          \n          scrapeData = await scrapeResponse.json() as ScrapeData;\n          \n          if (!scrapeData.success) {\n            throw new Error(scrapeData.error || 'Failed to scrape website');\n          }\n        }\n        }\n\n        setUrlStatus(brandExtensionMode ? ['Brand styles extracted!', 'Building your component...'] : ['Website scraped successfully!', 'Generating React app...']);\n\n        // Clear preparing design state and switch to generation tab\n        setIsPreparingDesign(false);\n        setIsScreenshotLoaded(false); // Reset loaded state\n        setUrlScreenshot(null); // Clear screenshot when starting generation\n        setTargetUrl(''); // Clear target URL\n\n        // Update loading stage to planning\n        setLoadingStage('planning');\n\n        // Brief pause before switching to generation tab\n        setTimeout(() => {\n          setLoadingStage('generating');","sourceCodeStart":2759,"sourceCodeEnd":2795,"githubUrl":"https://github.com/firecrawl/open-lovable/blob/69bd93bae7a9c97ef989eb70aabe6797fb3dac89/app/generation/page.tsx#L2759-L2795","documentation":"Application-level scraping failure: the scrape endpoint returned HTTP 200 but its JSON body has success === false, so scrapeData.error (or the fallback text) is thrown. The server handled the request but could not produce usable ScrapeData and reported the reason in the payload.","triggerScenarios":"scrapeResponse.json() yields { success: false, error: '...' }: the fetcher got a soft error page (200 with error HTML), extracted no meaningful content, or the page's structure defeated the parser.","commonSituations":"URL serves a 200 soft-404 or consent-wall page with no real content; SPA whose initial HTML has no body text; page too large or too slow so the extractor truncated and failed; site is non-HTML (PDF, image) so extraction returns no text; extraction heuristics fail on unusual markup.","solutions":["Read scrapeData.error — it is included in the thrown message","Confirm the URL renders real content server-side (view-source or curl the page)","Try the site's plain homepage or a simpler page instead of a deep SPA route","Retry once; some sites return transient challenge pages that pass on a second fetch","If content is client-rendered only, use a scraping path with JS rendering or supply content manually"],"exampleFix":"// before\nif (!scrapeData.success) {\n  throw new Error(scrapeData.error || 'Failed to scrape website');\n}\n// after\nif (!scrapeData?.success || !scrapeData?.content) {\n  throw new Error(scrapeData?.error || 'Scrape returned no usable content');\n}","handlingStrategy":"type-guard","validationCode":"const data = await scrapeResponse.json() as ScrapeData;\nif (!data || data.success !== true || !data.content) {\n  throw new Error(data?.error || 'Scrape returned no usable content');\n}","typeGuard":"function hasScrapeContent(d: unknown): d is ScrapeData & { success: true; content: { text: string } } {\n  return typeof d === 'object' && d !== null && (d as any).success === true &&\n    typeof (d as any).content === 'object' && (d as any).content !== null &&\n    typeof (d as any).content.text === 'string' && (d as any).content.text.length > 0;\n}","tryCatchPattern":"try {\n  const data = await scrapeResponse.json() as ScrapeData;\n  if (!hasScrapeContent(data)) throw new Error(data?.error || 'Failed to scrape website');\n  scrapeData = data;\n} catch (err: any) {\n  addChatMessage(`Scrape failed: ${err.message}`, 'system');\n  // let the user retry or supply a different URL\n}","preventionTips":["Validate both the success flag and that extracted content is non-empty before using it","Prefer URLs with server-rendered content over pure SPAs","Log data.error to identify soft-404/consent pages that return HTTP 200","Retry once for transient challenge pages","Guide users to simpler pages (homepage) when deep routes fail extraction"],"tags":["scraping","api-response","validation"],"backgroundTag":"scrape-request-blocked","analyzedSha":"69bd93bae7a9c97ef989eb70aabe6797fb3dac89","analyzedAt":"2026-08-28T22:20:32.339Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}