{"record":{"id":"ae357352ad516db3","repo":"FlowiseAI/Flowise","slug":"firecrawl-failed-to-scrape-url-error-response","errorCode":null,"errorMessage":"Firecrawl: Failed to scrape URL. Error: ${response.error}","messagePattern":"Firecrawl: Failed to scrape URL\\. Error: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts","lineNumber":603,"sourceCode":"                throw new Error(`Firecrawl: Failed to search. Warning: ${response.warning}`)\n            }\n\n            // Convert search results to FirecrawlDocument format\n            firecrawlDocs = (response.data || []).map((result) => ({\n                markdown: result.description,\n                metadata: {\n                    title: result.title,\n                    sourceURL: result.url,\n                    description: result.description\n                }\n            }))\n        } else if (this.mode === 'scrape') {\n            if (!this.url) {\n                throw new Error('Firecrawl: URL is required for scrape mode')\n            }\n            const response = await app.scrapeUrl(this.url, this.params)\n            if (!response.success) {\n                throw new Error(`Firecrawl: Failed to scrape URL. Error: ${response.error}`)\n            }\n            firecrawlDocs = [response.data as FirecrawlDocument]\n        } else if (this.mode === 'crawl') {\n            if (!this.url) {\n                throw new Error('Firecrawl: URL is required for crawl mode')\n            }\n            const response = await app.crawlUrl(this.url, this.params)\n            if ('status' in response) {\n                if (response.status === 'failed') {\n                    throw new Error('Firecrawl: Crawl job failed')\n                }\n                firecrawlDocs = response.data || []\n            } else {\n                if (!response.success) {\n                    throw new Error(`Firecrawl: Failed to scrape URL. Error: ${response.error}`)\n                }\n                firecrawlDocs = [response.data as FirecrawlDocument]\n            }","sourceCodeStart":585,"sourceCodeEnd":621,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts#L585-L621","documentation":"Thrown by FireCrawlLoader.load() in scrape mode when app.scrapeUrl resolves but `success === false`. Embeds `response.error` (the FireCrawl-supplied reason). Duplicates the throw inside scrapeUrl itself, so reaching this line means scrapeUrl stopped throwing.","triggerScenarios":"scrapeUrl returns `{ success: false, error: '...' }` without throwing. Possible if the library is patched, or in tests with a mock returning success:false. Common upstream reasons: invalid URL, blocked site, timeout.","commonSituations":"Target URL returns 4xx/5xx; site blocks FireCrawl user-agent; non-HTML resource; mock test double returning failure shape.","solutions":["Read `response.error` to get the upstream reason and act on it.","Retry once — many scrape failures are transient (rate limit, temporary block).","Try a simpler page or switch to crawl mode to spread load.","If mocking, return `{ success: true, data: {...} }`."],"exampleFix":"// before\nconst response = await app.scrapeUrl(this.url, this.params)\nif (!response.success) throw new Error(`Firecrawl: Failed to scrape URL. Error: ${response.error}`)\n\n// after\nconst response = await app.scrapeUrl(this.url, this.params)\nif (!response.success) throw new Error(`Firecrawl scrape of ${this.url} failed: ${response.error}`)","handlingStrategy":"retry","validationCode":"try { new URL(url) } catch { throw new Error(`invalid url: ${url}`) }","typeGuard":"function isScrapeSuccess(r: ScrapeResponse | undefined): r is ScrapeResponse & { success: true } {\n  return !!r && r.success === true\n}","tryCatchPattern":"for (let i = 0; i < 2; i++) {\n  try { return await loader.load() }\n  catch (e) { if (/Failed to scrape URL/.test((e as Error).message) && i === 0) continue; throw e }\n}","preventionTips":["Retry once — many scrape failures are transient blocks.","If persistently failing, try the URL in crawl mode to spread load.","Log response.error for diagnosis."],"tags":["firecrawl","scrape","loader","soft-failure","redundant"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}