{"record":{"id":"67222813c5426b9f","repo":"FlowiseAI/Flowise","slug":"failed-to-scrape-url-error-responsedata-error","errorCode":null,"errorMessage":"Failed to scrape URL. Error: ${responseData.error}","messagePattern":"Failed to scrape URL\\. Error: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts","lineNumber":198,"sourceCode":"        if (params?.extractorOptions) {\n            validParams.jsonOptions = {\n                schema: params.extractorOptions.extractionSchema,\n                prompt: params.extractorOptions.extractionPrompt\n            }\n        }\n\n        try {\n            const parameters = {\n                ...validParams,\n                integration: 'flowise'\n            }\n            const response: AxiosResponse = await this.postRequest(this.apiUrl + '/v1/scrape', parameters, headers)\n            if (response.status === 200) {\n                const responseData = response.data\n                if (responseData.success) {\n                    return responseData\n                } else {\n                    throw new Error(`Failed to scrape URL. Error: ${responseData.error}`)\n                }\n            } else {\n                this.handleError(response, 'scrape URL')\n            }\n        } catch (error: any) {\n            throw new Error(error.message)\n        }\n        return { success: false, error: 'Internal server error.' }\n    }\n\n    async crawlUrl(\n        url: string,\n        params: Params | null = null,\n        waitUntilDone: boolean = true,\n        pollInterval: number = 2,\n        idempotencyKey?: string\n    ): Promise<CrawlResponse | CrawlStatusResponse> {\n        const headers = this.prepareHeaders(idempotencyKey)","sourceCodeStart":180,"sourceCodeEnd":216,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts#L180-L216","documentation":"scrapeUrl received HTTP 200 from FireCrawl but the body carried success:false, meaning FireCrawl accepted the request but could not scrape the target. The error field in the response is surfaced. This is an application-level failure reported by the FireCrawl API.","triggerScenarios":"Target URL returned a paywall, login wall, or 4xx/5xx that FireCrawl could not bypass; target blocks FireCrawl's user agent; URL is a JS-rendered SPA that failed to render; URL points at a non-HTML resource FireCrawl will not scrape.","commonSituations":"Scraping a site behind Cloudflare bot protection; scraping an authenticated page; URL is a PDF or image but formats did not include them; target is geo-blocked from FireCrawl's region.","solutions":["Read responseData.error - it usually names the block type (e.g., '_blocked', 'timeout').","Try a different URL or the parent page; enable JS rendering / actions via scrapeOptions if the content requires interaction.","Use crawlUrl instead if the single-page scrape is brittle.","Verify the URL opens in an incognito browser from a different region."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"function isProbablyScrapable(url: string): boolean {\n  try {\n    const u = new URL(url)\n    return (u.protocol === 'http:' || u.protocol === 'https:') && u.hostname.includes('.')\n  } catch { return false }\n}\n// if (!isProbablyScrapable(url)) throw new Error('URL not scrapable by FireCrawl before calling scrapeUrl')","typeGuard":null,"tryCatchPattern":"try {\n  const result = await app.scrapeUrl(url, params)\n} catch (error) {\n  const msg = error instanceof Error ? error.message : String(error)\n  if (/Failed to scrape URL\\. Error:/.test(msg)) {\n    // API-level scrape failure - try alternate options (actions, formats) or a different URL\n  }\n  throw error\n}","preventionTips":["Point only at publicly reachable http(s) URLs.","Enable JS rendering / actions in scrapeOptions when the page requires interaction.","If the target is behind a paywall/login, scrape an authorized/preview endpoint instead."],"tags":["firecrawl","api-errors","scraping"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}