FlowiseAI/Flowise · error · Error

Failed to scrape URL. Error: ${responseData.error}

Error message

Failed to scrape URL. Error: ${responseData.error}

What it means

scrapeUrl received HTTP 200 from FireCrawl but the body carried success:false, meaning FireCrawl accepted the request but could not scrape the target. The error field in the response is surfaced. This is an application-level failure reported by the FireCrawl API.

Source

Thrown at packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts:198

        if (params?.extractorOptions) {
            validParams.jsonOptions = {
                schema: params.extractorOptions.extractionSchema,
                prompt: params.extractorOptions.extractionPrompt
            }
        }

        try {
            const parameters = {
                ...validParams,
                integration: 'flowise'
            }
            const response: AxiosResponse = await this.postRequest(this.apiUrl + '/v1/scrape', parameters, headers)
            if (response.status === 200) {
                const responseData = response.data
                if (responseData.success) {
                    return responseData
                } else {
                    throw new Error(`Failed to scrape URL. Error: ${responseData.error}`)
                }
            } else {
                this.handleError(response, 'scrape URL')
            }
        } catch (error: any) {
            throw new Error(error.message)
        }
        return { success: false, error: 'Internal server error.' }
    }

    async crawlUrl(
        url: string,
        params: Params | null = null,
        waitUntilDone: boolean = true,
        pollInterval: number = 2,
        idempotencyKey?: string
    ): Promise<CrawlResponse | CrawlStatusResponse> {
        const headers = this.prepareHeaders(idempotencyKey)

View on GitHub (pinned to abe4a8601a)

Solutions

1. Read responseData.error - it usually names the block type (e.g., '_blocked', 'timeout').
2. Try a different URL or the parent page; enable JS rendering / actions via scrapeOptions if the content requires interaction.
3. Use crawlUrl instead if the single-page scrape is brittle.
4. Verify the URL opens in an incognito browser from a different region.

function isProbablyScrapable(url: string): boolean {
  try {
    const u = new URL(url)
    return (u.protocol === 'http:' || u.protocol === 'https:') && u.hostname.includes('.')
  } catch { return false }
}
// if (!isProbablyScrapable(url)) throw new Error('URL not scrapable by FireCrawl before calling scrapeUrl')

try {
  const result = await app.scrapeUrl(url, params)
} catch (error) {
  const msg = error instanceof Error ? error.message : String(error)
  if (/Failed to scrape URL\. Error:/.test(msg)) {
    // API-level scrape failure - try alternate options (actions, formats) or a different URL
  }
  throw error
}

When it happens

Trigger: Target URL returned a paywall, login wall, or 4xx/5xx that FireCrawl could not bypass; target blocks FireCrawl's user agent; URL is a JS-rendered SPA that failed to render; URL points at a non-HTML resource FireCrawl will not scrape.

Common situations: Scraping a site behind Cloudflare bot protection; scraping an authenticated page; URL is a PDF or image but formats did not include them; target is geo-blocked from FireCrawl's region.

Related errors

• ${error.message}
• Crawl request failed: ${crawlResponse.error || 'Unknown erro
• Crawl failed: ${error.response.data.error}
• Failed to fetch ${url} from Airtable: ${error.message}, stat
• No relative links found

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/67222813c5426b9f. Report an issue: GitHub.