FlowiseAI/Flowise · error · Error

Failed to scrape URL. Error: ${responseData.error}

Error message

Failed to scrape URL. Error: ${responseData.error}

What it means

Inside scrapeUrl when the HTTP 200 response's first element reports status falsy. The code reads `responseData.error` for the reason, but the success/failure is actually signalled per-element via `responseData[0].status`, so responseData.error is frequently undefined — yielding 'Error: undefined'.

Source

Thrown at packages/components/nodes/documentloaders/Spider/SpiderApp.ts:67

        this.apiKey = apiKey || ''
        this.apiUrl = apiUrl || 'https://api.spider.cloud/v1'
        if (!this.apiKey) {
            throw new Error('No API key provided')
        }
    }

    async scrapeUrl(url: string, params: Params | null = null): Promise<ScrapeResponse> {
        const headers = this.prepareHeaders()
        const jsonData: Params = { url, limit: 1, ...params }

        try {
            const response: AxiosResponse = await this.postRequest('crawl', jsonData, headers)
            if (response.status === 200) {
                const responseData = response.data
                if (responseData[0].status) {
                    return { success: true, data: responseData[0] }
                } else {
                    throw new Error(`Failed to scrape URL. Error: ${responseData.error}`)
                }
            } else {
                this.handleError(response, 'scrape URL')
            }
        } catch (error: any) {
            throw new Error(error.message)
        }
        return { success: false, error: 'Internal server error.' }
    }

    async crawlUrl(url: string, params: Params | null = null, idempotencyKey?: string): Promise<CrawlResponse | any> {
        const headers = this.prepareHeaders(idempotencyKey)
        const jsonData: Params = { url, ...params }

        try {
            const response: AxiosResponse = await this.postRequest('crawl', jsonData, headers)
            if (response.status === 200) {
                return { success: true, data: response.data }

View on GitHub (pinned to abe4a8601a)

Solutions

1. Read responseData[0] for the actual failure field rather than responseData.error.
2. Retry the specific URL; transient per-item failures often succeed on retry.
3. Adjust params (render_js, proxy) to handle hostile targets.
4. Fix the message to interpolate the correct field (see exampleFix).

Example fix

// before
if (responseData[0].status) {
    return { success: true, data: responseData[0] }
} else {
    throw new Error(`Failed to scrape URL. Error: ${responseData.error}`)
}

// after
const item = responseData[0]
if (item?.status) {
    return { success: true, data: item }
}
throw new Error(`Failed to scrape URL. Error: ${item?.error ?? JSON.stringify(item)}`)

const item = Array.isArray(responseData) ? responseData[0] : undefined
if (!item || !item.status) {
    throw new Error(`Scrape item failed: ${item?.error ?? JSON.stringify(item)}`)
}

interface SpiderScrapeItem { status: boolean; error?: string; content?: string; url?: string }
function isSpiderScrapeItem(v: unknown): v is SpiderScrapeItem {
    return !!v && typeof (v as any).status === 'boolean'
}

try {
    const res = await app.scrapeUrl(url, params)
} catch (e: any) {
    if (/Failed to scrape URL\. Error: undefined/.test(e.message)) {
        // known bug: re-derive the real error from the raw response
    }
    throw e
}

When it happens

Trigger: Spider's /crawl endpoint returns 200 with an array whose first item has status:false — typically a per-URL scrape failure (blocked, timeout, invalid content) while the transport itself succeeded.

Common situations: Scraping a URL that returns 4xx/5xx to Spider, JS-only pages Spider cannot render, target returns empty body, or partial outages where Spider marks individual items failed.

Related errors

• Spider: Failed to scrape URL. Error: ${response.error}
• Spider: Failed to crawl URL. Error: ${response.error}
• Failed to ${action}. Status code: ${response.status}. Error:
• Unexpected error occurred while trying to ${action}. Status
• Spider API key not set. You can set it as SPIDER_API_KEY in

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/49d504bf620f2b82. Report an issue: GitHub.