FlowiseAI/Flowise · error · Error

Spider: Failed to crawl URL. Error: ${response.error}

Error message

Spider: Failed to crawl URL. Error: ${response.error}

What it means

Raised in crawl mode when SpiderApp.crawlUrl returns `{ success: false }`. Crawl submits a longer-running job; a structured failure means the job could not be accepted or returned an error payload.

Source

Thrown at packages/components/nodes/documentloaders/Spider/Spider.ts:57

    }

    public async load(): Promise<DocumentInterface[]> {
        const app = new SpiderApp({ apiKey: this.apiKey })
        let spiderDocs: any[]

        if (this.mode === 'scrape') {
            const response = await app.scrapeUrl(this.url, this.params)
            if (!response.success) {
                throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)
            }
            spiderDocs = [response.data]
        } else if (this.mode === 'crawl') {
            if (this.params) {
                this.params.limit = this.limit
            }
            const response = await app.crawlUrl(this.url, this.params)
            if (!response.success) {
                throw new Error(`Spider: Failed to crawl URL. Error: ${response.error}`)
            }
            spiderDocs = response.data
        } else {
            throw new Error(`Unrecognized mode '${this.mode}'. Expected one of 'crawl', 'scrape'.`)
        }

        return spiderDocs.map(
            (doc) =>
                new Document({
                    pageContent: doc.content || '',
                    metadata: {
                        ...(this.additionalMetadata || {}),
                        source: doc.url
                    }
                })
        )
    }
}

View on GitHub (pinned to abe4a8601a)

Solutions

1. Inspect response.error from the thrown message and fix the cited reason.
2. Validate the URL and set a reasonable numeric limit before calling crawlUrl.
3. Confirm crawl quota and allowed domains in the Spider dashboard.
4. Fall back to scrape mode if only a single page is needed.
5. Retry transient failures with exponential backoff.

Example fix

// before
this.params.limit = this.limit
const response = await app.crawlUrl(this.url, this.params)

// after
const params = { ...(this.params ?? {}), limit: Number.isFinite(this.limit) ? this.limit : undefined }
const response = await app.crawlUrl(this.url, params)

if (!isValidUrl(url)) throw new Error('Invalid crawl URL')
const params = { ...(userParams ?? {}) }
if (limit != null && !Number.isFinite(Number(limit))) throw new Error('limit must be a number')
if (limit != null) params.limit = Number(limit)

function isCrawlSuccess(r: unknown): r is { success: true; data: any[] } {
    return !!r && (r as any).success === true && Array.isArray((r as any).data)
}

for (const attempt of [0, 1, 2]) {
    try {
        const r = await app.crawlUrl(url, params)
        if (r.success) { spiderDocs = r.data; break }
        if (attempt < 2) { await sleep(2 ** attempt * 1000); continue }
        throw new Error(`Crawl failed: ${r.error}`)
    } catch (e) { if (attempt === 2) throw e }
}

When it happens

Trigger: Calling SpiderLoader.load() with mode 'crawl' where the target is unreachable, the crawl limit is invalid, the Spider job is rejected for policy/robots reasons, or the Spider API returns a non-200 handled into success:false.

Common situations: Crawling sites that block bots, exceeding concurrent crawl quota, passing a non-numeric or negative limit, or targeting a domain not allowed by the Spider plan.

Related errors

• Spider: Failed to scrape URL. Error: ${response.error}
• Crawl failed: ${error.response.data.error}
• Crawl failed: ${error.message}
• Failed to scrape URL. Error: ${responseData.error}
• Failed to ${action}. Status code: ${response.status}. Error:

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/ad2398c3b28013b2. Report an issue: GitHub.