FlowiseAI/Flowise · error · Error

Spider: Failed to crawl URL. Error

Error message

Spider: Failed to crawl URL. Error: ${response.error}

What it means

Raised in crawl mode when SpiderApp.crawlUrl returns `{ success: false }`. Crawl submits a longer-running job; a structured failure means the job could not be accepted or returned an error payload.

Solutions

1. Inspect response.error from the thrown message and fix the cited reason.
2. Validate the URL and set a reasonable numeric limit before calling crawlUrl.
3. Confirm crawl quota and allowed domains in the Spider dashboard.
4. Fall back to scrape mode if only a single page is needed.
5. Retry transient failures with exponential backoff.

Example fix

// before
this.params.limit = this.limit
const response = await app.crawlUrl(this.url, this.params)

// after
const params = { ...(this.params ?? {}), limit: Number.isFinite(this.limit) ? this.limit : undefined }
const response = await app.crawlUrl(this.url, params)

if (!isValidUrl(url)) throw new Error('Invalid crawl URL')
const params = { ...(userParams ?? {}) }
if (limit != null && !Number.isFinite(Number(limit))) throw new Error('limit must be a number')
if (limit != null) params.limit = Number(limit)

function isCrawlSuccess(r: unknown): r is { success: true; data: any[] } {
    return !!r && (r as any).success === true && Array.isArray((r as any).data)
}

for (const attempt of [0, 1, 2]) {
    try {
        const r = await app.crawlUrl(url, params)
        if (r.success) { spiderDocs = r.data; break }
        if (attempt < 2) { await sleep(2 ** attempt * 1000); continue }
        throw new Error(`Crawl failed: ${r.error}`)
    } catch (e) { if (attempt === 2) throw e }
}

When it happens

Trigger: Calling SpiderLoader.load() with mode 'crawl' where the target is unreachable, the crawl limit is invalid, the Spider job is rejected for policy/robots reasons, or the Spider API returns a non-200 handled into success:false.

Common situations: Crawling sites that block bots, exceeding concurrent crawl quota, passing a non-numeric or negative limit, or targeting a domain not allowed by the Spider plan.

Related errors

• Spider: Failed to scrape URL. Error
• Arxiv API error
• Crawl failed
• Crawl failed
• Failed to . Status code: . Error

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/ad2398c3b28013b2. Report an issue: GitHub.

Appendix: source

Thrown at packages/components/nodes/documentloaders/Spider/Spider.ts:57

    }

    public async load(): Promise<DocumentInterface[]> {
        const app = new SpiderApp({ apiKey: this.apiKey })
        let spiderDocs: any[]

        if (this.mode === 'scrape') {
            const response = await app.scrapeUrl(this.url, this.params)
            if (!response.success) {
                throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)
            }
            spiderDocs = [response.data]
        } else if (this.mode === 'crawl') {
            if (this.params) {
                this.params.limit = this.limit
            }
            const response = await app.crawlUrl(this.url, this.params)
            if (!response.success) {
                throw new Error(`Spider: Failed to crawl URL. Error: ${response.error}`)
            }
            spiderDocs = response.data
        } else {
            throw new Error(`Unrecognized mode '${this.mode}'. Expected one of 'crawl', 'scrape'.`)
        }

        return spiderDocs.map(
            (doc) =>
                new Document({
                    pageContent: doc.content || '',
                    metadata: {
                        ...(this.additionalMetadata || {}),
                        source: doc.url
                    }
                })
        )
    }
}

View on GitHub (pinned to abe4a8601a)