FlowiseAI/Flowise · error · Error

Spider: Failed to scrape URL. Error

Error message

Spider: Failed to scrape URL. Error: ${response.error}

What it means

Raised in scrape mode after SpiderApp.scrapeUrl returns `{ success: false }`. The Spider API responded but the response wrapper marks the call as failed, and response.error carries the upstream reason.

Solutions

1. Read response.error (it is interpolated) and address the specific upstream reason.
2. Verify the URL is publicly reachable from a browser/curl and includes https://.
3. Check the Spider dashboard for quota/plan limits and upgrade or wait if exhausted.
4. Pass params (proxy, headers) to bypass blocks, or switch to crawl mode.
5. Retry with backoff for transient Spider-side failures.

Example fix

// before
const response = await app.scrapeUrl(this.url, this.params)
if (!response.success) {
    throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)
}

// after
const response = await app.scrapeUrl(this.url, this.params)
if (!response.success) {
    throw new Error(`Spider: Failed to scrape URL '${this.url}'. Error: ${response.error ?? 'unknown'}`)
}

function isValidUrl(u: string): boolean {
    try { const x = new URL(u); return x.protocol === 'http:' || x.protocol === 'https:' } catch { return false }
}
if (!isValidUrl(url)) throw new Error(`Refusing to scrape invalid URL: ${url}`)

function isSpiderScrapeSuccess(r: unknown): r is { success: true; data: any } {
    return !!r && (r as any).success === true && !!(r as any).data
}

try {
    const r = await app.scrapeUrl(url, params)
    if (!r.success) throw new Error(`Scrape failed: ${r.error}`)
} catch (e: any) {
    if (/quota|429|402/i.test(e.message)) await backoffRetry()
    throw e
}

When it happens

Trigger: Calling SpiderLoader.load() with mode 'scrape' against a URL the Spider service cannot fetch — blocked target, DNS failure on Spider's side, robots.txt disallow, malformed URL, or rate-limit/quota where scrapeUrl returns a structured failure instead of an HTTP error.

Common situations: Targets behind Cloudflare/login walls, localhost or private IPs the Spider cluster cannot reach, quota exhaustion on the Spider plan, or a URL missing the scheme.

Related errors

• Spider: Failed to crawl URL. Error
• Arxiv API error
• Failed to . Status code: . Error
• Failed to scrape URL. Error
• Unexpected error occurred while trying to

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/902e56a8dd58f342. Report an issue: GitHub.

Appendix: source

Thrown at packages/components/nodes/documentloaders/Spider/Spider.ts:48

            throw new Error('Spider API key not set. You can set it as SPIDER_API_KEY in your .env file, or pass it to Spider.')
        }

        this.apiKey = apiKey
        this.url = url
        this.mode = mode
        this.limit = Number(limit)
        this.additionalMetadata = additionalMetadata
        this.params = params
    }

    public async load(): Promise<DocumentInterface[]> {
        const app = new SpiderApp({ apiKey: this.apiKey })
        let spiderDocs: any[]

        if (this.mode === 'scrape') {
            const response = await app.scrapeUrl(this.url, this.params)
            if (!response.success) {
                throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)
            }
            spiderDocs = [response.data]
        } else if (this.mode === 'crawl') {
            if (this.params) {
                this.params.limit = this.limit
            }
            const response = await app.crawlUrl(this.url, this.params)
            if (!response.success) {
                throw new Error(`Spider: Failed to crawl URL. Error: ${response.error}`)
            }
            spiderDocs = response.data
        } else {
            throw new Error(`Unrecognized mode '${this.mode}'. Expected one of 'crawl', 'scrape'.`)
        }

        return spiderDocs.map(
            (doc) =>
                new Document({

View on GitHub (pinned to abe4a8601a)