FlowiseAI/Flowise · error · Error

Spider: Failed to scrape URL. Error: ${response.error}

Error message

Spider: Failed to scrape URL. Error: ${response.error}

What it means

Raised in scrape mode after SpiderApp.scrapeUrl returns `{ success: false }`. The Spider API responded but the response wrapper marks the call as failed, and response.error carries the upstream reason.

Source

Thrown at packages/components/nodes/documentloaders/Spider/Spider.ts:48

            throw new Error('Spider API key not set. You can set it as SPIDER_API_KEY in your .env file, or pass it to Spider.')
        }

        this.apiKey = apiKey
        this.url = url
        this.mode = mode
        this.limit = Number(limit)
        this.additionalMetadata = additionalMetadata
        this.params = params
    }

    public async load(): Promise<DocumentInterface[]> {
        const app = new SpiderApp({ apiKey: this.apiKey })
        let spiderDocs: any[]

        if (this.mode === 'scrape') {
            const response = await app.scrapeUrl(this.url, this.params)
            if (!response.success) {
                throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)
            }
            spiderDocs = [response.data]
        } else if (this.mode === 'crawl') {
            if (this.params) {
                this.params.limit = this.limit
            }
            const response = await app.crawlUrl(this.url, this.params)
            if (!response.success) {
                throw new Error(`Spider: Failed to crawl URL. Error: ${response.error}`)
            }
            spiderDocs = response.data
        } else {
            throw new Error(`Unrecognized mode '${this.mode}'. Expected one of 'crawl', 'scrape'.`)
        }

        return spiderDocs.map(
            (doc) =>
                new Document({

View on GitHub (pinned to abe4a8601a)

Solutions

1. Read response.error (it is interpolated) and address the specific upstream reason.
2. Verify the URL is publicly reachable from a browser/curl and includes https://.
3. Check the Spider dashboard for quota/plan limits and upgrade or wait if exhausted.
4. Pass params (proxy, headers) to bypass blocks, or switch to crawl mode.
5. Retry with backoff for transient Spider-side failures.

Example fix

// before
const response = await app.scrapeUrl(this.url, this.params)
if (!response.success) {
    throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)
}

// after
const response = await app.scrapeUrl(this.url, this.params)
if (!response.success) {
    throw new Error(`Spider: Failed to scrape URL '${this.url}'. Error: ${response.error ?? 'unknown'}`)
}

function isValidUrl(u: string): boolean {
    try { const x = new URL(u); return x.protocol === 'http:' || x.protocol === 'https:' } catch { return false }
}
if (!isValidUrl(url)) throw new Error(`Refusing to scrape invalid URL: ${url}`)

function isSpiderScrapeSuccess(r: unknown): r is { success: true; data: any } {
    return !!r && (r as any).success === true && !!(r as any).data
}

try {
    const r = await app.scrapeUrl(url, params)
    if (!r.success) throw new Error(`Scrape failed: ${r.error}`)
} catch (e: any) {
    if (/quota|429|402/i.test(e.message)) await backoffRetry()
    throw e
}

When it happens

Trigger: Calling SpiderLoader.load() with mode 'scrape' against a URL the Spider service cannot fetch — blocked target, DNS failure on Spider's side, robots.txt disallow, malformed URL, or rate-limit/quota where scrapeUrl returns a structured failure instead of an HTTP error.

Common situations: Targets behind Cloudflare/login walls, localhost or private IPs the Spider cluster cannot reach, quota exhaustion on the Spider plan, or a URL missing the scheme.

Related errors

• Spider: Failed to crawl URL. Error: ${response.error}
• Failed to scrape URL. Error: ${responseData.error}
• Failed to ${action}. Status code: ${response.status}. Error:
• Unexpected error occurred while trying to ${action}. Status
• Arxiv API error: ${response.status} ${response.statusText}

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/902e56a8dd58f342. Report an issue: GitHub.