{"record":{"id":"902e56a8dd58f342","repo":"FlowiseAI/Flowise","slug":"spider-failed-to-scrape-url-error-response-er","errorCode":null,"errorMessage":"Spider: Failed to scrape URL. Error: ${response.error}","messagePattern":"Spider: Failed to scrape URL\\. Error: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/nodes/documentloaders/Spider/Spider.ts","lineNumber":48,"sourceCode":"            throw new Error('Spider API key not set. You can set it as SPIDER_API_KEY in your .env file, or pass it to Spider.')\n        }\n\n        this.apiKey = apiKey\n        this.url = url\n        this.mode = mode\n        this.limit = Number(limit)\n        this.additionalMetadata = additionalMetadata\n        this.params = params\n    }\n\n    public async load(): Promise<DocumentInterface[]> {\n        const app = new SpiderApp({ apiKey: this.apiKey })\n        let spiderDocs: any[]\n\n        if (this.mode === 'scrape') {\n            const response = await app.scrapeUrl(this.url, this.params)\n            if (!response.success) {\n                throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)\n            }\n            spiderDocs = [response.data]\n        } else if (this.mode === 'crawl') {\n            if (this.params) {\n                this.params.limit = this.limit\n            }\n            const response = await app.crawlUrl(this.url, this.params)\n            if (!response.success) {\n                throw new Error(`Spider: Failed to crawl URL. Error: ${response.error}`)\n            }\n            spiderDocs = response.data\n        } else {\n            throw new Error(`Unrecognized mode '${this.mode}'. Expected one of 'crawl', 'scrape'.`)\n        }\n\n        return spiderDocs.map(\n            (doc) =>\n                new Document({","sourceCodeStart":30,"sourceCodeEnd":66,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/nodes/documentloaders/Spider/Spider.ts#L30-L66","documentation":"Raised in scrape mode after SpiderApp.scrapeUrl returns `{ success: false }`. The Spider API responded but the response wrapper marks the call as failed, and response.error carries the upstream reason.","triggerScenarios":"Calling SpiderLoader.load() with mode 'scrape' against a URL the Spider service cannot fetch — blocked target, DNS failure on Spider's side, robots.txt disallow, malformed URL, or rate-limit/quota where scrapeUrl returns a structured failure instead of an HTTP error.","commonSituations":"Targets behind Cloudflare/login walls, localhost or private IPs the Spider cluster cannot reach, quota exhaustion on the Spider plan, or a URL missing the scheme.","solutions":["Read response.error (it is interpolated) and address the specific upstream reason.","Verify the URL is publicly reachable from a browser/curl and includes https://.","Check the Spider dashboard for quota/plan limits and upgrade or wait if exhausted.","Pass params (proxy, headers) to bypass blocks, or switch to crawl mode.","Retry with backoff for transient Spider-side failures."],"exampleFix":"// before\nconst response = await app.scrapeUrl(this.url, this.params)\nif (!response.success) {\n    throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)\n}\n\n// after\nconst response = await app.scrapeUrl(this.url, this.params)\nif (!response.success) {\n    throw new Error(`Spider: Failed to scrape URL '${this.url}'. Error: ${response.error ?? 'unknown'}`)\n}","handlingStrategy":"try-catch","validationCode":"function isValidUrl(u: string): boolean {\n    try { const x = new URL(u); return x.protocol === 'http:' || x.protocol === 'https:' } catch { return false }\n}\nif (!isValidUrl(url)) throw new Error(`Refusing to scrape invalid URL: ${url}`)","typeGuard":"function isSpiderScrapeSuccess(r: unknown): r is { success: true; data: any } {\n    return !!r && (r as any).success === true && !!(r as any).data\n}","tryCatchPattern":"try {\n    const r = await app.scrapeUrl(url, params)\n    if (!r.success) throw new Error(`Scrape failed: ${r.error}`)\n} catch (e: any) {\n    if (/quota|429|402/i.test(e.message)) await backoffRetry()\n    throw e\n}","preventionTips":["Validate URL shape and scheme before calling scrape.","Retry on transient per-item failures with bounded backoff.","Pass render_js/proxy params for hostile targets.","Monitor quota in the Spider dashboard to avoid structured failures."],"tags":["spider","scrape","api","network"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}