{"record":{"id":"ad2398c3b28013b2","repo":"FlowiseAI/Flowise","slug":"spider-failed-to-crawl-url-error-response-err","errorCode":null,"errorMessage":"Spider: Failed to crawl URL. Error: ${response.error}","messagePattern":"Spider: Failed to crawl URL\\. Error: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/nodes/documentloaders/Spider/Spider.ts","lineNumber":57,"sourceCode":"    }\n\n    public async load(): Promise<DocumentInterface[]> {\n        const app = new SpiderApp({ apiKey: this.apiKey })\n        let spiderDocs: any[]\n\n        if (this.mode === 'scrape') {\n            const response = await app.scrapeUrl(this.url, this.params)\n            if (!response.success) {\n                throw new Error(`Spider: Failed to scrape URL. Error: ${response.error}`)\n            }\n            spiderDocs = [response.data]\n        } else if (this.mode === 'crawl') {\n            if (this.params) {\n                this.params.limit = this.limit\n            }\n            const response = await app.crawlUrl(this.url, this.params)\n            if (!response.success) {\n                throw new Error(`Spider: Failed to crawl URL. Error: ${response.error}`)\n            }\n            spiderDocs = response.data\n        } else {\n            throw new Error(`Unrecognized mode '${this.mode}'. Expected one of 'crawl', 'scrape'.`)\n        }\n\n        return spiderDocs.map(\n            (doc) =>\n                new Document({\n                    pageContent: doc.content || '',\n                    metadata: {\n                        ...(this.additionalMetadata || {}),\n                        source: doc.url\n                    }\n                })\n        )\n    }\n}","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/nodes/documentloaders/Spider/Spider.ts#L39-L75","documentation":"Raised in crawl mode when SpiderApp.crawlUrl returns `{ success: false }`. Crawl submits a longer-running job; a structured failure means the job could not be accepted or returned an error payload.","triggerScenarios":"Calling SpiderLoader.load() with mode 'crawl' where the target is unreachable, the crawl limit is invalid, the Spider job is rejected for policy/robots reasons, or the Spider API returns a non-200 handled into success:false.","commonSituations":"Crawling sites that block bots, exceeding concurrent crawl quota, passing a non-numeric or negative limit, or targeting a domain not allowed by the Spider plan.","solutions":["Inspect response.error from the thrown message and fix the cited reason.","Validate the URL and set a reasonable numeric limit before calling crawlUrl.","Confirm crawl quota and allowed domains in the Spider dashboard.","Fall back to scrape mode if only a single page is needed.","Retry transient failures with exponential backoff."],"exampleFix":"// before\nthis.params.limit = this.limit\nconst response = await app.crawlUrl(this.url, this.params)\n\n// after\nconst params = { ...(this.params ?? {}), limit: Number.isFinite(this.limit) ? this.limit : undefined }\nconst response = await app.crawlUrl(this.url, params)","handlingStrategy":"retry","validationCode":"if (!isValidUrl(url)) throw new Error('Invalid crawl URL')\nconst params = { ...(userParams ?? {}) }\nif (limit != null && !Number.isFinite(Number(limit))) throw new Error('limit must be a number')\nif (limit != null) params.limit = Number(limit)","typeGuard":"function isCrawlSuccess(r: unknown): r is { success: true; data: any[] } {\n    return !!r && (r as any).success === true && Array.isArray((r as any).data)\n}","tryCatchPattern":"for (const attempt of [0, 1, 2]) {\n    try {\n        const r = await app.crawlUrl(url, params)\n        if (r.success) { spiderDocs = r.data; break }\n        if (attempt < 2) { await sleep(2 ** attempt * 1000); continue }\n        throw new Error(`Crawl failed: ${r.error}`)\n    } catch (e) { if (attempt === 2) throw e }\n}","preventionTips":["Pre-validate the URL and a numeric limit.","Retry transient crawl failures with exponential backoff.","Handle 409 (conflict) by polling the existing job.","Fall back to scrape mode when only one page is needed."],"tags":["spider","crawl","api","network"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}