{"record":{"id":"49d504bf620f2b82","repo":"FlowiseAI/Flowise","slug":"failed-to-scrape-url-error-responsedata-error-49d504","errorCode":null,"errorMessage":"Failed to scrape URL. Error: ${responseData.error}","messagePattern":"Failed to scrape URL\\. Error: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/nodes/documentloaders/Spider/SpiderApp.ts","lineNumber":67,"sourceCode":"        this.apiKey = apiKey || ''\n        this.apiUrl = apiUrl || 'https://api.spider.cloud/v1'\n        if (!this.apiKey) {\n            throw new Error('No API key provided')\n        }\n    }\n\n    async scrapeUrl(url: string, params: Params | null = null): Promise<ScrapeResponse> {\n        const headers = this.prepareHeaders()\n        const jsonData: Params = { url, limit: 1, ...params }\n\n        try {\n            const response: AxiosResponse = await this.postRequest('crawl', jsonData, headers)\n            if (response.status === 200) {\n                const responseData = response.data\n                if (responseData[0].status) {\n                    return { success: true, data: responseData[0] }\n                } else {\n                    throw new Error(`Failed to scrape URL. Error: ${responseData.error}`)\n                }\n            } else {\n                this.handleError(response, 'scrape URL')\n            }\n        } catch (error: any) {\n            throw new Error(error.message)\n        }\n        return { success: false, error: 'Internal server error.' }\n    }\n\n    async crawlUrl(url: string, params: Params | null = null, idempotencyKey?: string): Promise<CrawlResponse | any> {\n        const headers = this.prepareHeaders(idempotencyKey)\n        const jsonData: Params = { url, ...params }\n\n        try {\n            const response: AxiosResponse = await this.postRequest('crawl', jsonData, headers)\n            if (response.status === 200) {\n                return { success: true, data: response.data }","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/nodes/documentloaders/Spider/SpiderApp.ts#L49-L85","documentation":"Inside scrapeUrl when the HTTP 200 response's first element reports status falsy. The code reads `responseData.error` for the reason, but the success/failure is actually signalled per-element via `responseData[0].status`, so responseData.error is frequently undefined — yielding 'Error: undefined'.","triggerScenarios":"Spider's /crawl endpoint returns 200 with an array whose first item has status:false — typically a per-URL scrape failure (blocked, timeout, invalid content) while the transport itself succeeded.","commonSituations":"Scraping a URL that returns 4xx/5xx to Spider, JS-only pages Spider cannot render, target returns empty body, or partial outages where Spider marks individual items failed.","solutions":["Read responseData[0] for the actual failure field rather than responseData.error.","Retry the specific URL; transient per-item failures often succeed on retry.","Adjust params (render_js, proxy) to handle hostile targets.","Fix the message to interpolate the correct field (see exampleFix)."],"exampleFix":"// before\nif (responseData[0].status) {\n    return { success: true, data: responseData[0] }\n} else {\n    throw new Error(`Failed to scrape URL. Error: ${responseData.error}`)\n}\n\n// after\nconst item = responseData[0]\nif (item?.status) {\n    return { success: true, data: item }\n}\nthrow new Error(`Failed to scrape URL. Error: ${item?.error ?? JSON.stringify(item)}`)","handlingStrategy":"type-guard","validationCode":"const item = Array.isArray(responseData) ? responseData[0] : undefined\nif (!item || !item.status) {\n    throw new Error(`Scrape item failed: ${item?.error ?? JSON.stringify(item)}`)\n}","typeGuard":"interface SpiderScrapeItem { status: boolean; error?: string; content?: string; url?: string }\nfunction isSpiderScrapeItem(v: unknown): v is SpiderScrapeItem {\n    return !!v && typeof (v as any).status === 'boolean'\n}","tryCatchPattern":"try {\n    const res = await app.scrapeUrl(url, params)\n} catch (e: any) {\n    if (/Failed to scrape URL\\. Error: undefined/.test(e.message)) {\n        // known bug: re-derive the real error from the raw response\n    }\n    throw e\n}","preventionTips":["Read the per-item error field (responseData[0].error), not responseData.error.","Log the full first element when status is falsy.","Retry per-URL failures independently.","Pin the Spider response schema version you test against."],"tags":["spider","api","response-parsing","bug"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}