{"record":{"id":"30f419c9b959b74e","repo":"FlowiseAI/Flowise","slug":"crawl-job-failed","errorCode":null,"errorMessage":"Crawl job failed","messagePattern":"Crawl job failed","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts","lineNumber":491,"sourceCode":"\n    private getRequest(url: string, headers: AxiosRequestHeaders): Promise<AxiosResponse> {\n        return secureAxiosRequest({ method: 'GET', url, headers })\n    }\n\n    private async monitorJobStatus(jobId: string, headers: AxiosRequestHeaders, checkInterval: number): Promise<CrawlStatusResponse> {\n        let isJobCompleted = false\n        while (!isJobCompleted) {\n            const statusResponse: AxiosResponse = await this.getRequest(this.apiUrl + `/v1/crawl/${jobId}`, headers)\n            if (statusResponse.status === 200) {\n                const statusData = statusResponse.data as CrawlStatusResponse\n                switch (statusData.status) {\n                    case 'completed':\n                        isJobCompleted = true\n                        return statusData\n                    case 'scraping':\n                    case 'failed':\n                        if (statusData.status === 'failed') {\n                            throw new Error('Crawl job failed')\n                        }\n                        await new Promise((resolve) => setTimeout(resolve, Math.max(checkInterval, 2) * 1000))\n                        break\n                    default:\n                        throw new Error(`Unknown crawl status: ${statusData.status}`)\n                }\n            } else {\n                this.handleError(statusResponse, 'check crawl status')\n            }\n        }\n        throw new Error('Failed to monitor job status')\n    }\n\n    private async monitorExtractStatus(jobId: string, headers: AxiosRequestHeaders, checkInterval: number): Promise<ExtractStatusResponse> {\n        let isJobCompleted = false\n        while (!isJobCompleted) {\n            const statusResponse: AxiosResponse = await this.getRequest(this.apiUrl + `/v1/extract/${jobId}`, headers)\n            if (statusResponse.status === 200) {","sourceCodeStart":473,"sourceCodeEnd":509,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts#L473-L509","documentation":"Thrown by monitorJobStatus during crawl polling when the /v1/crawl/<id> status response reports `status: 'failed'`. This is FireCrawl saying the crawl itself could not complete; no partial data is returned.","triggerScenarios":"statusData.status === 'failed' on a 200 response from the crawl-status endpoint. Causes: target site blocks the crawler, returns only errors, robots.txt disallow, JS-only page that times out, or FireCrawl internal failure.","commonSituations":"Target site behind Cloudflare/WAF; robots.txt disallows the path; target returns 4xx/5xx for all pages; crawl budget exhausted mid-run; FireCrawl-side incident.","solutions":["Re-run with scrapeOptions allowing more lenient behavior (skipTlsVerification, larger timeout).","Try the same URL in 'scrape' mode to confirm FireCrawl can fetch a single page.","Check FireCrawl dashboard logs for the job id to see the per-page error.","Respect robots.txt or switch to a different source URL."],"exampleFix":"// before\nconst response = await app.crawlUrl(this.url, this.params)\n\n// after\nconst response = await app.crawlUrl(this.url, {\n  ...this.params,\n  scrapeOptions: { ...(this.params?.scrapeOptions || {}), skipTlsVerification: true, timeout: 60000 }\n})","handlingStrategy":"fallback","validationCode":"if (!url) throw new Error('crawl requires a url')\ntry { new URL(url) } catch { throw new Error(`invalid url: ${url}`) }","typeGuard":"function isCrawlStatusFailed(s: unknown): boolean {\n  return typeof s === 'object' && s !== null && (s as any).status === 'failed'\n}","tryCatchPattern":"try {\n  return await app.crawlUrl(url, params)\n} catch (e) {\n  if (/Crawl job failed/.test((e as Error).message)) {\n    const fb = await app.scrapeUrl(url, params) // degrade to single page\n    return fb.success ? fb.data : null\n  }\n  throw e\n}","preventionTips":["Set a smaller limit/maxDepth on first try to validate reachability.","Keep a scrape-mode fallback for crawl failures.","Check the job id in FireCrawl dashboard for per-page error detail."],"tags":["firecrawl","crawl","polling","job-failed"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-13T19:17:28.613Z"}