FlowiseAI/Flowise · error · Error

Crawl job failed

Error message

Crawl job failed

What it means

Thrown by monitorJobStatus during crawl polling when the /v1/crawl/<id> status response reports `status: 'failed'`. This is FireCrawl saying the crawl itself could not complete; no partial data is returned.

Source

Thrown at packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts:491

    private getRequest(url: string, headers: AxiosRequestHeaders): Promise<AxiosResponse> {
        return secureAxiosRequest({ method: 'GET', url, headers })
    }

    private async monitorJobStatus(jobId: string, headers: AxiosRequestHeaders, checkInterval: number): Promise<CrawlStatusResponse> {
        let isJobCompleted = false
        while (!isJobCompleted) {
            const statusResponse: AxiosResponse = await this.getRequest(this.apiUrl + `/v1/crawl/${jobId}`, headers)
            if (statusResponse.status === 200) {
                const statusData = statusResponse.data as CrawlStatusResponse
                switch (statusData.status) {
                    case 'completed':
                        isJobCompleted = true
                        return statusData
                    case 'scraping':
                    case 'failed':
                        if (statusData.status === 'failed') {
                            throw new Error('Crawl job failed')
                        }
                        await new Promise((resolve) => setTimeout(resolve, Math.max(checkInterval, 2) * 1000))
                        break
                    default:
                        throw new Error(`Unknown crawl status: ${statusData.status}`)
                }
            } else {
                this.handleError(statusResponse, 'check crawl status')
            }
        }
        throw new Error('Failed to monitor job status')
    }

    private async monitorExtractStatus(jobId: string, headers: AxiosRequestHeaders, checkInterval: number): Promise<ExtractStatusResponse> {
        let isJobCompleted = false
        while (!isJobCompleted) {
            const statusResponse: AxiosResponse = await this.getRequest(this.apiUrl + `/v1/extract/${jobId}`, headers)
            if (statusResponse.status === 200) {

View on GitHub (pinned to abe4a8601a)

Solutions

1. Re-run with scrapeOptions allowing more lenient behavior (skipTlsVerification, larger timeout).
2. Try the same URL in 'scrape' mode to confirm FireCrawl can fetch a single page.
3. Check FireCrawl dashboard logs for the job id to see the per-page error.
4. Respect robots.txt or switch to a different source URL.

Example fix

// before
const response = await app.crawlUrl(this.url, this.params)

// after
const response = await app.crawlUrl(this.url, {
  ...this.params,
  scrapeOptions: { ...(this.params?.scrapeOptions || {}), skipTlsVerification: true, timeout: 60000 }
})

if (!url) throw new Error('crawl requires a url')
try { new URL(url) } catch { throw new Error(`invalid url: ${url}`) }

function isCrawlStatusFailed(s: unknown): boolean {
  return typeof s === 'object' && s !== null && (s as any).status === 'failed'
}

try {
  return await app.crawlUrl(url, params)
} catch (e) {
  if (/Crawl job failed/.test((e as Error).message)) {
    const fb = await app.scrapeUrl(url, params) // degrade to single page
    return fb.success ? fb.data : null
  }
  throw e
}

When it happens

Trigger: statusData.status === 'failed' on a 200 response from the crawl-status endpoint. Causes: target site blocks the crawler, returns only errors, robots.txt disallow, JS-only page that times out, or FireCrawl internal failure.

Common situations: Target site behind Cloudflare/WAF; robots.txt disallows the path; target returns 4xx/5xx for all pages; crawl budget exhausted mid-run; FireCrawl-side incident.

Related errors

• Extract job failed
• Firecrawl: Crawl job failed
• Crawl request failed: ${crawlResponse.error || 'Unknown erro
• Crawl failed: ${error.response.data.error}
• Crawl failed: ${error.message}

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/30f419c9b959b74e. Report an issue: GitHub.