FlowiseAI/Flowise · error · Error

Crawl request failed: ${crawlResponse.error || 'Unknown erro

Error message

Crawl request failed: ${crawlResponse.error || 'Unknown error'}

What it means

crawlUrl received HTTP 200 from POST /v1/crawl but the body had success:false. The crawl job was not accepted. The error field (or 'Unknown error' if absent) is surfaced. This fires before any job-status polling begins.

Source

Thrown at packages/components/nodes/documentloaders/FireCrawl/FireCrawl.ts:298

                }
            })
        }

        // Only add scrapeOptions if it has more than just the default values
        if (Object.keys(scrapeOptions).length > 2) {
            validParams.scrapeOptions = scrapeOptions
        }

        try {
            const parameters = {
                ...validParams,
                integration: 'flowise'
            }
            const response: AxiosResponse = await this.postRequest(this.apiUrl + '/v1/crawl', parameters, headers)
            if (response.status === 200) {
                const crawlResponse = response.data as CrawlResponse
                if (!crawlResponse.success) {
                    throw new Error(`Crawl request failed: ${crawlResponse.error || 'Unknown error'}`)
                }

                if (waitUntilDone) {
                    return this.monitorJobStatus(crawlResponse.id, headers, pollInterval)
                } else {
                    return crawlResponse
                }
            } else {
                this.handleError(response, 'start crawl job')
            }
        } catch (error: any) {
            if (error.response?.data?.error) {
                throw new Error(`Crawl failed: ${error.response.data.error}`)
            }
            throw new Error(`Crawl failed: ${error.message}`)
        }

        return { success: false, id: '', url: '' }

View on GitHub (pinned to abe4a8601a)

Solutions

Inspect crawlResponse.error for the exact reason.
Simplify: drop includePaths/excludePaths/maxDepth and retry with a bare URL.
Confirm the account has crawl credits and is not rate-limited (check dashboard).
Ensure the URL uses http or https and is publicly reachable.

Defensive patterns

Strategy: try-catch

Validate before calling

function validateCrawlRequest(url: string, params: any): void {
  try { new URL(url) } catch { throw new Error(`crawlUrl url is invalid: ${url}`) }
  if (!/^https?:/i.test(new URL(url).protocol)) throw new Error('crawlUrl requires http(s) url')
  if (params?.maxDepth !== undefined && (!Number.isFinite(params.maxDepth) || params.maxDepth < 0)) {
    throw new Error('crawlUrl maxDepth must be a non-negative number')
  }
  if (params?.limit !== undefined && (!Number.isFinite(params.limit) || params.limit <= 0)) {
    throw new Error('crawlUrl limit must be positive')
  }
}
// validateCrawlRequest(url, params) before app.crawlUrl(...)

Try / catch

try {
  await app.crawlUrl(url, params, true, pollInterval, idempotencyKey)
} catch (error) {
  const msg = error instanceof Error ? error.message : String(error)
  if (/Crawl request failed:/.test(msg)) {
    // API rejected the crawl - simplify params and retry once
    await app.crawlUrl(url, null, true, pollInterval)
  } else throw error
}

Prevention

Start with a bare URL and no params to confirm crawl acceptance, then add constraints.
Ensure the account has crawl credits and the URL is publicly reachable.
Use an http(s) URL; other schemes are rejected.

When it happens

Trigger: URL is invalid or blocked; maxDepth/limit combination rejected; includePaths/excludePaths produced an empty crawl set; account out of credits or rate-limited at job-creation time; URL scheme unsupported (e.g., ftp://).

Common situations: Passing a non-http(s) URL; overly restrictive includePaths that exclude the seed; Free-tier quota exceeded; URL blocked upstream by FireCrawl policy.

Related errors

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/7442cc2fa3bd3940. Report an issue: GitHub.