koala73/worldmonitor · error

Firecrawl error: ${data.error ?? 'unknown'}

Error message

Firecrawl error: ${data.error ?? 'unknown'}

What it means

The scrape HTTP call succeeded (2xx) but the FirecrawlScrapeResponse body reports success:false or lacks data — a provider-level failure delivered in-band. The embedded data.error names the cause ('unknown' when absent), typically a failed render, an upstream block, or a provider-internal error.

Source

Thrown at consumer-prices-core/src/acquisition/firecrawl.ts:79

  async fetch(url: string, opts: FetchOptions = {}): Promise<FetchResult> {
    const resp = await fetch(`${this.baseUrl}/scrape`, {
      method: 'POST',
      headers: this.headers(),
      body: JSON.stringify({
        url,
        formats: ['html', 'markdown'],
        waitFor: opts.waitForSelector ? 2000 : 0,
        timeout: opts.timeout ?? 30_000,
        headers: opts.headers,
      }),
      signal: AbortSignal.timeout((opts.timeout ?? 30_000) + 5_000),
    });

    if (!resp.ok) throw new Error(`Firecrawl scrape failed: HTTP ${resp.status}`);

    const data = (await resp.json()) as FirecrawlScrapeResponse;
    if (!data.success || !data.data) {
      throw new Error(`Firecrawl error: ${data.error ?? 'unknown'}`);
    }

    return {
      url,
      html: data.data.html ?? '',
      markdown: data.data.markdown ?? '',
      statusCode: 200,
      provider: this.name,
      fetchedAt: new Date(),
      metadata: data.data.metadata,
    };
  }

  async search(query: string, opts: SearchOptions = {}): Promise<SearchResult[]> {
    const resp = await fetch(`${this.baseUrl}/search`, {
      method: 'POST',
      headers: this.headers(),
      body: JSON.stringify({

View on GitHub (pinned to eeab0a219f)

Solutions

1. Read the embedded data.error string — it names the provider-side cause
2. Retry with waitForSelector and/or a higher timeout for render-dependent pages
3. Route persistently blocked domains to a different provider (Exa) or the relay path
4. If data.error mentions quota/credits, top up the account

Example fix

// before
if (!data.success || !data.data) {
  throw new Error(`Firecrawl error: ${data.error ?? 'unknown'}`);
}

// after — keep the provider error, but make it classifiable for cooldown logic
if (!data.success || !data.data) {
  const err = new Error(`Firecrawl error: ${data.error ?? 'unknown'}`) as Error & { providerError?: string };
  err.providerError = data.error ?? 'unknown';
  throw err;
}

interface FirecrawlScrapeResponse { success?: boolean; data?: unknown; error?: string }
function isProviderFailure(body: FirecrawlScrapeResponse | undefined): body is { success: false; error?: string } {
  return !body?.success || body?.data === undefined;
}

try {
  return await firecrawl.fetch(url);
} catch (err) {
  if (err instanceof Error && err.message.startsWith('Firecrawl error:')) {
    // in-band provider failure: consult data.error for the cause; render issues are retryable with waitFor
    if (/block|403|forbidden/i.test(err.message)) return routeToAlternativeProvider(url);
    return retryOnce(() => firecrawl.fetch(url, { waitForSelector: 'body' }));
  }
  throw err;
}

When it happens

Trigger: Target site blocks Firecrawl (bot protection, JS challenge, 403 at origin); the page render times out inside Firecrawl while the HTTP envelope still returns 200; a provider incident producing error bodies; quota-related in-band errors.

Common situations: Hardened retailer sites during price scraping; regional blocks; intermittent render failures on specific templates.

Related errors

• Exa returned no content for ${url}
• Firecrawl scrape failed: HTTP ${resp.status}
• Firecrawl extract error: ${data.error ?? 'unknown'}
• Firecrawl search failed: HTTP ${resp.status}
• Firecrawl extract failed: HTTP ${resp.status}

AI-assisted analysis of koala73/worldmonitor@eeab0a219f (2026-08-21). Data as JSON: /api/errors/65bda32426dbc2ba. Report an issue: GitHub.