koala73/worldmonitor · error

Firecrawl extract error: ${data.error ?? 'unknown'}

Error message

Firecrawl extract error: ${data.error ?? 'unknown'}

What it means

FirecrawlProvider.extract() throws this when POST https://api.firecrawl.dev/v1/scrape (formats: ['extract','markdown']) returns HTTP 200 but the body carries success:false — a provider-side failure such as rate limiting, exhausted quota, or a rejected request. Per the code comment this is deliberately a transport condition: the caller's ProviderCooldownGate counts it toward the per-scrape Firecrawl cooldown. A page that simply has nothing to extract is NOT this error; that path returns success:true with an empty extract so the caller records 'missing-price' without penalizing the provider.

Source

Thrown at consumer-prices-core/src/acquisition/firecrawl.ts:168

        // Late-hydrating storefronts capture as a breadcrumb shell without a
        // settle delay; the abort deadline below must absorb it too.
        ...(opts.waitFor ? { waitFor: opts.waitFor } : {}),
      }),
      signal: AbortSignal.timeout(extractAbortMs(opts.timeout) + (opts.waitFor ?? 0)),
    });

    if (!resp.ok) throw new Error(`Firecrawl extract failed: HTTP ${resp.status}`);

    const data = (await resp.json()) as FirecrawlExtractResponse;
    // Throw ONLY for a provider-side failure (quota exhausted, rate limited,
    // bad request) — those are transport conditions the caller's cooldown
    // should count. A successful call that simply found nothing to extract is
    // a PAGE-level outcome: return empty so the caller records `missing-price`
    // and moves to the next candidate URL without accruing an outage streak.
    // Conflating the two lets two ordinary no-product pages disable Firecrawl
    // for the rest of the scrape, on every retailer, not just opted-in ones.
    if (!data.success) {
      throw new Error(`Firecrawl extract error: ${data.error ?? 'unknown'}`);
    }

    return {
      url,
      data: (data.data?.extract ?? {}) as T,
      provider: this.name,
      fetchedAt: new Date(),
      ...(typeof data.data?.markdown === 'string' && data.data.markdown.trim()
        ? { pageContent: data.data.markdown }
        : {}),
    };
  }

  async validate(): Promise<boolean> {
    try {
      const resp = await fetch(`${this.baseUrl}/scrape`, {
        method: 'POST',
        headers: this.headers(),

View on GitHub (pinned to eeab0a219f)

Solutions

1. Read the interpolated data.error text — 'rate limit' means back off, 'credits/quota' means billing, a schema complaint means fix the ExtractSchema
2. Verify the Firecrawl plan's remaining credits and rate limits in the dashboard
3. If rate-limited, wait out the cooldown window — the gate half-opens and probes automatically; do not force calls inside it
4. Confirm the schema encoding: Firecrawl accepts JSON Schema type:[T,'null'] unions for nullable fields (Exa must use anyOf instead — see the #6182 warning in firecrawl.ts)
5. If Firecrawl outages are routine for this retailer, set searchConfig.extractionFallback = 'exa' so the cooldown abort is bypassed

Example fix

// before — treats an empty extract as a provider failure
const res = await firecrawl.extract(url, schema);
if (!Object.keys(res.data).length) throw new Error('provider broke');

// after — empty extract is a page-level outcome; only !success throws
try {
  const res = await firecrawl.extract(url, schema);
  if (!Object.keys(res.data).length) recordOutcome('missing-price'); // keep going, no outage streak
} catch (err) {
  gate.recordFailure(); // provider-side failure: let the cooldown count it
}

// before the scrape run: fail fast on a missing key
if (!process.env.FIRECRAWL_API_KEY) {
  throw new Error('FIRECRAWL_API_KEY not set — every Firecrawl extract will throw');
}

function isFirecrawlExtractError(err: unknown): boolean {
  return err instanceof Error && err.message.startsWith('Firecrawl extract error:');
}

try {
  const res = await provider.extract(url, schema);
  if (!Object.keys(res.data).length) recordOutcome('missing-price'); // page-level, NOT an error
} catch (err) {
  if (isFirecrawlExtractError(err)) {
    // provider-side failure: let the cooldown gate absorb it, retry on the half-open probe
    return await retryLater(() => provider.extract(url, schema));
  }
  throw err;
}

When it happens

Trigger: Calling extract() with a valid key while the Firecrawl plan is out of credits or rate-limited (the error string arrives in-band with HTTP 200); sending an extract payload Firecrawl rejects after acceptance; Firecrawl internal errors surfaced as success:false instead of a non-2xx status.

Common situations: A scrape run works for hours, then every Firecrawl extract fails after a quota or billing change; two consecutive provider-side errors open the adapter's firecrawlGate cooldown so the whole retailer scrape starts failing fast; a schema change (nullable-field encoding) that Firecrawl rejects in-band.

Related errors

• P0 scrape failed: HTTP ${resp.status}
• Firecrawl error: ${data.error ?? 'unknown'}
• Webhook URL must not point to a metadata endpoint
• Exa returned no content for ${url}
• Exa ${endpoint.slice(1)} failed HTTP ${response.status}: ${d

AI-assisted analysis of koala73/worldmonitor@eeab0a219f (2026-08-21). Data as JSON: /api/errors/311f8b539ee88617. Report an issue: GitHub.