What causes "Firecrawl extract failed: HTTP ${resp.status}" in koala73/worldmonitor?

Raise opts.timeout and set opts.waitFor so the budget covers storefront hydration

Firecrawl extract failed: HTTP

Error message

Firecrawl extract failed: HTTP ${resp.status}

What it means

FirecrawlProvider.extract POSTs to /v1/scrape with formats ['extract','markdown'] plus a JSON schema, and throws on any non-2xx with the status. The abort budget deliberately adds opts.waitFor on top of extractAbortMs(timeout) because late-hydrating storefronts need the settle delay — so slow extractions surface here as timeout-class statuses, distinct from the in-band `Firecrawl extract error` reserved for provider-side failures (an extract that simply finds nothing returns empty instead).

Solutions

Raise opts.timeout and set opts.waitFor so the budget covers storefront hydration
Verify key and credits on 401/402
Simplify the extract schema if the validator rejects it (400)
Classify correctly: this is a transport failure for cooldown counting; a page with nothing to extract is not an error

Example fix

// before — budget too small for a late-hydrating storefront
await firecrawl.extract(url, schema, { timeout: 15_000 });   // HTTP 408/504

// after — extend both knobs; the abort deadline absorbs waitFor
await firecrawl.extract(url, schema, { timeout: 45_000, waitFor: 3_000 });

Defensive patterns

Strategy: retry

Validate before calling

// Budget the call up front: abort deadline = extractAbortMs(timeout) + waitFor
const opts = { timeout: slowStorefront(url) ? 45_000 : 30_000, waitFor: slowStorefront(url) ? 3_000 : 0 };

Try / catch

try {
  return await firecrawl.extract(url, schema, opts);
} catch (err) {
  const m = /HTTP (\d+)/.exec(err.message);
  const status = m ? Number(m[1]) : 0;
  if (status === 408 || status === 504) {
    return firecrawl.extract(url, schema, { ...opts, timeout: (opts.timeout ?? 30_000) * 2, waitFor: (opts.waitFor ?? 0) + 2000 });
  }
  throw err;   // 401/402/400 are not timeout-budget issues
}

Prevention

Set opts.waitFor for late-hydrating storefronts so the render settles inside the budget
Scale timeout to page weight instead of using the 30s default everywhere
Remember the contract: transport failures throw here; a page with nothing to extract returns empty and must not count against provider cooldowns

When it happens

Trigger: 408/504 when extraction exceeds the timeout+waitFor abort budget on slow storefronts; 401/402 key or credit problems; 400 for a schema the extract validator rejects; provider 5xx incidents.

Common situations: Late-hydrating SPA product pages where the price appears seconds after load; complex extraction schemas; quota exhausted mid-run.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of koala73/worldmonitor@a96956387a (2026-08-21). Data as JSON: /api/errors/0f2897f8639b34e6. Report an issue: GitHub.

Appendix: source

Thrown at consumer-prices-core/src/acquisition/firecrawl.ts:160

    const resp = await fetch(`${this.baseUrl}/scrape`, {
      method: 'POST',
      headers: this.headers(),
      body: JSON.stringify({
        url,
        // markdown rides along in the same render so the caller can verify an
        // extracted price actually appears on the page (price-evidence.ts).
        formats: ['extract', 'markdown'],
        extract: { schema: jsonSchema, ...(schema.prompt ? { prompt: schema.prompt } : {}) },
        timeout: opts.timeout ?? 30_000,
        // Late-hydrating storefronts capture as a breadcrumb shell without a
        // settle delay; the abort deadline below must absorb it too.
        ...(opts.waitFor ? { waitFor: opts.waitFor } : {}),
      }),
      signal: AbortSignal.timeout(extractAbortMs(opts.timeout) + (opts.waitFor ?? 0)),
    });

    if (!resp.ok) throw new Error(`Firecrawl extract failed: HTTP ${resp.status}`);

    const data = (await resp.json()) as FirecrawlExtractResponse;
    // Throw ONLY for a provider-side failure (quota exhausted, rate limited,
    // bad request) — those are transport conditions the caller's cooldown
    // should count. A successful call that simply found nothing to extract is
    // a PAGE-level outcome: return empty so the caller records `missing-price`
    // and moves to the next candidate URL without accruing an outage streak.
    // Conflating the two lets two ordinary no-product pages disable Firecrawl
    // for the rest of the scrape, on every retailer, not just opted-in ones.
    if (!data.success) {
      throw new Error(`Firecrawl extract error: ${data.error ?? 'unknown'}`);
    }

    return {
      url,
      data: (data.data?.extract ?? {}) as T,
      provider: this.name,
      fetchedAt: new Date(),

View on GitHub (pinned to a96956387a)