koala73/worldmonitor · error

Firecrawl scrape failed: HTTP ${resp.status}

Error message

Firecrawl scrape failed: HTTP ${resp.status}

What it means

FirecrawlProvider.fetch POSTs to /v1/scrape and throws on any non-2xx before parsing the body. The common causes map to status codes: 401 bad or missing API key in headers(), 402 credits exhausted, and timeout-class statuses when a heavy page exceeds the (timeout + 5s) abort budget.

Source

Thrown at consumer-prices-core/src/acquisition/firecrawl.ts:75

      'User-Agent': 'worldmonitor-consumer-prices/1.0',
    };
  }

  async fetch(url: string, opts: FetchOptions = {}): Promise<FetchResult> {
    const resp = await fetch(`${this.baseUrl}/scrape`, {
      method: 'POST',
      headers: this.headers(),
      body: JSON.stringify({
        url,
        formats: ['html', 'markdown'],
        waitFor: opts.waitForSelector ? 2000 : 0,
        timeout: opts.timeout ?? 30_000,
        headers: opts.headers,
      }),
      signal: AbortSignal.timeout((opts.timeout ?? 30_000) + 5_000),
    });

    if (!resp.ok) throw new Error(`Firecrawl scrape failed: HTTP ${resp.status}`);

    const data = (await resp.json()) as FirecrawlScrapeResponse;
    if (!data.success || !data.data) {
      throw new Error(`Firecrawl error: ${data.error ?? 'unknown'}`);
    }

    return {
      url,
      html: data.data.html ?? '',
      markdown: data.data.markdown ?? '',
      statusCode: 200,
      provider: this.name,
      fetchedAt: new Date(),
      metadata: data.data.metadata,
    };
  }

  async search(query: string, opts: SearchOptions = {}): Promise<SearchResult[]> {

View on GitHub (pinned to eeab0a219f)

Solutions

1. Verify the Firecrawl API key and that credits remain (dashboard billing)
2. Raise opts.timeout and consider waitForSelector for render-heavy pages
3. Confirm the URL is well-formed and publicly reachable
4. On persistent 5xx, check the Firecrawl status page and retry later

Example fix

// before
await firecrawl.fetch(url);                       // HTTP 408/504 on a heavy storefront

// after — give the render a real budget
await firecrawl.fetch(url, { timeout: 60_000, waitForSelector: '.price' });

// Preflight key and give heavy pages a real budget
if (!process.env.FIRECRAWL_API_KEY) throw new Error('FIRECRAWL_API_KEY not set');
const opts = isHeavyStorefront(url) ? { timeout: 60_000, waitForSelector: '.price' } : {};

try {
  return await firecrawl.fetch(url, opts);
} catch (err) {
  const m = /HTTP (\d+)/.exec(err.message);
  const status = m ? Number(m[1]) : 0;
  if (status === 401 || status === 402) throw new ConfigError(err.message);   // key/credits
  if (status === 408 || status >= 500) return retryWithBackoff(() => firecrawl.fetch(url, { ...opts, timeout: (opts.timeout ?? 30_000) * 2 }));
  throw err;
}

When it happens

Trigger: Invalid or unset Firecrawl API key; out-of-credit account; slow SPA storefront exceeding opts.timeout (default 30s) plus the 5s abort margin; a malformed target URL rejected by the API.

Common situations: Trial keys expiring mid-scrape; quota drained by a large batch; JavaScript-heavy product pages needing more than 30s to render.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Firecrawl search failed: HTTP ${resp.status}
• Exa ${endpoint.slice(1)} failed HTTP ${response.status}: ${d
• Firecrawl error: ${data.error ?? 'unknown'}
• Firecrawl extract failed: HTTP ${resp.status}
• ${label} HTTP ${response.status}

AI-assisted analysis of koala73/worldmonitor@eeab0a219f (2026-08-21). Data as JSON: /api/errors/87ec0a28e8f605e4. Report an issue: GitHub.