koala73/worldmonitor · error

Firecrawl search failed: HTTP ${resp.status}

Error message

Firecrawl search failed: HTTP ${resp.status}

What it means

FirecrawlProvider.search POSTs to /v1/search with the query, limit (default 10) and optional includeDomains; any non-2xx throws with the HTTP status. Typical codes: 401 auth failure, 402 credits exhausted, 400 for invalid parameters such as malformed includeDomains entries, and 5xx provider incidents.

Source

Thrown at consumer-prices-core/src/acquisition/firecrawl.ts:105

      provider: this.name,
      fetchedAt: new Date(),
      metadata: data.data.metadata,
    };
  }

  async search(query: string, opts: SearchOptions = {}): Promise<SearchResult[]> {
    const resp = await fetch(`${this.baseUrl}/search`, {
      method: 'POST',
      headers: this.headers(),
      body: JSON.stringify({
        query,
        limit: opts.numResults ?? 10,
        includeDomains: opts.includeDomains,
        scrapeOptions: { formats: ['markdown'] },
      }),
    });

    if (!resp.ok) throw new Error(`Firecrawl search failed: HTTP ${resp.status}`);

    const data = (await resp.json()) as FirecrawlSearchResponse;
    return (data.data ?? []).map((r) => ({
      url: r.url,
      title: r.title,
      text: r.description ?? r.markdown,
    }));
  }

  async extract<T = Record<string, unknown>>(
    url: string,
    schema: ExtractSchema,
    opts: FetchOptions = {},
  ): Promise<ExtractResult<T>> {
    // Nullable is encoded DIFFERENTLY here than in ExaProvider.extract, and the
    // divergence is deliberate — do not "unify" these without re-testing both
    // providers live. Firecrawl accepts the JSON Schema `type: [T,'null']`
    // union (verified: HTTP 200, extract returned). Exa's /contents validator

View on GitHub (pinned to eeab0a219f)

Solutions

1. Check the API key and credit balance first — 401/402 cover most cases
2. Validate query options (non-empty query, well-formed includeDomains) against the current search API
3. Retry with backoff only on 5xx; treat 4xx as a caller-side fix
4. Batch discovery queries and cache results to conserve credits

// Validate search inputs before spending a credit
if (!query.trim()) throw new Error('query must be non-empty');
if (includeDomains?.some((d) => !/^[a-z0-9.-]+\.[a-z]{2,}$/i.test(d))) {
  throw new Error('includeDomains entries must be bare hostnames');
}

try {
  return await firecrawl.search(query, opts);
} catch (err) {
  const m = /HTTP (\d+)/.exec(err.message);
  const status = m ? Number(m[1]) : 0;
  if (status >= 500 || status === 429) return backoffRetry(() => firecrawl.search(query, opts));
  throw err;   // 400/401/402 are caller-side: fix options or credentials
}

When it happens

Trigger: Missing or wrong API key; exhausted credits after search-heavy discovery loops; malformed query options (empty query, invalid domain filters); provider 5xx.

Common situations: Discovery phases firing many searches and draining quota; new options added without checking the current Firecrawl search API contract.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Firecrawl scrape failed: HTTP ${resp.status}
• Exa ${endpoint.slice(1)} failed HTTP ${response.status}: ${d
• Firecrawl extract failed: HTTP ${resp.status}
• ${label} HTTP ${response.status}
• ${operation} HTTP ${status}: ${safeCode}

AI-assisted analysis of koala73/worldmonitor@eeab0a219f (2026-08-21). Data as JSON: /api/errors/e28679167d1f798a. Report an issue: GitHub.