koala73/worldmonitor · warning · SearchTargetError

Firecrawl extraction cooldown is open for "${canonicalName}"

Error message

Firecrawl extraction cooldown is open for "${canonicalName}"

What it means

Not a provider failure — this SearchTargetError is the Firecrawl extraction cooldown gate doing its job. After consecutive Firecrawl extraction failures, firecrawlGate enters half-open: while the bounded skip window lasts, each target fails fast (durationMs 0, failures [{provider:'firecrawl', reason:'provider-cooldown'}]) instead of burning a doomed API call, and the attempt after the window acts as the recovery probe inside _extractFromUrl. The abort is suppressed entirely when searchConfig.extractionFallback === 'exa'.

Source

Thrown at consumer-prices-core/src/adapters/search.ts:653

            statusCode: 200,
            fetchedAt: new Date(),
          };
        }
        ctx.logger.warn(
          `  [search:pin] ${ctx.config.slug}/${canonicalName}: pin extraction failed (${formatExtractionFailures(attempt.failures)}), falling back to Exa`,
        );
      } catch (err) {
        const detail = err instanceof Error ? err.message : String(err);
        failures.push({ provider: 'firecrawl', reason: 'provider-error', detail });
        ctx.logger.warn(`  [search:pin] ${ctx.config.slug}/${canonicalName}: pin fetch error, falling back to Exa: ${err}`);
      }
    }

    // Half-open (#6182): while the skip window lasts, the target fails fast on
    // provider-cooldown; the attempt after the window proceeds as the recovery
    // probe (the probe call itself happens inside _extractFromUrl).
    if (ctx.config.searchConfig?.extractionFallback !== 'exa' && this.firecrawlGate.consumeSkip()) {
      throw new SearchTargetError(
        `Firecrawl extraction cooldown is open for "${canonicalName}"`,
        0,
        [{ provider: 'firecrawl', reason: 'provider-cooldown' }],
      );
    }

    // Only the DISCOVERY cooldown can abort the target: Exa is the sole URL
    // discovery provider, so without it there is nothing to extract from. An
    // Exa *extraction* cooldown must not abort — Firecrawl is the primary
    // extractor and is frequently healthy at that moment (its own streak resets
    // on every success), and `_extractFromUrl` already skips the cooled-down
    // provider per candidate. Aborting here would turn a fallback outage into
    // a whole-basket loss, which is the COVERAGE_PARTIAL this adapter exists
    // to prevent.
    if (this.exaDiscoveryGate.consumeSkip()) {
      throw new SearchTargetError(
        `Exa discovery cooldown is open for "${canonicalName}"`,
        0,

View on GitHub (pinned to eeab0a219f)

Solutions

1. Recognize it as flow control: count the item as provider-skipped, not as a missing price
2. Fix the upstream Firecrawl condition (quota/rate — see the data.error from the failures that opened the gate)
3. If Firecrawl outages are routine for this retailer, set searchConfig.extractionFallback = 'exa' to bypass the abort
4. Let the half-open probe run — do not force calls inside the skip window

Example fix

// before — every cooldown abort logged as an item failure
} catch (err) { failures.push({ item: name, reason: err.message }); }

// after — classify provider-cooldown separately
} catch (err) {
  if (err instanceof SearchTargetError && err.failures.some((f) => f.reason === 'provider-cooldown')) {
    skippedForCooldown++; continue; // flow control, not a data outcome
  }
  failures.push({ item: name, reason: err.message });
}

function isProviderCooldown(err: unknown): boolean {
  return err instanceof SearchTargetError
    && err.failures.some((f) => f.reason === 'provider-cooldown');
}

try {
  await adapter.fetchTarget(ctx, target);
} catch (err) {
  if (isProviderCooldown(err)) {
    stats.cooldownSkips++; // flow control: do NOT count as item failure or outage streak
    continue;
  }
  throw err;
}

When it happens

Trigger: Two consecutive Firecrawl provider-side extraction failures (the 'Firecrawl extract error' shape) open the gate; every subsequent target then throws this until the skip window elapses; a failing recovery probe re-opens the window for another round.

Common situations: Firecrawl quota exhausted mid-scrape so a whole retailer run logs one provider-cooldown per item; operators reading these as item-level failures and double-counting them in coverage reports; confusion about why Firecrawl calls stopped entirely during an outage.

Related errors

• Exa discovery cooldown is open for "${canonicalName}"
• Firecrawl scrape failed: HTTP ${resp.status}
• Firecrawl error: ${data.error ?? 'unknown'}
• Firecrawl search failed: HTTP ${resp.status}
• Firecrawl extract failed: HTTP ${resp.status}

AI-assisted analysis of koala73/worldmonitor@eeab0a219f (2026-08-21). Data as JSON: /api/errors/57035461541c7f70. Report an issue: GitHub.