{"record":{"id":"57035461541c7f70","repo":"koala73/worldmonitor","slug":"firecrawl-extraction-cooldown-is-open-for-canon","errorCode":null,"errorMessage":"Firecrawl extraction cooldown is open for \"${canonicalName}\"","messagePattern":"Firecrawl extraction cooldown is open for \"(.+?)\"","errorType":"exception","errorClass":"SearchTargetError","httpStatus":null,"severity":"warning","filePath":"consumer-prices-core/src/adapters/search.ts","lineNumber":653,"sourceCode":"            statusCode: 200,\n            fetchedAt: new Date(),\n          };\n        }\n        ctx.logger.warn(\n          `  [search:pin] ${ctx.config.slug}/${canonicalName}: pin extraction failed (${formatExtractionFailures(attempt.failures)}), falling back to Exa`,\n        );\n      } catch (err) {\n        const detail = err instanceof Error ? err.message : String(err);\n        failures.push({ provider: 'firecrawl', reason: 'provider-error', detail });\n        ctx.logger.warn(`  [search:pin] ${ctx.config.slug}/${canonicalName}: pin fetch error, falling back to Exa: ${err}`);\n      }\n    }\n\n    // Half-open (#6182): while the skip window lasts, the target fails fast on\n    // provider-cooldown; the attempt after the window proceeds as the recovery\n    // probe (the probe call itself happens inside _extractFromUrl).\n    if (ctx.config.searchConfig?.extractionFallback !== 'exa' && this.firecrawlGate.consumeSkip()) {\n      throw new SearchTargetError(\n        `Firecrawl extraction cooldown is open for \"${canonicalName}\"`,\n        0,\n        [{ provider: 'firecrawl', reason: 'provider-cooldown' }],\n      );\n    }\n\n    // Only the DISCOVERY cooldown can abort the target: Exa is the sole URL\n    // discovery provider, so without it there is nothing to extract from. An\n    // Exa *extraction* cooldown must not abort — Firecrawl is the primary\n    // extractor and is frequently healthy at that moment (its own streak resets\n    // on every success), and `_extractFromUrl` already skips the cooled-down\n    // provider per candidate. Aborting here would turn a fallback outage into\n    // a whole-basket loss, which is the COVERAGE_PARTIAL this adapter exists\n    // to prevent.\n    if (this.exaDiscoveryGate.consumeSkip()) {\n      throw new SearchTargetError(\n        `Exa discovery cooldown is open for \"${canonicalName}\"`,\n        0,","sourceCodeStart":635,"sourceCodeEnd":671,"githubUrl":"https://github.com/koala73/worldmonitor/blob/eeab0a219fce0f02a00603b532dbae9041b934ac/consumer-prices-core/src/adapters/search.ts#L635-L671","documentation":"Not a provider failure — this SearchTargetError is the Firecrawl extraction cooldown gate doing its job. After consecutive Firecrawl extraction failures, firecrawlGate enters half-open: while the bounded skip window lasts, each target fails fast (durationMs 0, failures [{provider:'firecrawl', reason:'provider-cooldown'}]) instead of burning a doomed API call, and the attempt after the window acts as the recovery probe inside _extractFromUrl. The abort is suppressed entirely when searchConfig.extractionFallback === 'exa'.","triggerScenarios":"Two consecutive Firecrawl provider-side extraction failures (the 'Firecrawl extract error' shape) open the gate; every subsequent target then throws this until the skip window elapses; a failing recovery probe re-opens the window for another round.","commonSituations":"Firecrawl quota exhausted mid-scrape so a whole retailer run logs one provider-cooldown per item; operators reading these as item-level failures and double-counting them in coverage reports; confusion about why Firecrawl calls stopped entirely during an outage.","solutions":["Recognize it as flow control: count the item as provider-skipped, not as a missing price","Fix the upstream Firecrawl condition (quota/rate — see the data.error from the failures that opened the gate)","If Firecrawl outages are routine for this retailer, set searchConfig.extractionFallback = 'exa' to bypass the abort","Let the half-open probe run — do not force calls inside the skip window"],"exampleFix":"// before — every cooldown abort logged as an item failure\n} catch (err) { failures.push({ item: name, reason: err.message }); }\n\n// after — classify provider-cooldown separately\n} catch (err) {\n  if (err instanceof SearchTargetError && err.failures.some((f) => f.reason === 'provider-cooldown')) {\n    skippedForCooldown++; continue; // flow control, not a data outcome\n  }\n  failures.push({ item: name, reason: err.message });\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"function isProviderCooldown(err: unknown): boolean {\n  return err instanceof SearchTargetError\n    && err.failures.some((f) => f.reason === 'provider-cooldown');\n}","tryCatchPattern":"try {\n  await adapter.fetchTarget(ctx, target);\n} catch (err) {\n  if (isProviderCooldown(err)) {\n    stats.cooldownSkips++; // flow control: do NOT count as item failure or outage streak\n    continue;\n  }\n  throw err;\n}","preventionTips":["Classify SearchTargetError by failures[].reason — 'provider-cooldown' means skip, 'provider-error' means investigate","Never retry a provider-cooldown target immediately; the gate's half-open probe decides when to retest","Alert on the '[search:provider-cooldown]' warn line — it marks the moment the gate opened, with the causing error"],"tags":["circuit-breaker","cooldown","firecrawl","flow-control","search-adapter"],"backgroundTag":"circuit-breaker-open","analyzedSha":"eeab0a219fce0f02a00603b532dbae9041b934ac","analyzedAt":"2026-08-21T16:51:25.751Z","schemaVersion":2},"datasetVersion":"2026-08-23T21:17:23.414Z"}