{"record":{"id":"f3b1284970b4b2e2","repo":"firecrawl/firecrawl","slug":"failed-to-fetch-supported-url-patterns-from-audio","errorCode":null,"errorMessage":"Failed to fetch supported URL patterns from audio service","messagePattern":"Failed to fetch supported URL patterns from audio service","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"apps/api/src/scraper/scrapeURL/transformers/audio.ts","lineNumber":23,"sourceCode":"import { AudioUnsupportedUrlError, throwIfMediaAccessDenied } from \"../error\";\n\n// Downloads can be large (long videos → hundreds of MB), so this is generous —\n// but an unbounded fetch that hangs would consume the whole scrape budget and\n// surface as an opaque timeout rather than a clean failure.\nconst DOWNLOAD_FETCH_TIMEOUT_MS = 240_000;\n\nlet cachedUrlRegex: RegExp | null = null;\nlet cacheTimestamp = 0;\nconst CACHE_TTL_MS = 5 * 60 * 1000;\n\nasync function getSupportedUrlRegex(): Promise<RegExp> {\n  if (cachedUrlRegex && Date.now() - cacheTimestamp < CACHE_TTL_MS) {\n    return cachedUrlRegex;\n  }\n\n  const res = await fetch(`${config.AVGRAB_SERVICE_URL}/supported-urls`);\n  if (!res.ok) {\n    throw new Error(\n      \"Failed to fetch supported URL patterns from audio service\",\n    );\n  }\n\n  const data = await res.json().catch(() => null);\n  if (!data || typeof data.regex !== \"string\") {\n    throw new Error(\"Audio service returned invalid supported URL patterns\");\n  }\n\n  try {\n    cachedUrlRegex = new RegExp(data.regex);\n  } catch {\n    throw new Error(\"Audio service returned invalid supported URL patterns\");\n  }\n  cacheTimestamp = Date.now();\n  return cachedUrlRegex;\n}\n","sourceCodeStart":5,"sourceCodeEnd":41,"githubUrl":"https://github.com/firecrawl/firecrawl/blob/656bffcc2883f1af5befe38766b1ff5f0469993a/apps/api/src/scraper/scrapeURL/transformers/audio.ts#L5-L41","documentation":"Thrown by getSupportedUrlRegex in transformers/audio.ts when the AVGRAB_SERVICE_URL/supported-urls GET returns a non-2xx status. This is the discovery call fetchAudio makes before the actual /download; if it fails, audio extraction cannot proceed because the regex of supported URLs is unavailable. Cached for 5 minutes (CACHE_TTL_MS) so a transient failure may repeat for that window after recovery.","triggerScenarios":"A scrape with formats including 'audio' on a non-lockdown request, AVGRAB_SERVICE_URL is set, and the GET /supported-urls returns 4xx/5xx. Lockdown and unset AVGRAB_SERVICE_URL skip this call. The cache means a single failure is sticky for up to 5 minutes of subsequent audio requests.","commonSituations":"AVGRAB_SERVICE_URL points at a stale or rolling-updating avgrab deployment. Network blip between API and avgrab. Avgrab is overloaded and returning 503 on the lightweight /supported-urls endpoint. Avgrab deployed without the /supported-urls route (older version). DNS resolution flapping.","solutions":["Confirm AVGRAB_SERVICE_URL is correct and the avgrab service is up and answering /supported-urls with 200.","Wait out the 5-minute cache window if the service has just recovered (cachedUrlRegex stays null on failure, so this is only relevant if it cached a stale ok then broke — re-check).","On self-hosted, ensure avgrab image version exposes /supported-urls.","Drop the 'audio' format from the request if avgrab is unavailable, to avoid the discovery call entirely.","Add retry/circuit-breaker around avgrab in your deployment if this recurs."],"exampleFix":"// before\nawait firecrawl.scrapeUrl(url, { formats: ['markdown', 'audio'] });\n\n// after — drop audio when avgrab is unavailable\nconst wantAudio = process.env.AVGRAB_OK === '1';\nawait firecrawl.scrapeUrl(url, {\n  formats: ['markdown', ...(wantAudio ? ['audio'] : [])],\n});","handlingStrategy":"validation","validationCode":"async function avgrabSupportedUrlsReachable(): Promise<boolean> {\n  try {\n    const r = await fetch(`${process.env.AVGRAB_SERVICE_URL}/supported-urls`, { signal: AbortSignal.timeout(3_000) });\n    return r.ok;\n  } catch { return false; }\n}\nif (formats.includes('audio') && !(await avgrabSupportedUrlsReachable())) {\n  formats = formats.filter(f => f !== 'audio');\n}","typeGuard":null,"tryCatchPattern":"try {\n  return await fetchAudio(meta, document);\n} catch (e) {\n  if (/Failed to fetch supported URL patterns/.test(e.message)) {\n    document.warning = 'Audio format unavailable (avgrab unreachable).';\n    return document;\n  }\n  throw e;\n}","preventionTips":["Gate the 'audio' format on avgrab health in your deployment.","Treat audio extraction as optional; degrade to markdown rather than failing the scrape.","Monitor AVGRAB_SERVICE_URL/supported-urls response rate.","Keep avgrab version aligned with the API."],"tags":["audio","avgrab","upstream-service","discovery","config"],"analyzedSha":"656bffcc2883f1af5befe38766b1ff5f0469993a","analyzedAt":"2026-08-12T01:18:00.488Z","schemaVersion":2},"datasetVersion":"2026-08-12T06:17:24.410Z"}