Failed to fetch supported URL patterns from audio service

Error message

Failed to fetch supported URL patterns from audio service

What it means

Thrown by getSupportedUrlRegex in transformers/audio.ts when the AVGRAB_SERVICE_URL/supported-urls GET returns a non-2xx status. This is the discovery call fetchAudio makes before the actual /download; if it fails, audio extraction cannot proceed because the regex of supported URLs is unavailable. Cached for 5 minutes (CACHE_TTL_MS) so a transient failure may repeat for that window after recovery.

Source

Thrown at apps/api/src/scraper/scrapeURL/transformers/audio.ts:23

import { AudioUnsupportedUrlError, throwIfMediaAccessDenied } from "../error";

// Downloads can be large (long videos → hundreds of MB), so this is generous —
// but an unbounded fetch that hangs would consume the whole scrape budget and
// surface as an opaque timeout rather than a clean failure.
const DOWNLOAD_FETCH_TIMEOUT_MS = 240_000;

let cachedUrlRegex: RegExp | null = null;
let cacheTimestamp = 0;
const CACHE_TTL_MS = 5 * 60 * 1000;

async function getSupportedUrlRegex(): Promise<RegExp> {
  if (cachedUrlRegex && Date.now() - cacheTimestamp < CACHE_TTL_MS) {
    return cachedUrlRegex;
  }

  const res = await fetch(`${config.AVGRAB_SERVICE_URL}/supported-urls`);
  if (!res.ok) {
    throw new Error(
      "Failed to fetch supported URL patterns from audio service",
    );
  }

  const data = await res.json().catch(() => null);
  if (!data || typeof data.regex !== "string") {
    throw new Error("Audio service returned invalid supported URL patterns");
  }

  try {
    cachedUrlRegex = new RegExp(data.regex);
  } catch {
    throw new Error("Audio service returned invalid supported URL patterns");
  }
  cacheTimestamp = Date.now();
  return cachedUrlRegex;
}

View on GitHub (pinned to 656bffcc28)

Solutions

Confirm AVGRAB_SERVICE_URL is correct and the avgrab service is up and answering /supported-urls with 200.
Wait out the 5-minute cache window if the service has just recovered (cachedUrlRegex stays null on failure, so this is only relevant if it cached a stale ok then broke — re-check).
On self-hosted, ensure avgrab image version exposes /supported-urls.
Drop the 'audio' format from the request if avgrab is unavailable, to avoid the discovery call entirely.
Add retry/circuit-breaker around avgrab in your deployment if this recurs.

Example fix

// before
await firecrawl.scrapeUrl(url, { formats: ['markdown', 'audio'] });

// after — drop audio when avgrab is unavailable
const wantAudio = process.env.AVGRAB_OK === '1';
await firecrawl.scrapeUrl(url, {
  formats: ['markdown', ...(wantAudio ? ['audio'] : [])],
});

Defensive patterns

Strategy: validation

Validate before calling

async function avgrabSupportedUrlsReachable(): Promise<boolean> {
  try {
    const r = await fetch(`${process.env.AVGRAB_SERVICE_URL}/supported-urls`, { signal: AbortSignal.timeout(3_000) });
    return r.ok;
  } catch { return false; }
}
if (formats.includes('audio') && !(await avgrabSupportedUrlsReachable())) {
  formats = formats.filter(f => f !== 'audio');
}

Try / catch

try {
  return await fetchAudio(meta, document);
} catch (e) {
  if (/Failed to fetch supported URL patterns/.test(e.message)) {
    document.warning = 'Audio format unavailable (avgrab unreachable).';
    return document;
  }
  throw e;
}

Prevention

Gate the 'audio' format on avgrab health in your deployment.
Treat audio extraction as optional; degrade to markdown rather than failing the scrape.
Monitor AVGRAB_SERVICE_URL/supported-urls response rate.
Keep avgrab version aligned with the API.

When it happens

Trigger: A scrape with formats including 'audio' on a non-lockdown request, AVGRAB_SERVICE_URL is set, and the GET /supported-urls returns 4xx/5xx. Lockdown and unset AVGRAB_SERVICE_URL skip this call. The cache means a single failure is sticky for up to 5 minutes of subsequent audio requests.

Common situations: AVGRAB_SERVICE_URL points at a stale or rolling-updating avgrab deployment. Network blip between API and avgrab. Avgrab is overloaded and returning 503 on the lightweight /supported-urls endpoint. Avgrab deployed without the /supported-urls route (older version). DNS resolution flapping.

Related errors

AI-assisted analysis of firecrawl/firecrawl@656bffcc28 (2026-08-12). Data as JSON: /api/errors/f3b1284970b4b2e2. Report an issue: GitHub.