koala73/worldmonitor · error

UNSAFE_SOURCE_URL

UNSAFE_SOURCE_URL

Error message

UNSAFE_SOURCE_URL

What it means

fetchBoundedTextWithStatus performs an SSRF/safe-fetch preflight before any network call. Every outbound URL fetched on behalf of a cross-strait source must pass isAllowedSourceUrl(url, sourceContract), which enforces the per-source allowlist (host, scheme, path) declared in CROSS_STRAIT_SOURCE_CONTRACTS. If the URL falls outside that contract, the fetch is refused with UNSAFE_SOURCE_URL instead of issuing an unapproved request.

Solutions

1. Log the offending URL and check it against the relevant contract in CROSS_STRAIT_SOURCE_CONTRACTS (scripts/cross-strait-activity/adapters.mjs) to see which host/path/scheme rule it violates.
2. If the upstream legitimately moved hosts or paths, update the source contract (allowlisted hosts, path patterns, shadowIndexUrl) to include it rather than bypassing the check.
3. If the URL is malformed or attacker-influenced (parsed from HTML), fix the extraction/resolution so only contract-conformant URLs are fetched, and let disallowed ones be skipped.
4. Do not weaken isAllowedSourceUrl; add a narrowly scoped allowlist entry if a new genuine source endpoint is required.

Example fix

// before: fetching every href extracted from the index
for (const href of hrefs) {
  const text = await fetchBoundedText(fetchFn, new URL(href, base).href, contract);
}

// after: only fetch URLs the source contract allows
for (const href of hrefs) {
  const url = new URL(href, contract.shadowIndexUrl);
  if (!isAllowedSourceUrl(url.href, contract)) continue; // skip disallowed link
  const text = await fetchBoundedText(fetchFn, url.href, contract);
}

// Pre-check before calling the fetch helper
const url = new URL(candidateHref, contract.shadowIndexUrl);
if (!isAllowedSourceUrl(url.href, contract)) {
  skipOrReport(candidateHref); // don't call fetchBoundedTextWithStatus
}

function isContractSafeUrl(href, contract) {
  let url;
  try { url = new URL(href, contract.shadowIndexUrl); } catch { return false; }
  return typeof url.href === 'string' && isAllowedSourceUrl(url.href, contract);
}

try {
  const { text } = await fetchBoundedTextWithStatus(fetchFn, url, contract);
  return text;
} catch (error) {
  if (error?.message === 'UNSAFE_SOURCE_URL') {
    logger.warn({ url }, 'source URL rejected by contract');
    return null; // skip this link
  }
  throw error;
}

When it happens

Trigger: Called with a url that fails isAllowedSourceUrl against the given sourceContract — e.g. a href parsed from a Japan MOD index page that points to a different host or a non-allowlisted path, a relative URL resolved against the wrong shadowIndexUrl, or a caller passing a URL belonging to another source's contract.

Common situations: Upstream site redesigns its index so extracted document links land on a CDN/subdomain not in the contract allowlist; a developer adds a new index URL but forgets to extend the source contract; test fixtures pass a synthetic URL that the contract rejects; redirects are followed manually into disallowed hosts.

Understand the failure class

Background: "Invalid URL" / "URL cannot be empty": fix the malformed or missing URL behind request-construction failures — this error's family across 50 libraries.

Related errors

• URL protocol not allowed
• Redirect to disallowed domain
• ${operation} HTTP ${status}: ${safeCode}
• DNS ${recordType} lookup failed: HTTP ${response.status}
• callbackUrl is not a valid URL

AI-assisted analysis of koala73/worldmonitor@7d06c8633d (2026-09-15). Data as JSON: /api/errors/532ee5b5b382eb0d. Report an issue: GitHub.

Appendix: source

Thrown at scripts/cross-strait-activity/adapters.mjs:1686

  }
  return Buffer.concat(chunks, total).toString('utf8');
}

function boundedHtmlRequestInit(sourceContract) {
  return {
    headers: {
      Accept: 'text/html,application/xhtml+xml;q=0.9,*/*;q=0.1',
      'Accept-Language': 'en',
      'User-Agent': USER_AGENT,
    },
    redirect: sourceContract.redirectPolicy,
    signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
  };
}

async function fetchBoundedTextWithStatus(fetchFn, url, sourceContract, diagnostic = null) {
  if (!isAllowedSourceUrl(url, sourceContract)) {
    throw new Error('UNSAFE_SOURCE_URL');
  }
  let response;
  try {
    response = await fetchFn(url, boundedHtmlRequestInit(sourceContract));
  } catch (error) {
    if (diagnostic?.transport === 'proxy') {
      const details = error?.proxyFailure;
      diagnostic.stage = ['proxy_connection', 'proxy_connect', 'target_tls', 'response_headers', 'response_body']
        .includes(details?.stage) ? details.stage : 'unknown';
      diagnostic.httpStatus = diagnostic.stage === 'response_body'
        && Number.isInteger(details?.httpStatus) && details.httpStatus >= 100 && details.httpStatus <= 599
        ? details.httpStatus : null;
      diagnostic.proxyConnectStatus = ['proxy_connect', 'target_tls'].includes(diagnostic.stage)
        && Number.isInteger(details?.proxyConnectStatus) && details.proxyConnectStatus >= 100 && details.proxyConnectStatus <= 599
        ? details.proxyConnectStatus : null;
    }
    throw error;
  }

View on GitHub (pinned to 7d06c8633d)