{"record":{"id":"532ee5b5b382eb0d","repo":"koala73/worldmonitor","slug":"unsafe-source-url","errorCode":"UNSAFE_SOURCE_URL","errorMessage":"UNSAFE_SOURCE_URL","messagePattern":"UNSAFE_SOURCE_URL","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scripts/cross-strait-activity/adapters.mjs","lineNumber":1686,"sourceCode":"  }\n  return Buffer.concat(chunks, total).toString('utf8');\n}\n\nfunction boundedHtmlRequestInit(sourceContract) {\n  return {\n    headers: {\n      Accept: 'text/html,application/xhtml+xml;q=0.9,*/*;q=0.1',\n      'Accept-Language': 'en',\n      'User-Agent': USER_AGENT,\n    },\n    redirect: sourceContract.redirectPolicy,\n    signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),\n  };\n}\n\nasync function fetchBoundedTextWithStatus(fetchFn, url, sourceContract, diagnostic = null) {\n  if (!isAllowedSourceUrl(url, sourceContract)) {\n    throw new Error('UNSAFE_SOURCE_URL');\n  }\n  let response;\n  try {\n    response = await fetchFn(url, boundedHtmlRequestInit(sourceContract));\n  } catch (error) {\n    if (diagnostic?.transport === 'proxy') {\n      const details = error?.proxyFailure;\n      diagnostic.stage = ['proxy_connection', 'proxy_connect', 'target_tls', 'response_headers', 'response_body']\n        .includes(details?.stage) ? details.stage : 'unknown';\n      diagnostic.httpStatus = diagnostic.stage === 'response_body'\n        && Number.isInteger(details?.httpStatus) && details.httpStatus >= 100 && details.httpStatus <= 599\n        ? details.httpStatus : null;\n      diagnostic.proxyConnectStatus = ['proxy_connect', 'target_tls'].includes(diagnostic.stage)\n        && Number.isInteger(details?.proxyConnectStatus) && details.proxyConnectStatus >= 100 && details.proxyConnectStatus <= 599\n        ? details.proxyConnectStatus : null;\n    }\n    throw error;\n  }","sourceCodeStart":1668,"sourceCodeEnd":1704,"githubUrl":"https://github.com/koala73/worldmonitor/blob/7d06c8633d256c18e38133030bc3613976a96ec9/scripts/cross-strait-activity/adapters.mjs#L1668-L1704","documentation":"fetchBoundedTextWithStatus performs an SSRF/safe-fetch preflight before any network call. Every outbound URL fetched on behalf of a cross-strait source must pass isAllowedSourceUrl(url, sourceContract), which enforces the per-source allowlist (host, scheme, path) declared in CROSS_STRAIT_SOURCE_CONTRACTS. If the URL falls outside that contract, the fetch is refused with UNSAFE_SOURCE_URL instead of issuing an unapproved request.","triggerScenarios":"Called with a url that fails isAllowedSourceUrl against the given sourceContract — e.g. a href parsed from a Japan MOD index page that points to a different host or a non-allowlisted path, a relative URL resolved against the wrong shadowIndexUrl, or a caller passing a URL belonging to another source's contract.","commonSituations":"Upstream site redesigns its index so extracted document links land on a CDN/subdomain not in the contract allowlist; a developer adds a new index URL but forgets to extend the source contract; test fixtures pass a synthetic URL that the contract rejects; redirects are followed manually into disallowed hosts.","solutions":["Log the offending URL and check it against the relevant contract in CROSS_STRAIT_SOURCE_CONTRACTS (scripts/cross-strait-activity/adapters.mjs) to see which host/path/scheme rule it violates.","If the upstream legitimately moved hosts or paths, update the source contract (allowlisted hosts, path patterns, shadowIndexUrl) to include it rather than bypassing the check.","If the URL is malformed or attacker-influenced (parsed from HTML), fix the extraction/resolution so only contract-conformant URLs are fetched, and let disallowed ones be skipped.","Do not weaken isAllowedSourceUrl; add a narrowly scoped allowlist entry if a new genuine source endpoint is required."],"exampleFix":"// before: fetching every href extracted from the index\nfor (const href of hrefs) {\n  const text = await fetchBoundedText(fetchFn, new URL(href, base).href, contract);\n}\n\n// after: only fetch URLs the source contract allows\nfor (const href of hrefs) {\n  const url = new URL(href, contract.shadowIndexUrl);\n  if (!isAllowedSourceUrl(url.href, contract)) continue; // skip disallowed link\n  const text = await fetchBoundedText(fetchFn, url.href, contract);\n}","handlingStrategy":"try-catch","validationCode":"// Pre-check before calling the fetch helper\nconst url = new URL(candidateHref, contract.shadowIndexUrl);\nif (!isAllowedSourceUrl(url.href, contract)) {\n  skipOrReport(candidateHref); // don't call fetchBoundedTextWithStatus\n}","typeGuard":"function isContractSafeUrl(href, contract) {\n  let url;\n  try { url = new URL(href, contract.shadowIndexUrl); } catch { return false; }\n  return typeof url.href === 'string' && isAllowedSourceUrl(url.href, contract);\n}","tryCatchPattern":"try {\n  const { text } = await fetchBoundedTextWithStatus(fetchFn, url, contract);\n  return text;\n} catch (error) {\n  if (error?.message === 'UNSAFE_SOURCE_URL') {\n    logger.warn({ url }, 'source URL rejected by contract');\n    return null; // skip this link\n  }\n  throw error;\n}","preventionTips":["Always resolve extracted hrefs against the contract's shadowIndexUrl before fetching.","Run a fixture-based test asserting every URL the extractor produces passes isAllowedSourceUrl.","When upstream sites change, diff extracted URLs against the contract allowlist in CI.","Never bypass or loosen isAllowedSourceUrl to make a fetch succeed; extend the contract explicitly."],"tags":["ssrf","url-validation","allowlist","fetch"],"backgroundTag":"invalid-url","analyzedSha":"7d06c8633d256c18e38133030bc3613976a96ec9","analyzedAt":"2026-09-15T16:44:39.439Z","contentChangedAt":"2026-09-15T16:44:39.439Z","schemaVersion":2},"datasetVersion":"2026-09-15T18:17:12.389Z"}