{"record":{"id":"9f2add13c3b38222","repo":"santifer/career-ops","slug":"refusing-non-http-s-url-url","errorCode":null,"errorMessage":"Refusing non-HTTP(S) URL: ${url}","messagePattern":"Refusing non-HTTP\\(S\\) URL: (.+?)","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"openrouter-runner.mjs","lineNumber":411,"sourceCode":"    'CV (Markdown):',\n    ctx.cv,\n    '---',\n    'OUTPUT LANGUAGE:',\n    languageInstruction,\n  ].filter(Boolean).join('\\n\\n');\n}\n\n// ---------------------------------------------------------------------------\n// Job page content fetcher (Playwright-first, plain fetch fallback)\n// ---------------------------------------------------------------------------\n// Reject unsafe fetch targets (SSRF defense-in-depth): http(s) only, never\n// loopback / link-local / private / cloud-metadata hosts. URLs come from the\n// user's own portals.yml / pipeline.md, but we still fail closed.\nfunction assertSafeRemoteUrl(url) {\n  let u;\n  try { u = new URL(url); } catch { throw new Error(`Invalid URL: ${url}`); }\n  if (u.protocol !== 'https:' && u.protocol !== 'http:') {\n    throw new Error(`Refusing non-HTTP(S) URL: ${url}`);\n  }\n  const host = u.hostname.toLowerCase();\n  const blocked = host === 'localhost' || host === '::1' || host.endsWith('.local') ||\n    /^127\\./.test(host) || /^10\\./.test(host) || /^192\\.168\\./.test(host) ||\n    /^169\\.254\\./.test(host) || /^172\\.(1[6-9]|2\\d|3[01])\\./.test(host);\n  if (blocked) throw new Error(`Refusing private/loopback host: ${host}`);\n  return u;\n}\n\nasync function fetchJobPage(url) {\n  assertSafeRemoteUrl(url);\n  let chromium;\n  try {\n    ({ chromium } = await import('playwright'));\n  } catch {\n    console.warn('[fetch] Playwright unavailable — falling back to plain fetch.');\n  }\n","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/openrouter-runner.mjs#L393-L429","documentation":"The second gate in assertSafeRemoteUrl: the string parses as a URL but its protocol is neither https: nor http:. The library only fetches web pages over HTTP(S), so schemes like file:, ftp:, data:, or javascript: are rejected outright as part of SSRF fail-closed hardening.","triggerScenarios":"fetchJobPage receives a URL whose protocol check fails — e.g. 'file:///home/user/job.html', 'ftp://example.com/posting', 'data:text/html,...', or a config entry like 'localhost.job' with a typo'd scheme such as 'https//example.com' (which parses with protocol 'https:/' malformed... actually parses as scheme 'https' with invalid rest) or 'webcal://...'.","commonSituations":"Someone pointed a pipeline entry at a locally saved HTML file with file://, an internal tool emitted a custom-scheme deep link, or a copy-paste from a desktop app produced a non-web scheme.","solutions":["Replace the entry with an https:// URL pointing at the live posting","If you have a local copy of the JD, don't fetch it as a URL — save it as a jds/ capture (e.g. via archive-posting.mjs) and reference local:jds/{file}","Correct typos in the scheme (http// → https://) so the URL parses with a valid protocol","Remove ftp:/data:/javascript: style entries from portals.yml / pipeline.md entirely"],"exampleFix":"// before (pipeline.md / portals.yml)\nurl: file:///tmp/job-posting.html\n\n// after\nurl: https://jobs.acme.com/postings/12345","handlingStrategy":"validation","validationCode":"function isWebUrl(s) {\n  try { const u = new URL(s); return u.protocol === 'https:' || u.protocol === 'http:'; }\n  catch { return false; }\n}\n// reject file://, ftp://, data:, etc. before calling fetchJobPage","typeGuard":"function hasHttpScheme(v) {\n  if (typeof v !== 'string') return false;\n  const m = v.match(/^https?:\\/\\//i);\n  return m !== null;\n}","tryCatchPattern":"try {\n  const text = await fetchJobPage(url);\n} catch (e) {\n  if (e.message.startsWith('Refusing non-HTTP(S) URL')) {\n    console.error(`Unsupported scheme for ${url} — use an https:// link or a local jds/ capture`);\n    return null;\n  }\n  throw e;\n}","preventionTips":["Store local JD copies as jds/ captures referenced via local:jds/{file}, not as file:// URLs","Correct scheme typos immediately (http// → https://) when editing pipeline entries","Sanitize links pasted from desktop apps that emit custom schemes (webcal:, slack:, etc.)","Add a config lint that asserts every careers_url entry starts with https://"],"tags":["url","ssrf","protocol","validation","security"],"backgroundTag":"invalid-url","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}