{"record":{"id":"656bdbf9490822ec","repo":"santifer/career-ops","slug":"invalid-or-blocked-url-after-redirect-finalreje","errorCode":null,"errorMessage":"Invalid or blocked URL after redirect: ${finalRejected.reason}","messagePattern":"Invalid or blocked URL after redirect: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"batch-evaluate-gemini.mjs","lineNumber":190,"sourceCode":"    throw new Error(`Invalid or blocked URL: ${rejected.reason}`);\n  }\n  \n  const page = await browser.newPage();\n  try {\n    await page.route('**/*', (route) => {\n      const targetUrl = route.request().url();\n      const interceptedRejected = rejectPrivateOrInvalid(targetUrl);\n      if (interceptedRejected) {\n        return route.abort('accessdenied');\n      }\n      return route.continue();\n    });\n\n    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });\n    \n    const finalRejected = rejectPrivateOrInvalid(page.url());\n    if (finalRejected) {\n      throw new Error(`Invalid or blocked URL after redirect: ${finalRejected.reason}`);\n    }\n\n    await page.waitForTimeout(2000); // wait for dynamic content\n    const text = await page.evaluate(() => {\n      document.querySelectorAll('script, style, noscript, iframe, svg, img').forEach(s => s.remove());\n      return document.body.innerText;\n    });\n    return text.trim();\n  } finally {\n    await page.close();\n  }\n}\n\nasync function evaluateWithRetry(jdText, retries = 5) {\n  if (typeof retries !== 'number' || isNaN(retries) || retries < 1) retries = 1;\n  let attempt = 0;\n  let delay = 5000;\n  while (attempt < retries) {","sourceCodeStart":172,"sourceCodeEnd":208,"githubUrl":"https://github.com/santifer/career-ops/blob/60398d6549a46f5266929538af21cfab94badc75/batch-evaluate-gemini.mjs#L172-L208","documentation":"Thrown by the Playwright scraper in batch-evaluate-gemini.mjs after page.goto(). The URL was allowed before navigation and its subresource requests were routed through rejectPrivateOrInvalid, but HTTP redirects happen server-side, so after landing the code re-checks page.url() against the egress guard (liveness-browser.mjs rejectPrivateOrInvalid). If the final URL is a private/loopback/link-local host, a non-http(s) protocol, or unparseable, the error carries the guard's reason (e.g. 'blocked host 10.0.0.5', 'unsupported protocol file:'). This is the SSRF guard working as designed: the queued URL looked public but redirected somewhere forbidden.","triggerScenarios":"A data/pipeline.md entry 301/302-redirects to an internal hostname or private IP (jobs.acme.com -> 10.2.3.4); a shortener resolving to a private address; a redirect chain ending in a non-http(s) scheme; split-horizon DNS where a public name resolves internally on the run host; a queued http://localhost test link.","commonSituations":"Batch runs over many queued URLs where one entry points at a staging/internal ATS host; corporate VPN/DNS rewriting a public careers domain to an internal IP; pasting test URLs pointing at localhost; a dead tracker redirect that falls back to an internal error page.","solutions":["Read the reason embedded in the message: 'blocked host X' means the redirect landed on a private/reserved address, 'unsupported protocol X' means a non-http(s) scheme, 'invalid URL' means the final location did not parse.","Follow the redirect chain yourself (curl -IL <url> or browser devtools) to see the actual final destination.","If the final destination is legitimately public, replace the pipeline entry with that final URL so no redirect is involved and re-run.","If the destination is genuinely internal/localhost, remove or SKIP the entry; the guard is an SSRF control and must not be bypassed.","Re-run the batch: the per-URL try/catch marks this entry failed and processing continues with the remaining URLs."],"exampleFix":"// before (data/pipeline.md entry that redirects to an internal host)\n| 1 | 2026-08-20 | Acme | Senior EE | https://acme.example/track/abc (302 -> http://10.2.3.4/careers) |\n\n// after (queue the final public destination directly)\n| 1 | 2026-08-20 | Acme | Senior EE | https://acme.example/careers/123 |","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// in the batch loop, per URL:\ntry {\n  const jdText = await scrapeUrl(browser, url);\n} catch (err) {\n  if (/Invalid or blocked URL after redirect/.test(err.message)) {\n    // deterministic SSRF-guard outcome: record reason, never auto-retry\n    console.warn(`SKIP ${url}: egress guard -> ${err.message}`);\n    continue; // next pipeline entry\n  }\n  throw err;\n}","preventionTips":["Queue final-destination URLs in data/pipeline.md, not redirect chains or shorteners.","Never queue internal, staging, or localhost URLs - the guard is by design not bypassable.","Treat 'blocked host'/'unsupported protocol' reasons as verdicts about the entry, not bugs to work around."],"tags":["ssrf","redirect","playwright","url-validation","batch-evaluation"],"backgroundTag":"ssrf-blocked-url","analyzedSha":"60398d6549a46f5266929538af21cfab94badc75","analyzedAt":"2026-08-20T23:00:06.764Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}