{"record":{"id":"6b3cb9eb81d7bdbe","repo":"santifer/career-ops","slug":"playwright-extraction-failed-or-blocked-trying-fa","errorCode":null,"errorMessage":"Playwright extraction failed or blocked, trying fallback WebFetch...","messagePattern":"Playwright extraction failed or blocked, trying fallback WebFetch\\.\\.\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"upskill.mjs","lineNumber":912,"sourceCode":"          browser = await chromium.launch({ headless: true });\n          const page = await browser.newPage();\n\n          await page.route('**/*', async (route) => {\n            const requestUrl = route.request().url();\n            try {\n              await validateUrlSecurity(requestUrl);\n              await route.continue();\n            } catch (err) {\n              console.error(`Security Violation on Redirect: ${err.message}`);\n              await route.abort('blockedbyclient');\n              process.exit(1);\n            }\n          });\n\n          await page.goto(secureUrl, { waitUntil: 'networkidle', timeout: 30000 });\n          targetText = await page.innerText('body');\n        } catch (err) {\n          console.warn('Playwright extraction failed or blocked, trying fallback WebFetch...', err.message);\n          try {\n            const secureUrl = await validateUrlSecurity(inputSource);\n            // validateUrlSecurity only vets the initial URL; a redirect could still\n            // steer the fetch at an internal host (SSRF). The Playwright path\n            // re-validates per hop, but this plain fetch must refuse redirects\n            // outright — fail closed rather than follow an unvetted Location (#1851).\n            const res = await fetch(secureUrl, { signal: AbortSignal.timeout(30000), redirect: 'error' });\n            if (!res.ok) throw new Error(`HTTP ${res.status} ${res.statusText}`);\n            targetText = await res.text();\n          } catch (fetchErr) {\n            console.error(`Fatal: Failed to fetch JD from URL: ${fetchErr.message}`);\n            process.exit(1);\n          }\n        } finally {\n          if (browser) await browser.close();\n        }\n\n        // Whitespace-collapse + length-cap the fetched page text. Use compactText","sourceCodeStart":894,"sourceCodeEnd":930,"githubUrl":"https://github.com/santifer/career-ops/blob/60398d6549a46f5266929538af21cfab94badc75/upskill.mjs#L894-L930","documentation":"upskill.mjs's targeted JD extraction failed in Playwright — chromium.launch failing or page.goto(waitUntil:'networkidle', 30s) timing out / erroring — and the code falls back to a plain fetch with redirect:'error' and a 30s AbortSignal timeout. If the fallback also fails (a redirect is present, non-2xx status, or network error), the script exits fatally. The SSRF posture is deliberate: the Playwright path re-validates every request per hop, while the fallback refuses redirects outright so an unvetted Location header cannot aim the fetch at an internal host (#1851).","triggerScenarios":"networkidle never fires on pages with constant analytics pings → 30s goto timeout; ERR_NAME_NOT_RESOLVED or TLS errors; the URL issues a redirect → fallback fetch throws (redirect:'error'); non-2xx status → fallback throws an HTTP error; both paths fail → 'Fatal: Failed to fetch JD from URL' and exit 1.","commonSituations":"Heavy SPA job boards whose network never goes idle; CDNs redirecting http→https or adding auth hops; proxies injecting redirects; slow origins on constrained networks.","solutions":["Retry once — networkidle timeouts are frequently transient.","Resolve redirects yourself (curl -ILs -o /dev/null -w '%{url_effective}' URL) and pass the FINAL url.","Save the JD to a local file and run node upskill.mjs --url-text /path/to/file — it bypasses fetching entirely.","For sites that never go idle, fetch the page's underlying JSON/API instead of scraping rendered DOM."],"exampleFix":"# before — the http→https redirect makes the fallback fatal (redirect: 'error')\nnode upskill.mjs --url-text http://example.com/jobs/123\n# after — pass the final URL so no redirect occurs\nnode upskill.mjs --url-text https://example.com/jobs/123","handlingStrategy":"fallback","validationCode":"const pre = await fetch(url, { method: 'HEAD', redirect: 'error', signal: AbortSignal.timeout(5000) })\n  .catch((e) => e);\nif (pre instanceof Error) {\n  console.warn('URL redirects or is unreachable — resolve the final URL before running upskill');\n}","typeGuard":null,"tryCatchPattern":"try {\n  targetText = await extractWithPlaywright(url);\n} catch (e) {\n  console.warn('Playwright extraction failed, trying fallback fetch', e.message);\n  const res = await fetch(url, { redirect: 'error', signal: AbortSignal.timeout(30_000) });\n  if (!res.ok) throw new Error(`HTTP ${res.status}`);\n  targetText = await res.text();\n} // the fallback must stay redirect-refusing: never follow an unvetted Location","preventionTips":["Pass final (post-redirect) URLs only","Prefer local JD files (jds/*.md) for upskill — deterministic and offline","Prefer 'domcontentloaded' over 'networkidle' for analytics-heavy pages when you control the code","Keep redirect:'error' on the fallback — it is the SSRF backstop, not an inconvenience"],"tags":["playwright","fetch","fallback","timeout","ssrf","redirect"],"backgroundTag":"page-goto-timeout","analyzedSha":"60398d6549a46f5266929538af21cfab94badc75","analyzedAt":"2026-08-20T23:00:06.764Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}