{"record":{"id":"ce7dbcaa7bdd4f7a","repo":"santifer/career-ops","slug":"playwright-extraction-failed-or-blocked-trying-fallback","errorCode":null,"errorMessage":"Playwright extraction failed or blocked, trying fallback WebFetch... ${err.message}","messagePattern":"Playwright extraction failed or blocked, trying fallback WebFetch\\.\\.\\. (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"upskill.mjs","lineNumber":974,"sourceCode":"          browser = await chromium.launch({ headless: true });\n          const page = await browser.newPage();\n\n          await page.route('**/*', async (route) => {\n            const requestUrl = route.request().url();\n            try {\n              await validateUrlSecurity(requestUrl);\n              await route.continue();\n            } catch (err) {\n              console.error(`Security Violation on Redirect: ${err.message}`);\n              await route.abort('blockedbyclient');\n              process.exit(1);\n            }\n          });\n\n          await page.goto(secureUrl, { waitUntil: 'networkidle', timeout: 30000 });\n          targetText = await page.innerText('body');\n        } catch (err) {\n          console.warn('Playwright extraction failed or blocked, trying fallback WebFetch...', err.message);\n          try {\n            const secureUrl = await validateUrlSecurity(inputSource);\n            // validateUrlSecurity only vets the initial URL; a redirect could still\n            // steer the fetch at an internal host (SSRF). The Playwright path\n            // re-validates per hop, but this plain fetch must refuse redirects\n            // outright — fail closed rather than follow an unvetted Location (#1851).\n            const res = await fetch(secureUrl, { signal: AbortSignal.timeout(30000), redirect: 'error' });\n            if (!res.ok) throw new Error(`HTTP ${res.status} ${res.statusText}`);\n            targetText = await res.text();\n          } catch (fetchErr) {\n            console.error(`Fatal: Failed to fetch JD from URL: ${fetchErr.message}`);\n            process.exit(1);\n          }\n        } finally {\n          if (browser) await browser.close();\n        }\n\n        // Whitespace-collapse + length-cap the fetched page text. Use compactText","sourceCodeStart":956,"sourceCodeEnd":992,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/upskill.mjs#L956-L992","documentation":"Fallback warning in upskill.mjs: Playwright-based extraction of a job posting/source page failed (goto timeout, navigation blocked, page closed, or innerText unavailable). The script logs err.message and proceeds to a validated plain-HTTP WebFetch fallback, which additionally refuses redirects to prevent SSRF (#1851).","triggerScenarios":"page.goto with waitUntil:'networkidle' times out at 30s on JS-heavy or bot-protected pages; navigation blocked by anti-bot challenge; page never reaches network idle; innerText throws on a detached/null body.","commonSituations":"Headless Chrome blocked by Cloudflare/Datadome on ATS career pages; slow networks making 30s networkidle unreachable; containers without installed Playwright browsers.","solutions":["Install browsers: npx playwright install chromium","Read err.message: TimeoutError → switch waitUntil to 'domcontentloaded' or raise timeout","For bot-blocked pages, set a real user-agent/viewport or use the ATS API path (fetch-jd.mjs) instead of scraping","Verify the fallback fetch produced usable text; expect degraded extraction on purely JS-rendered pages"],"exampleFix":"// before\nawait page.goto(secureUrl, { waitUntil: 'networkidle', timeout: 30000 });\ntargetText = await page.innerText('body');\n// after\nawait page.goto(secureUrl, { waitUntil: 'domcontentloaded', timeout: 45000 });\nawait page.waitForTimeout(1500); // let SPA settle without requiring full network idle\ntargetText = await page.innerText('body').catch(() => '');","handlingStrategy":"fallback","validationCode":"import { chromium } from 'playwright';\ntry { const b = await chromium.launch(); await b.close(); } catch { console.warn('Playwright unavailable; WebFetch fallback will be used (redirects refused)'); }","typeGuard":"function extractable(page) { return page && !page.isClosed() && typeof page.innerText === 'function'; }","tryCatchPattern":"let targetText; try { targetText = await playwrightExtract(secureUrl); } catch (err) { console.warn('Playwright failed, trying fallback...', err.message); targetText = await validatedPlainFetch(secureUrl, { redirect: 'error' }); }","preventionTips":["Install Playwright browsers in every runtime environment","Use 'domcontentloaded' + settle delay instead of strict 'networkidle' for JS-heavy ATS pages","For bot-protected boards prefer the ATS JSON APIs (fetch-jd.mjs) over scraping","Keep the plain-fetch fallback with redirect:'error' to avoid SSRF via redirects"],"tags":["playwright","scraping","timeout","fallback","ssrf"],"backgroundTag":"browser-automation-failed","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}