Playwright extraction failed or blocked, trying fallback…

Error message

Playwright extraction failed or blocked, trying fallback WebFetch... ${err.message}

What it means

Fallback warning in upskill.mjs: Playwright-based extraction of a job posting/source page failed (goto timeout, navigation blocked, page closed, or innerText unavailable). The script logs err.message and proceeds to a validated plain-HTTP WebFetch fallback, which additionally refuses redirects to prevent SSRF (#1851).

Solutions

Install browsers: npx playwright install chromium
Read err.message: TimeoutError → switch waitUntil to 'domcontentloaded' or raise timeout
For bot-blocked pages, set a real user-agent/viewport or use the ATS API path (fetch-jd.mjs) instead of scraping
Verify the fallback fetch produced usable text; expect degraded extraction on purely JS-rendered pages

Example fix

// before
await page.goto(secureUrl, { waitUntil: 'networkidle', timeout: 30000 });
targetText = await page.innerText('body');
// after
await page.goto(secureUrl, { waitUntil: 'domcontentloaded', timeout: 45000 });
await page.waitForTimeout(1500); // let SPA settle without requiring full network idle
targetText = await page.innerText('body').catch(() => '');

Defensive patterns

Strategy: fallback

Validate before calling

import { chromium } from 'playwright';
try { const b = await chromium.launch(); await b.close(); } catch { console.warn('Playwright unavailable; WebFetch fallback will be used (redirects refused)'); }

Type guard

function extractable(page) { return page && !page.isClosed() && typeof page.innerText === 'function'; }

Try / catch

let targetText; try { targetText = await playwrightExtract(secureUrl); } catch (err) { console.warn('Playwright failed, trying fallback...', err.message); targetText = await validatedPlainFetch(secureUrl, { redirect: 'error' }); }

Prevention

Install Playwright browsers in every runtime environment
Use 'domcontentloaded' + settle delay instead of strict 'networkidle' for JS-heavy ATS pages
For bot-protected boards prefer the ATS JSON APIs (fetch-jd.mjs) over scraping
Keep the plain-fetch fallback with redirect:'error' to avoid SSRF via redirects

When it happens

Trigger: page.goto with waitUntil:'networkidle' times out at 30s on JS-heavy or bot-protected pages; navigation blocked by anti-bot challenge; page never reaches network idle; innerText throws on a detached/null body.

Common situations: Headless Chrome blocked by Cloudflare/Datadome on ATS career pages; slow networks making 30s networkidle unreachable; containers without installed Playwright browsers.

Related errors

AI-assisted analysis of santifer/career-ops@aac998c7ed (2026-09-16). Data as JSON: /api/errors/ce7dbcaa7bdd4f7a. Report an issue: GitHub.

Appendix: source

Thrown at upskill.mjs:974

          browser = await chromium.launch({ headless: true });
          const page = await browser.newPage();

          await page.route('**/*', async (route) => {
            const requestUrl = route.request().url();
            try {
              await validateUrlSecurity(requestUrl);
              await route.continue();
            } catch (err) {
              console.error(`Security Violation on Redirect: ${err.message}`);
              await route.abort('blockedbyclient');
              process.exit(1);
            }
          });

          await page.goto(secureUrl, { waitUntil: 'networkidle', timeout: 30000 });
          targetText = await page.innerText('body');
        } catch (err) {
          console.warn('Playwright extraction failed or blocked, trying fallback WebFetch...', err.message);
          try {
            const secureUrl = await validateUrlSecurity(inputSource);
            // validateUrlSecurity only vets the initial URL; a redirect could still
            // steer the fetch at an internal host (SSRF). The Playwright path
            // re-validates per hop, but this plain fetch must refuse redirects
            // outright — fail closed rather than follow an unvetted Location (#1851).
            const res = await fetch(secureUrl, { signal: AbortSignal.timeout(30000), redirect: 'error' });
            if (!res.ok) throw new Error(`HTTP ${res.status} ${res.statusText}`);
            targetText = await res.text();
          } catch (fetchErr) {
            console.error(`Fatal: Failed to fetch JD from URL: ${fetchErr.message}`);
            process.exit(1);
          }
        } finally {
          if (browser) await browser.close();
        }

        // Whitespace-collapse + length-cap the fetched page text. Use compactText

View on GitHub (pinned to aac998c7ed)