Playwright extraction failed or blocked, trying fallback Web

Error message

Playwright extraction failed or blocked, trying fallback WebFetch...

What it means

upskill.mjs's targeted JD extraction failed in Playwright — chromium.launch failing or page.goto(waitUntil:'networkidle', 30s) timing out / erroring — and the code falls back to a plain fetch with redirect:'error' and a 30s AbortSignal timeout. If the fallback also fails (a redirect is present, non-2xx status, or network error), the script exits fatally. The SSRF posture is deliberate: the Playwright path re-validates every request per hop, while the fallback refuses redirects outright so an unvetted Location header cannot aim the fetch at an internal host (#1851).

Source

Thrown at upskill.mjs:912

          browser = await chromium.launch({ headless: true });
          const page = await browser.newPage();

          await page.route('**/*', async (route) => {
            const requestUrl = route.request().url();
            try {
              await validateUrlSecurity(requestUrl);
              await route.continue();
            } catch (err) {
              console.error(`Security Violation on Redirect: ${err.message}`);
              await route.abort('blockedbyclient');
              process.exit(1);
            }
          });

          await page.goto(secureUrl, { waitUntil: 'networkidle', timeout: 30000 });
          targetText = await page.innerText('body');
        } catch (err) {
          console.warn('Playwright extraction failed or blocked, trying fallback WebFetch...', err.message);
          try {
            const secureUrl = await validateUrlSecurity(inputSource);
            // validateUrlSecurity only vets the initial URL; a redirect could still
            // steer the fetch at an internal host (SSRF). The Playwright path
            // re-validates per hop, but this plain fetch must refuse redirects
            // outright — fail closed rather than follow an unvetted Location (#1851).
            const res = await fetch(secureUrl, { signal: AbortSignal.timeout(30000), redirect: 'error' });
            if (!res.ok) throw new Error(`HTTP ${res.status} ${res.statusText}`);
            targetText = await res.text();
          } catch (fetchErr) {
            console.error(`Fatal: Failed to fetch JD from URL: ${fetchErr.message}`);
            process.exit(1);
          }
        } finally {
          if (browser) await browser.close();
        }

        // Whitespace-collapse + length-cap the fetched page text. Use compactText

View on GitHub (pinned to 60398d6549)

Solutions

Retry once — networkidle timeouts are frequently transient.
Resolve redirects yourself (curl -ILs -o /dev/null -w '%{url_effective}' URL) and pass the FINAL url.
Save the JD to a local file and run node upskill.mjs --url-text /path/to/file — it bypasses fetching entirely.
For sites that never go idle, fetch the page's underlying JSON/API instead of scraping rendered DOM.

Example fix

# before — the http→https redirect makes the fallback fatal (redirect: 'error')
node upskill.mjs --url-text http://example.com/jobs/123
# after — pass the final URL so no redirect occurs
node upskill.mjs --url-text https://example.com/jobs/123

Defensive patterns

Strategy: fallback

Validate before calling

const pre = await fetch(url, { method: 'HEAD', redirect: 'error', signal: AbortSignal.timeout(5000) })
  .catch((e) => e);
if (pre instanceof Error) {
  console.warn('URL redirects or is unreachable — resolve the final URL before running upskill');
}

Try / catch

try {
  targetText = await extractWithPlaywright(url);
} catch (e) {
  console.warn('Playwright extraction failed, trying fallback fetch', e.message);
  const res = await fetch(url, { redirect: 'error', signal: AbortSignal.timeout(30_000) });
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  targetText = await res.text();
} // the fallback must stay redirect-refusing: never follow an unvetted Location

Prevention

Pass final (post-redirect) URLs only
Prefer local JD files (jds/*.md) for upskill — deterministic and offline
Prefer 'domcontentloaded' over 'networkidle' for analytics-heavy pages when you control the code
Keep redirect:'error' on the fallback — it is the SSRF backstop, not an inconvenience

When it happens

Trigger: networkidle never fires on pages with constant analytics pings → 30s goto timeout; ERR_NAME_NOT_RESOLVED or TLS errors; the URL issues a redirect → fallback fetch throws (redirect:'error'); non-2xx status → fallback throws an HTTP error; both paths fail → 'Fatal: Failed to fetch JD from URL' and exit 1.

Common situations: Heavy SPA job boards whose network never goes idle; CDNs redirecting http→https or adding auth hops; proxies injecting redirects; slow origins on constrained networks.

Related errors

AI-assisted analysis of santifer/career-ops@60398d6549 (2026-08-20). Data as JSON: /api/errors/6b3cb9eb81d7bdbe. Report an issue: GitHub.