santifer/career-ops · error

Extracted text too short (likely blocked or empty)

Error message

Extracted text too short (likely blocked or empty)

What it means

scrapeUrl() navigated successfully and passed the egress guard, but after removing script/style/noscript/iframe/svg/img nodes the remaining document.body.innerText was falsy or under 100 characters. The page rendered essentially nothing readable. Typical causes are bot walls, cookie-consent interstitials, JS-rendered apps that had not mounted within the fixed 2s waitForTimeout, or a posting already taken down.

Source

Thrown at batch-evaluate-gemini.mjs:243

  }
}

export async function processOffer(browser, line, idx, _evaluate = evaluateWithRetry) {
  const match = line.match(/- \[\s*\]\s+(https?:\/\/\S+)(?:\s*\|\s*([^|]+)\s*\|\s*(.+))?/);
  if (!match) return { line, processed: false };

  const url = match[1];
  let companyHint = match[2] ? match[2].trim() : 'Unknown';
  let titleHint = match[3] ? match[3].trim() : 'Unknown';

  console.log(`\n========================================`);
  console.log(`🔄 Processing [${idx}]: ${companyHint} - ${titleHint}`);
  console.log(`🔗 URL: ${url}`);

  try {
    const jdText = await scrapeUrl(browser, url);
    if (!jdText || jdText.length < 100) {
      throw new Error('Extracted text too short (likely blocked or empty)');
    }

    console.log(`🧠 Calling Gemini (${modelName})...`);
    const evaluationText = await _evaluate(`URL: ${url}\n\n${jdText}`);

    // Parse output
    const summaryMatch = evaluationText.match(/---SCORE_SUMMARY---\s*([\s\S]*?)---END_SUMMARY---/);
    if (!summaryMatch) {
      console.error('Missing SCORE_SUMMARY block from model output:\n' + evaluationText.slice(0, 500));
      throw new Error('Missing SCORE_SUMMARY block from model output');
    }
    
    const block = summaryMatch[1];
    const extract = (key) => {
      const m = block.match(new RegExp(`^\\s*${key}:\\s*(.+)$`, 'mi'));
      return m ? m[1].trim() : 'unknown';
    };

View on GitHub (pinned to 60398d6549)

Solutions

1. Open the same URL in a real browser (or curl with a desktop UA) to tell a bot wall apart from a genuinely empty/gone page.
2. Re-run just that entry: some challenges pass on a second, warmer attempt.
3. If it is a slow SPA, replace the fixed page.waitForTimeout(2000) with waitForSelector on a JD-specific element and a longer timeout.
4. If the host hard-blocks headless browsers, fetch the JD manually, save it under jds/, and evaluate the text directly instead of the URL.
5. If the posting is gone, drop the URL from the pipeline.

Example fix

// before
await page.waitForTimeout(2000); // wait for dynamic content

// after
await page
  .waitForSelector('.job-description, [data-job-description]', { timeout: 15000 })
  .catch(() => page.waitForTimeout(5000));

try {
  const jdText = await scrapeUrl(browser, url);
} catch (err) {
  if (/too short/.test(err.message)) {
    await new Promise(r => setTimeout(r, 5000));
    const retry = await scrapeUrl(browser, url); // one warm retry
    if (!retry || retry.length < 100) throw err;
  } else throw err;
}

When it happens

Trigger: Cloudflare/Akamai 'verify you are human' challenge served instead of the JD; a cookie banner that replaces body content; an SPA careers site whose content mounts after the hard-coded 2s wait; a removed posting returning a near-empty 404 shell; page content living entirely inside elements the cleanup step removes.

Common situations: Batch runs across many portals where one or two ATS hosts bot-block headless Chromium; running on slow/CI machines where client-side rendering exceeds 2s; sites that render only after interaction.

Related errors

• Could not fetch job page: ${e.message}
• [fetch] Playwright error: ${e.message} — falling back to pla
• [fetch] Playwright unavailable — falling back to plain fetch
• refusing to archive restricted destination: ${preGuard.reaso
• refusing to archive restricted destination after redirect: $

AI-assisted analysis of santifer/career-ops@60398d6549 (2026-08-20). Data as JSON: /api/errors/3b1a21e99580afaf. Report an issue: GitHub.