{"record":{"id":"3b1a21e99580afaf","repo":"santifer/career-ops","slug":"extracted-text-too-short-likely-blocked-or-empty","errorCode":null,"errorMessage":"Extracted text too short (likely blocked or empty)","messagePattern":"Extracted text too short \\(likely blocked or empty\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"batch-evaluate-gemini.mjs","lineNumber":243,"sourceCode":"  }\n}\n\nexport async function processOffer(browser, line, idx, _evaluate = evaluateWithRetry) {\n  const match = line.match(/- \\[\\s*\\]\\s+(https?:\\/\\/\\S+)(?:\\s*\\|\\s*([^|]+)\\s*\\|\\s*(.+))?/);\n  if (!match) return { line, processed: false };\n\n  const url = match[1];\n  let companyHint = match[2] ? match[2].trim() : 'Unknown';\n  let titleHint = match[3] ? match[3].trim() : 'Unknown';\n\n  console.log(`\\n========================================`);\n  console.log(`🔄 Processing [${idx}]: ${companyHint} - ${titleHint}`);\n  console.log(`🔗 URL: ${url}`);\n\n  try {\n    const jdText = await scrapeUrl(browser, url);\n    if (!jdText || jdText.length < 100) {\n      throw new Error('Extracted text too short (likely blocked or empty)');\n    }\n\n    console.log(`🧠 Calling Gemini (${modelName})...`);\n    const evaluationText = await _evaluate(`URL: ${url}\\n\\n${jdText}`);\n\n    // Parse output\n    const summaryMatch = evaluationText.match(/---SCORE_SUMMARY---\\s*([\\s\\S]*?)---END_SUMMARY---/);\n    if (!summaryMatch) {\n      console.error('Missing SCORE_SUMMARY block from model output:\\n' + evaluationText.slice(0, 500));\n      throw new Error('Missing SCORE_SUMMARY block from model output');\n    }\n    \n    const block = summaryMatch[1];\n    const extract = (key) => {\n      const m = block.match(new RegExp(`^\\\\s*${key}:\\\\s*(.+)$`, 'mi'));\n      return m ? m[1].trim() : 'unknown';\n    };\n","sourceCodeStart":225,"sourceCodeEnd":261,"githubUrl":"https://github.com/santifer/career-ops/blob/60398d6549a46f5266929538af21cfab94badc75/batch-evaluate-gemini.mjs#L225-L261","documentation":"scrapeUrl() navigated successfully and passed the egress guard, but after removing script/style/noscript/iframe/svg/img nodes the remaining document.body.innerText was falsy or under 100 characters. The page rendered essentially nothing readable. Typical causes are bot walls, cookie-consent interstitials, JS-rendered apps that had not mounted within the fixed 2s waitForTimeout, or a posting already taken down.","triggerScenarios":"Cloudflare/Akamai 'verify you are human' challenge served instead of the JD; a cookie banner that replaces body content; an SPA careers site whose content mounts after the hard-coded 2s wait; a removed posting returning a near-empty 404 shell; page content living entirely inside elements the cleanup step removes.","commonSituations":"Batch runs across many portals where one or two ATS hosts bot-block headless Chromium; running on slow/CI machines where client-side rendering exceeds 2s; sites that render only after interaction.","solutions":["Open the same URL in a real browser (or curl with a desktop UA) to tell a bot wall apart from a genuinely empty/gone page.","Re-run just that entry: some challenges pass on a second, warmer attempt.","If it is a slow SPA, replace the fixed page.waitForTimeout(2000) with waitForSelector on a JD-specific element and a longer timeout.","If the host hard-blocks headless browsers, fetch the JD manually, save it under jds/, and evaluate the text directly instead of the URL.","If the posting is gone, drop the URL from the pipeline."],"exampleFix":"// before\nawait page.waitForTimeout(2000); // wait for dynamic content\n\n// after\nawait page\n  .waitForSelector('.job-description, [data-job-description]', { timeout: 15000 })\n  .catch(() => page.waitForTimeout(5000));","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  const jdText = await scrapeUrl(browser, url);\n} catch (err) {\n  if (/too short/.test(err.message)) {\n    await new Promise(r => setTimeout(r, 5000));\n    const retry = await scrapeUrl(browser, url); // one warm retry\n    if (!retry || retry.length < 100) throw err;\n  } else throw err;\n}","preventionTips":["Prefer ATS JSON endpoints (Greenhouse/Ashby/Lever via scan.mjs) over HTML scraping where available.","For known bot-walled hosts, save the JD manually under jds/ and evaluate from the text, skipping the scraper.","Keep the 100-char floor in mind: pages that only render after interaction will always need the manual path."],"tags":["scraping","anti-bot","playwright","empty-response"],"backgroundTag":"scraping-blocked-empty-content","analyzedSha":"60398d6549a46f5266929538af21cfab94badc75","analyzedAt":"2026-08-20T23:00:06.764Z","schemaVersion":2},"datasetVersion":"2026-08-21T18:17:14.833Z"}