{"record":{"id":"896e021165e70f2f","repo":"jackwener/OpenCLI","slug":"failed-to-scrape-indeed-job-detail-dom-e-messa","errorCode":null,"errorMessage":"Failed to scrape Indeed job detail DOM: ${e?.message ?? e}","messagePattern":"Failed to scrape Indeed job detail DOM: (.+?)","errorType":"exception","errorClass":"CommandExecutionError","httpStatus":null,"severity":"error","filePath":"clis/indeed/job.js","lineNumber":62,"sourceCode":"                    await new Promise(r => setTimeout(r, 500));\n                    ready = !!document.querySelector('#jobDescriptionText, h1, [data-testid=\"error-page\"]');\n                }\n                const challenge = (document.title || '').includes('Just a moment') || !!document.querySelector('[id^=\"cf-\"]');\n                const notFound = !!document.querySelector('[data-testid=\"error-page\"]') || /Page Not Found|not found/i.test(document.querySelector('h1')?.textContent || '');\n                const title = document.querySelector('h1')?.textContent?.trim() ?? '';\n                const company = document.querySelector('[data-testid=\"inlineHeader-companyName\"] a, [data-testid=\"inlineHeader-companyName\"], [data-company-name=\"true\"]')?.textContent?.trim() ?? '';\n                const location = document.querySelector('[data-testid=\"jobsearch-JobInfoHeader-companyLocation\"] div, [data-testid=\"inlineHeader-companyLocation\"]')?.textContent?.trim() ?? '';\n                const salary = document.querySelector('[id*=\"salaryInfoAndJobType\"] span, [data-testid=\"job-salary\"]')?.textContent?.trim() ?? '';\n                const jobType = Array.from(document.querySelectorAll('[id*=\"salaryInfoAndJobType\"] span, [data-testid=\"job-type\"]'))\n                    .map(s => (s.textContent || '').trim())\n                    .filter(t => t && t !== salary)\n                    .join(', ');\n                const description = document.querySelector('#jobDescriptionText')?.innerText?.trim() ?? '';\n                return { ready, challenge, notFound, title, company, location, salary, jobType, description };\n            })()`);\n        }\n        catch (e) {\n            throw new CommandExecutionError(`Failed to scrape Indeed job detail DOM: ${e?.message ?? e}`, 'The page may not have fully loaded; try again.');\n        }\n\n        if (detail?.challenge) {\n            throw new CommandExecutionError('Indeed served a Cloudflare challenge page', 'Open https://www.indeed.com in the connected browser and clear the challenge, then retry.');\n        }\n        if (!detail?.ready) {\n            throw new CommandExecutionError('Indeed job page did not expose detail or error markers within 15s', 'Indeed may still be loading or the DOM shape may have changed; retry after opening Indeed in the connected browser.');\n        }\n        if (detail?.notFound || (!detail?.title && !detail?.description)) {\n            throw new EmptyResultError('indeed job', `No Indeed job posting found for jk \"${jk}\"`);\n        }\n\n        return [{\n            id: jk,\n            title: detail.title.replace(/\\s+/g, ' ').trim(),\n            company: detail.company.replace(/\\s+/g, ' ').trim(),\n            location: detail.location.replace(/\\s+/g, ' ').trim(),\n            salary: detail.salary.replace(/\\s+/g, ' ').trim(),","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/indeed/job.js#L44-L80","documentation":"Wrapper error thrown when the in-page DOM scraping script for an Indeed job detail page throws (e.g. page.evaluate rejects). It wraps the underlying exception message and indicates the page had not fully loaded when the scraper ran.","triggerScenarios":"The page.evaluate(...) call that reads #jobDescriptionText, title, company, salary etc. throws while scraping the Indeed job detail page; e?.message is interpolated into the error.","commonSituations":"Slow network so the DOM never became ready before evaluation; navigation interrupted by redirect; browser tab closed or crashed mid-scrape; the connected browser blocked script execution or an extension interfered; Indeed served an unexpected interstitial that broke selectors.","solutions":["Retry the command after a moment to let the page fully load","Open the job URL in the connected browser, confirm it renders completely, then retry","Disable interfering extensions (ad blockers, privacy tools) in the connected browser","Improve network conditions or increase the wait/poll timeout before scraping","If persistent, verify Indeed's selectors (#jobDescriptionText etc.) still exist and update the scraper"],"exampleFix":"// before\nconst detail = await page.evaluate(main); // throws on slow load\n// after\nawait page.waitForSelector('#jobDescriptionText', { timeout: 20000 }).catch(() => {});\nconst detail = await page.evaluate(main);","handlingStrategy":"try-catch","validationCode":"await page.waitForSelector('#jobDescriptionText', { timeout: 20000 }).catch(() => {});","typeGuard":null,"tryCatchPattern":"try {\n  const job = await indeed.job(jk);\n} catch (e) {\n  if (/Failed to scrape Indeed job detail DOM/.test(e.message)) {\n    // back off, then retry once with a fresh page load\n  }\n  throw e;\n}","preventionTips":["Ensure the page fully loads before scraping (waitForSelector/networkidle)","Disable ad-blocker/privacy extensions in the connected browser","Retry transient failures with exponential backoff","Keep browser and CLI versions current"],"tags":["scraping","dom","page-load","indeed"],"backgroundTag":"scrape-dom-evaluation-failed","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}