jackwener/OpenCLI · error · CommandExecutionError

Failed to scrape Indeed job detail DOM: ${e?.message ?? e}

Error message

Failed to scrape Indeed job detail DOM: ${e?.message ?? e}

What it means

Wrapper error thrown when the in-page DOM scraping script for an Indeed job detail page throws (e.g. page.evaluate rejects). It wraps the underlying exception message and indicates the page had not fully loaded when the scraper ran.

Source

Thrown at clis/indeed/job.js:62

                    await new Promise(r => setTimeout(r, 500));
                    ready = !!document.querySelector('#jobDescriptionText, h1, [data-testid="error-page"]');
                }
                const challenge = (document.title || '').includes('Just a moment') || !!document.querySelector('[id^="cf-"]');
                const notFound = !!document.querySelector('[data-testid="error-page"]') || /Page Not Found|not found/i.test(document.querySelector('h1')?.textContent || '');
                const title = document.querySelector('h1')?.textContent?.trim() ?? '';
                const company = document.querySelector('[data-testid="inlineHeader-companyName"] a, [data-testid="inlineHeader-companyName"], [data-company-name="true"]')?.textContent?.trim() ?? '';
                const location = document.querySelector('[data-testid="jobsearch-JobInfoHeader-companyLocation"] div, [data-testid="inlineHeader-companyLocation"]')?.textContent?.trim() ?? '';
                const salary = document.querySelector('[id*="salaryInfoAndJobType"] span, [data-testid="job-salary"]')?.textContent?.trim() ?? '';
                const jobType = Array.from(document.querySelectorAll('[id*="salaryInfoAndJobType"] span, [data-testid="job-type"]'))
                    .map(s => (s.textContent || '').trim())
                    .filter(t => t && t !== salary)
                    .join(', ');
                const description = document.querySelector('#jobDescriptionText')?.innerText?.trim() ?? '';
                return { ready, challenge, notFound, title, company, location, salary, jobType, description };
            })()`);
        }
        catch (e) {
            throw new CommandExecutionError(`Failed to scrape Indeed job detail DOM: ${e?.message ?? e}`, 'The page may not have fully loaded; try again.');
        }

        if (detail?.challenge) {
            throw new CommandExecutionError('Indeed served a Cloudflare challenge page', 'Open https://www.indeed.com in the connected browser and clear the challenge, then retry.');
        }
        if (!detail?.ready) {
            throw new CommandExecutionError('Indeed job page did not expose detail or error markers within 15s', 'Indeed may still be loading or the DOM shape may have changed; retry after opening Indeed in the connected browser.');
        }
        if (detail?.notFound || (!detail?.title && !detail?.description)) {
            throw new EmptyResultError('indeed job', `No Indeed job posting found for jk "${jk}"`);
        }

        return [{
            id: jk,
            title: detail.title.replace(/\s+/g, ' ').trim(),
            company: detail.company.replace(/\s+/g, ' ').trim(),
            location: detail.location.replace(/\s+/g, ' ').trim(),
            salary: detail.salary.replace(/\s+/g, ' ').trim(),

View on GitHub (pinned to 49907e53dc)

Solutions

1. Retry the command after a moment to let the page fully load
2. Open the job URL in the connected browser, confirm it renders completely, then retry
3. Disable interfering extensions (ad blockers, privacy tools) in the connected browser
4. Improve network conditions or increase the wait/poll timeout before scraping
5. If persistent, verify Indeed's selectors (#jobDescriptionText etc.) still exist and update the scraper

Example fix

// before
const detail = await page.evaluate(main); // throws on slow load
// after
await page.waitForSelector('#jobDescriptionText', { timeout: 20000 }).catch(() => {});
const detail = await page.evaluate(main);

await page.waitForSelector('#jobDescriptionText', { timeout: 20000 }).catch(() => {});

try {
  const job = await indeed.job(jk);
} catch (e) {
  if (/Failed to scrape Indeed job detail DOM/.test(e.message)) {
    // back off, then retry once with a fresh page load
  }
  throw e;
}

When it happens

Trigger: The page.evaluate(...) call that reads #jobDescriptionText, title, company, salary etc. throws while scraping the Indeed job detail page; e?.message is interpolated into the error.

Common situations: Slow network so the DOM never became ready before evaluation; navigation interrupted by redirect; browser tab closed or crashed mid-scrape; the connected browser blocked script execution or an extension interfered; Indeed served an unexpected interstitial that broke selectors.

Related errors

• Failed to scrape Indeed search DOM: ${e?.message ?? e}
• Indeed job page did not expose detail or error markers withi
• Indeed search page did not expose result or empty-state mark
• Failed to extract Booking.com cards: ${err?.message || err}
• No visible Claude messages were found for conversation ${id}

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/896e021165e70f2f. Report an issue: GitHub.