jackwener/OpenCLI · error · CommandExecutionError

Failed to scrape Indeed search DOM: ${e?.message ?? e}

Error message

Failed to scrape Indeed search DOM: ${e?.message ?? e}

What it means

Wrapper error thrown when the Indeed search-page DOM scraping script throws inside page.evaluate. As with the job detail variant, it wraps the underlying message and usually means the page had not fully loaded when selectors were evaluated.

Source

Thrown at clis/indeed/search.js:94

                    const tags = Array.from(b.querySelectorAll('.metadataContainer li span'))
                        .map(s => (s.textContent || '').trim())
                        .filter(Boolean);
                    out.push({
                        jk,
                        title: b.querySelector('h2.jobTitle span')?.textContent?.trim() ?? '',
                        company: b.querySelector('[data-testid="company-name"]')?.textContent?.trim() ?? '',
                        location: b.querySelector('[data-testid="text-location"]')?.textContent?.trim() ?? '',
                        salary: b.querySelector('.salary-snippet-container span')?.textContent?.trim() ?? '',
                        tags,
                    });
                }
                const blockedHeadline = document.title || '';
                const challenge = blockedHeadline.includes('Just a moment') || !!document.querySelector('[id^="cf-"]');
                return { cards: out, challenge, ready };
            })()`);
        }
        catch (e) {
            throw new CommandExecutionError(`Failed to scrape Indeed search DOM: ${e?.message ?? e}`, 'The page may not have fully loaded; try again.');
        }

        if (cards?.challenge) {
            throw new CommandExecutionError('Indeed served a Cloudflare challenge page', 'Open https://www.indeed.com in the connected browser and clear the challenge, then retry.');
        }
        if (!cards?.ready) {
            throw new CommandExecutionError('Indeed search page did not expose result or empty-state markers within 15s', 'Indeed may still be loading or the DOM shape may have changed; retry after opening Indeed in the connected browser.');
        }

        const list = Array.isArray(cards?.cards) ? cards.cards : [];
        if (list.length === 0) {
            throw new EmptyResultError('indeed search', `No Indeed jobs matched "${query}"${location ? ` in ${location}` : ''}`);
        }
        return list.slice(0, limit).map((c, i) => searchCardToRow(c, start + i + 1));
    },
});

View on GitHub (pinned to 49907e53dc)

Solutions

1. Retry the search to let the page fully load
2. Open the Indeed search URL in the connected browser, confirm rendering, then retry
3. Disable interfering browser extensions and retry
4. Increase wait time before evaluation (waitForSelector on result containers)
5. If persistent, check whether Indeed changed search DOM structure and update the scraper

Example fix

// before
const cards = await page.evaluate(main);
// after
await page.waitForSelector('[data-testid="job-card"] , .job_seen_beacon', { timeout: 20000 }).catch(() => {});
const cards = await page.evaluate(main);

await page.waitForSelector('.job_seen_beacon, [data-testid="job-card"], .empty_serp', { timeout: 20000 }).catch(() => {});

try {
  const rows = await indeed.search({ query, location });
} catch (e) {
  if (/Failed to scrape Indeed search DOM/.test(e.message)) {
    // back off and retry once with a fresh page
  }
  throw e;
}

When it happens

Trigger: The evaluate() call that collects search result cards and challenge flags throws during the indeed search command; e?.message is interpolated.

Common situations: Slow/partial page load causing selector access errors; navigation interrupted mid-evaluation; browser tab crashed; extension or CSP interference; Indeed served an unexpected page variant that broke the script.

Related errors

• Failed to scrape Indeed job detail DOM: ${e?.message ?? e}
• Indeed job page did not expose detail or error markers withi
• Indeed search page did not expose result or empty-state mark
• Failed to extract Booking.com cards: ${err?.message || err}
• No visible Claude messages were found for conversation ${id}

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/dbf0f067f646773c. Report an issue: GitHub.