apify/crawlee · error · Error

Selector '${selector}' not found.

Error message

Selector '${selector}' not found.

What it means

HttpCrawler's `waitForSelector` helper loads the response body into cheerio and throws immediately if no element matches the given CSS selector. Unlike browser crawlers there is no waiting: the check runs once against the already-downloaded static HTML, so a selector that would appear later after JS execution is simply 'not found'. It is a convenience/assertion API, not a real wait primitive.

Source

Thrown at packages/http-crawler/src/internals/http-crawler.ts:619

        const remaining = remainingNavigationWindowMillis(crawlingContext, this.#navigationTimeoutMillis);
        if (remaining <= 0) {
            throw new TimeoutError(`Navigation timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`);
        }
        const parsed = await addTimeoutToPromise(
            async () => this.parseResponse(crawlingContext.request, crawlingContext.response),
            remaining,
            `Navigation timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`,
        );
        tryCancel();
        const response = parsed.response!;
        const contentType = parsed.contentType!;

        const waitForSelector = async (selector: string, _timeoutMs?: number) => {
            const cheerio = await import('cheerio');
            const $ = cheerio.load(parsed.body!.toString());

            if ($(selector).get().length === 0) {
                throw new Error(`Selector '${selector}' not found.`);
            }
        };
        const parseWithCheerio = async (selector?: string, timeoutMs?: number) => {
            const cheerio = await import('cheerio');
            const $ = cheerio.load(parsed.body!.toString());

            if (selector) {
                await (crawlingContext as InternalHttpCrawlingContext).waitForSelector(selector, timeoutMs);
            }

            return $;
        };

        this.throwOnBlockedRequest(response.status);

        if (this.#saveResponseCookies) {
            try {
                for (const cookie of getCookiesFromResponse(response)) {

View on GitHub (pinned to dbe57fb09c)

Solutions

1. Check the raw response body (e.g. curl the URL or log parsed.body) and confirm the element exists in static HTML; if not, switch to a browser crawler (PuppeteerCrawler/PlaywrightCrawler).
2. Fix the selector: verify it against the actual DOM of the served HTML (typos, wrong casing, dynamic class names).
3. Remove waitForSelector and instead run cheerio parsing in your requestHandler, throwing your own error only if extraction yields nothing.
4. If the content is behind authentication or interaction, request the underlying API endpoint that returns the data instead of the rendered page.

Example fix

// before
await context.waitForSelector('.product-price');

// after
const $ = await context.parseWithCheerio();
const price = $('.product-price').first().text();
if (!price) throw new Error('Product price missing from static HTML');

const html = await got(url).text();
if (!html.includes('data-target-selector-hook')) {
  console.warn('Static HTML does not contain expected element; needs browser rendering');
}

function elementExists($: cheerio.CheerioAPI, selector: string): boolean {
  return $(selector).length > 0;
}

try {
  await context.waitForSelector('.item');
} catch (err) {
  if (err instanceof Error && err.message.startsWith("Selector '")) {
    log.warning(`Selector missing in static HTML: ${err.message}`);
    return; // skip or fall back to browser crawler
  }
  throw err;
}

When it happens

Trigger: Calling `waitForSelector(selector)` (directly or via crawling context) when the fetched static HTML contains zero nodes matching the selector. The `_timeoutMs` argument is ignored, so passing a timeout does not change behavior.

Common situations: Scraping JavaScript-rendered SPAs where the target element is injected client-side; typos in CSS selectors; selecting elements rendered only after login or pagination; expecting behavior like Puppeteer's waitForSelector on a plain HTTP crawler.

Related errors

• Selector '${selector}' not found.
• Selector '${selector}' not found.
• Could not parse cookie header string: ${cookieHeaderString}
• Request with GET method cannot have a payload.
• Cannot fetch a request list from ${requestsFromUrl}: ${err}

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/2481658eabe340bc. Report an issue: GitHub.