{"record":{"id":"2481658eabe340bc","repo":"apify/crawlee","slug":"selector-selector-not-found-248165","errorCode":null,"errorMessage":"Selector '${selector}' not found.","messagePattern":"Selector '(.+?)' not found\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":619,"sourceCode":"        const remaining = remainingNavigationWindowMillis(crawlingContext, this.#navigationTimeoutMillis);\n        if (remaining <= 0) {\n            throw new TimeoutError(`Navigation timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`);\n        }\n        const parsed = await addTimeoutToPromise(\n            async () => this.parseResponse(crawlingContext.request, crawlingContext.response),\n            remaining,\n            `Navigation timed out after ${this.#navigationTimeoutMillis / 1000} seconds.`,\n        );\n        tryCancel();\n        const response = parsed.response!;\n        const contentType = parsed.contentType!;\n\n        const waitForSelector = async (selector: string, _timeoutMs?: number) => {\n            const cheerio = await import('cheerio');\n            const $ = cheerio.load(parsed.body!.toString());\n\n            if ($(selector).get().length === 0) {\n                throw new Error(`Selector '${selector}' not found.`);\n            }\n        };\n        const parseWithCheerio = async (selector?: string, timeoutMs?: number) => {\n            const cheerio = await import('cheerio');\n            const $ = cheerio.load(parsed.body!.toString());\n\n            if (selector) {\n                await (crawlingContext as InternalHttpCrawlingContext).waitForSelector(selector, timeoutMs);\n            }\n\n            return $;\n        };\n\n        this.throwOnBlockedRequest(response.status);\n\n        if (this.#saveResponseCookies) {\n            try {\n                for (const cookie of getCookiesFromResponse(response)) {","sourceCodeStart":601,"sourceCodeEnd":637,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L601-L637","documentation":"HttpCrawler's `waitForSelector` helper loads the response body into cheerio and throws immediately if no element matches the given CSS selector. Unlike browser crawlers there is no waiting: the check runs once against the already-downloaded static HTML, so a selector that would appear later after JS execution is simply 'not found'. It is a convenience/assertion API, not a real wait primitive.","triggerScenarios":"Calling `waitForSelector(selector)` (directly or via crawling context) when the fetched static HTML contains zero nodes matching the selector. The `_timeoutMs` argument is ignored, so passing a timeout does not change behavior.","commonSituations":"Scraping JavaScript-rendered SPAs where the target element is injected client-side; typos in CSS selectors; selecting elements rendered only after login or pagination; expecting behavior like Puppeteer's waitForSelector on a plain HTTP crawler.","solutions":["Check the raw response body (e.g. curl the URL or log parsed.body) and confirm the element exists in static HTML; if not, switch to a browser crawler (PuppeteerCrawler/PlaywrightCrawler).","Fix the selector: verify it against the actual DOM of the served HTML (typos, wrong casing, dynamic class names).","Remove waitForSelector and instead run cheerio parsing in your requestHandler, throwing your own error only if extraction yields nothing.","If the content is behind authentication or interaction, request the underlying API endpoint that returns the data instead of the rendered page."],"exampleFix":"// before\nawait context.waitForSelector('.product-price');\n\n// after\nconst $ = await context.parseWithCheerio();\nconst price = $('.product-price').first().text();\nif (!price) throw new Error('Product price missing from static HTML');","handlingStrategy":"try-catch","validationCode":"const html = await got(url).text();\nif (!html.includes('data-target-selector-hook')) {\n  console.warn('Static HTML does not contain expected element; needs browser rendering');\n}","typeGuard":"function elementExists($: cheerio.CheerioAPI, selector: string): boolean {\n  return $(selector).length > 0;\n}","tryCatchPattern":"try {\n  await context.waitForSelector('.item');\n} catch (err) {\n  if (err instanceof Error && err.message.startsWith(\"Selector '\")) {\n    log.warning(`Selector missing in static HTML: ${err.message}`);\n    return; // skip or fall back to browser crawler\n  }\n  throw err;\n}","preventionTips":["Remember HttpCrawler does not execute JavaScript — use Puppeteer/PlaywrightCrawler for client-rendered content.","Paste the selector into browser devtools against the raw fetched HTML (view-source, not dev DOM) before using it.","Prefer parseWithCheerio plus an explicit empty-result check over waitForSelector for clearer control flow.","Centralize selectors in constants and test them against fixture HTML."],"tags":["http","cheerio","selector","static-html"],"backgroundTag":"selector-not-found","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}