jackwener/OpenCLI · error · Error

selector_drift

selector_drift

Error message

[taxonomy=selector_drift] site=${site} command=detail detail page blocked by verification challenge: ${targetUrl}

What it means

After extracting a detail payload, runProcurementDetail concatenates title and detailText and tests them against DETAIL_AUTH_CHALLENGE_PATTERNS (e.g. verification/captcha wording). If matched, it throws a taxonomy error with code 'selector_drift' stating the detail page was blocked by a verification challenge — the site served an anti-bot wall instead of content.

Source

Thrown at clis/jianyu/shared/procurement-detail.js:73

    }
    let lastError = null;
    for (let attempt = 1; attempt <= DETAIL_MAX_ATTEMPTS; attempt += 1) {
        try {
            const payload = await extractDetailPayload(page, targetUrl);
            if (!payload || typeof payload !== 'object') {
                throw taxonomyError('extraction_drift', {
                    site,
                    command: 'detail',
                    detail: `detail extraction returned invalid payload: ${targetUrl}`,
                });
            }
            const row = payload;
            const title = cleanText(row.title);
            const detailText = cleanText(row.detailText);
            const publishTime = cleanText(row.publishTime);
            const authGateText = cleanText(`${title} ${detailText}`);
            if (DETAIL_AUTH_CHALLENGE_PATTERNS.some((pattern) => pattern.test(authGateText))) {
                throw taxonomyError('selector_drift', {
                    site,
                    command: 'detail',
                    detail: `detail page blocked by verification challenge: ${targetUrl}`,
                });
            }
            if (!title && !detailText) {
                throw taxonomyError('empty_result', {
                    site,
                    command: 'detail',
                    detail: `detail page has no readable content: ${targetUrl}`,
                });
            }
            return [
                toProcurementDetailRecord({
                    title: title || targetUrl,
                    url: targetUrl,
                    contextText: detailText,
                    publishTime,

View on GitHub (pinned to 49907e53dc)

Solutions

Slow down request rate and add randomized delays between detail fetches
Route traffic through cleaner/residential proxies or rotate IPs
Use a persistent logged-in browser profile with valid cookies
Solve the challenge interactively once, then reuse the session; or flag the URL for manual retry

Example fix

// before
for (const url of urls) {
  rows.push(await runProcurementDetail(page, { site, url }));
}
// after
for (const url of urls) {
  try {
    rows.push(await runProcurementDetail(page, { site, url }));
  } catch (e) {
    if (/selector_drift/.test(String(e.message))) {
      await solveChallenge(page); // human-in-the-loop verification
      rows.push(await runProcurementDetail(page, { site, url }));
    } else throw e;
  }
  await sleep(2000 + Math.random() * 3000);
}

Defensive patterns

Strategy: retry

Validate before calling

// pre-check: skip URLs known to hit challenges; throttle request rate
await throttle(site, { minDelayMs: 3000, jitter: true });

Type guard

function isChallengeBlock(e) {
  return e instanceof Error && /verification challenge/.test(e.message);
}

Try / catch

try {
  return await runProcurementDetail(page, { site, url });
} catch (e) {
  if (isChallengeBlock(e)) {
    await backoffAndRotateProxy();
    return runProcurementDetail(page, { site, url });
  }
  throw e;
}

Prevention

Rate-limit detail fetches with randomized delays
Use residential proxies or rotate IPs for high-volume scraping
Maintain a logged-in, persistent browser session
Stop and solve challenges interactively instead of hammering retries

When it happens

Trigger: The detail page's extracted title/detailText contains challenge markers (captcha, verify-you-are-human text), causing the pattern test to hit and throw.

Common situations: Scraping too fast from one IP triggering rate-limit captchas; datacenter/proxy IPs flagged by the site; missing browser fingerprint (headless detection); session cookies expired so the site forces verification.

Related errors

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/f7e1e7cb4bef59c6. Report an issue: GitHub.