{"record":{"id":"f7e1e7cb4bef59c6","repo":"jackwener/OpenCLI","slug":"selector-drift","errorCode":"selector_drift","errorMessage":"[taxonomy=selector_drift] site=${site} command=detail detail page blocked by verification challenge: ${targetUrl}","messagePattern":"\\[taxonomy=selector_drift\\] site=(.+?) command=detail detail page blocked by verification challenge: (.+?)","errorType":"error_code","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"clis/jianyu/shared/procurement-detail.js","lineNumber":73,"sourceCode":"    }\n    let lastError = null;\n    for (let attempt = 1; attempt <= DETAIL_MAX_ATTEMPTS; attempt += 1) {\n        try {\n            const payload = await extractDetailPayload(page, targetUrl);\n            if (!payload || typeof payload !== 'object') {\n                throw taxonomyError('extraction_drift', {\n                    site,\n                    command: 'detail',\n                    detail: `detail extraction returned invalid payload: ${targetUrl}`,\n                });\n            }\n            const row = payload;\n            const title = cleanText(row.title);\n            const detailText = cleanText(row.detailText);\n            const publishTime = cleanText(row.publishTime);\n            const authGateText = cleanText(`${title} ${detailText}`);\n            if (DETAIL_AUTH_CHALLENGE_PATTERNS.some((pattern) => pattern.test(authGateText))) {\n                throw taxonomyError('selector_drift', {\n                    site,\n                    command: 'detail',\n                    detail: `detail page blocked by verification challenge: ${targetUrl}`,\n                });\n            }\n            if (!title && !detailText) {\n                throw taxonomyError('empty_result', {\n                    site,\n                    command: 'detail',\n                    detail: `detail page has no readable content: ${targetUrl}`,\n                });\n            }\n            return [\n                toProcurementDetailRecord({\n                    title: title || targetUrl,\n                    url: targetUrl,\n                    contextText: detailText,\n                    publishTime,","sourceCodeStart":55,"sourceCodeEnd":91,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/jianyu/shared/procurement-detail.js#L55-L91","documentation":"After extracting a detail payload, runProcurementDetail concatenates title and detailText and tests them against DETAIL_AUTH_CHALLENGE_PATTERNS (e.g. verification/captcha wording). If matched, it throws a taxonomy error with code 'selector_drift' stating the detail page was blocked by a verification challenge — the site served an anti-bot wall instead of content.","triggerScenarios":"The detail page's extracted title/detailText contains challenge markers (captcha, verify-you-are-human text), causing the pattern test to hit and throw.","commonSituations":"Scraping too fast from one IP triggering rate-limit captchas; datacenter/proxy IPs flagged by the site; missing browser fingerprint (headless detection); session cookies expired so the site forces verification.","solutions":["Slow down request rate and add randomized delays between detail fetches","Route traffic through cleaner/residential proxies or rotate IPs","Use a persistent logged-in browser profile with valid cookies","Solve the challenge interactively once, then reuse the session; or flag the URL for manual retry"],"exampleFix":"// before\nfor (const url of urls) {\n  rows.push(await runProcurementDetail(page, { site, url }));\n}\n// after\nfor (const url of urls) {\n  try {\n    rows.push(await runProcurementDetail(page, { site, url }));\n  } catch (e) {\n    if (/selector_drift/.test(String(e.message))) {\n      await solveChallenge(page); // human-in-the-loop verification\n      rows.push(await runProcurementDetail(page, { site, url }));\n    } else throw e;\n  }\n  await sleep(2000 + Math.random() * 3000);\n}","handlingStrategy":"retry","validationCode":"// pre-check: skip URLs known to hit challenges; throttle request rate\nawait throttle(site, { minDelayMs: 3000, jitter: true });","typeGuard":"function isChallengeBlock(e) {\n  return e instanceof Error && /verification challenge/.test(e.message);\n}","tryCatchPattern":"try {\n  return await runProcurementDetail(page, { site, url });\n} catch (e) {\n  if (isChallengeBlock(e)) {\n    await backoffAndRotateProxy();\n    return runProcurementDetail(page, { site, url });\n  }\n  throw e;\n}","preventionTips":["Rate-limit detail fetches with randomized delays","Use residential proxies or rotate IPs for high-volume scraping","Maintain a logged-in, persistent browser session","Stop and solve challenges interactively instead of hammering retries"],"tags":["anti-bot","captcha","scraping","blocked"],"backgroundTag":"anti-bot-verification-challenge","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}