{"record":{"id":"c03ec54b560f5922","repo":"jackwener/OpenCLI","slug":"weread-search-page-request-failed-http-resp-sta","errorCode":null,"errorMessage":"WeRead search page request failed: HTTP ${resp.status}","messagePattern":"WeRead search page request failed: HTTP (.+?)","errorType":"exception","errorClass":"CommandExecutionError","httpStatus":null,"severity":"error","filePath":"clis/weread/search.js","lineNumber":113,"sourceCode":"/**\n * Extract rendered search result reader URLs from the server-rendered search page.\n * The public JSON API still returns bookId, but the current web app links results\n * through /web/reader/<opaque-id> rather than /web/bookDetail/<bookId>.\n */\nasync function loadSearchHtmlEntries(query) {\n    const url = new URL('/web/search/books', WEREAD_WEB_ORIGIN);\n    url.searchParams.set('keyword', query);\n    let resp;\n    try {\n        resp = await fetch(url.toString(), {\n            headers: { 'User-Agent': WEREAD_UA },\n        });\n    }\n    catch (error) {\n        throw new CommandExecutionError(`Failed to fetch WeRead search page: ${error instanceof Error ? error.message : String(error)}`);\n    }\n    if (!resp.ok) {\n        throw new CommandExecutionError(`WeRead search page request failed: HTTP ${resp.status}`);\n    }\n    const html = await resp.text();\n    const items = Array.from(html.matchAll(/<li[^>]*class=\"wr_bookList_item\"[^>]*>([\\s\\S]*?)<\\/li>/g));\n    return items.map((match) => {\n        const chunk = match[1];\n        const hrefMatch = chunk.match(/<a[^>]*href=\"([^\"]+)\"[^>]*class=\"wr_bookList_item_link\"[^>]*>|<a[^>]*class=\"wr_bookList_item_link\"[^>]*href=\"([^\"]+)\"[^>]*>/);\n        const titleMatch = chunk.match(/<p[^>]*class=\"wr_bookList_item_title\"[^>]*>([\\s\\S]*?)<\\/p>/);\n        const authorMatch = chunk.match(/<p[^>]*class=\"wr_bookList_item_author\"[^>]*>([\\s\\S]*?)<\\/p>/);\n        const href = hrefMatch?.[1] || hrefMatch?.[2] || '';\n        const title = decodeHtmlText(titleMatch?.[1] || '');\n        const author = decodeHtmlText(authorMatch?.[1] || '');\n        return {\n            author,\n            url: href ? new URL(href, WEREAD_WEB_ORIGIN).toString() : '',\n            title,\n        };\n    }).filter((item) => item.url && item.title);\n}","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/weread/search.js#L95-L131","documentation":"Thrown by loadSearchHtmlEntries when the WeRead search page request completes but responds with a non-2xx status (resp.ok is false). The library surfaces the HTTP status code in a CommandExecutionError because the HTML result list cannot be scraped from an error page (403 anti-bot, 404 route change, 5xx outage).","triggerScenarios":"GET https://weread.qq.com/web/search/books?keyword=... returns 403 (bot/rate-limit protection), 429 (rate limited), 404 (route removed after a web app redesign), or 5xx (server outage). Any command invocation of `weread search` can hit this.","commonSituations":"WeRead WAF rejecting the spoofed Chrome User-Agent due to missing browser cookies/headers; hammering the search endpoint in a loop; WeRead renaming or removing the /web/search/books route in a frontend deploy; regional outages of weread.qq.com.","solutions":["Retry after a delay with fewer requests — 403/429 usually indicate rate limiting or anti-bot throttling.","Verify the route still exists: open https://weread.qq.com/web/search/books?keyword=test in a browser; a 404 after a redesign means the scraper regex/URL must be updated.","Capture the response body on failure (resp.text() before throwing) to see whether WeRead returns a captcha/challenge page.","Add realistic headers (Accept, Accept-Language) alongside WEREAD_UA to reduce WAF rejections.","Check status specifics: 5xx → wait for WeRead to recover; 403/429 → slow down or use cookies from a real logged-in browser session."],"exampleFix":"null","handlingStrategy":"retry","validationCode":"// Probe the scrape target and treat non-2xx as a stop signal\nconst r = await fetch('https://weread.qq.com/web/search/books?keyword=test', { headers: { 'User-Agent': WEREAD_UA } });\nif (!r.ok) console.warn(`search page unhealthy (HTTP ${r.status}); expect CommandExecutionError`);","typeGuard":null,"tryCatchPattern":"try {\n  const books = await runWereadSearch(query);\n} catch (e) {\n  const m = /HTTP (\\d{3})/.exec(String(e.message));\n  if (m && ['403', '429'].includes(m[1])) {\n    await sleep(10000); // back off WAF/rate limit\n    return retryWithFewerRequests();\n  }\n  if (m && m[1].startsWith('5')) return fallbackToApiOnly();\n  throw e;\n}","preventionTips":["Throttle search request rate in loops to avoid WAF 403/429","Send complete browser-like headers (Accept, Accept-Language) with WEREAD_UA","Periodically verify the /web/search/books route still exists after WeRead frontend deploys","Log response bodies on failure to detect captcha/challenge pages early"],"tags":["http","weread","http-status","scraping","rate-limit"],"backgroundTag":"http-non-2xx-response","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}