{"record":{"id":"5931f41d424a7cf2","repo":"jackwener/OpenCLI","slug":"zhihu-label-pagination-exceeded-its-fetch-budge","errorCode":null,"errorMessage":"Zhihu ${label} pagination exceeded its fetch budget","messagePattern":"Zhihu (.+?) pagination exceeded its fetch budget","errorType":"exception","errorClass":"CommandExecutionError","httpStatus":null,"severity":"error","filePath":"clis/zhihu/answer-comments-helpers.js","lineNumber":140,"sourceCode":"            throw new CommandExecutionError(`Zhihu answer comments returned conflicting data for comment ${descriptor.id}`);\n        }\n        const oldCount = previous.comment.child_comment_count;\n        const newCount = comment.child_comment_count;\n        if (Number.isInteger(newCount) && (!Number.isInteger(oldCount) || newCount > oldCount)) {\n            previous.comment = { ...previous.comment, child_comment_count: newCount };\n        }\n    }\n}\nasync function fetchPages(page, options) {\n    const { firstUrl, limit, label, role, expectedRootId = '', normalizeNext, notFoundDetail = '' } = options;\n    const byId = new Map();\n    const visited = new Set();\n    const maxPages = Math.ceil(limit / PAGE_SIZE) + PAGE_OVERLAP_ALLOWANCE;\n    let pageCount = 0;\n    let url = firstUrl;\n    while (byId.size < limit) {\n        if (visited.has(url)) throw new CommandExecutionError(`Zhihu ${label} pagination returned a repeated next URL`);\n        if (pageCount >= maxPages) throw new CommandExecutionError(`Zhihu ${label} pagination exceeded its fetch budget`);\n        visited.add(url);\n        pageCount += 1;\n        const payload = await fetchCommentPage(page, url, label, notFoundDetail);\n        addPageComments(byId, payload.data, role, expectedRootId);\n        if (payload.paging.is_end || byId.size >= limit) break;\n        url = normalizeNext(payload.paging.next);\n        if (!url) throw new CommandExecutionError(`Zhihu ${label} pagination returned a malformed next URL`);\n    }\n    return [...byId.values()].slice(0, limit).map(({ comment }) => comment);\n}\nexport function fetchRootComments(page, answerId, order, limit) {\n    const apiOrder = order === 'latest' ? 'ts' : 'score';\n    const path = `/api/v4/comment_v5/answers/${answerId}/root_comment`;\n    return fetchPages(page, {\n        firstUrl: `https://www.zhihu.com${path}?order_by=${apiOrder}&limit=${PAGE_SIZE}&offset=`,\n        limit,\n        label: 'answer root comments',\n        role: 'root',","sourceCodeStart":122,"sourceCodeEnd":158,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/zhihu/answer-comments-helpers.js#L122-L158","documentation":"fetchPages caps total requests at ceil(limit / PAGE_SIZE) + PAGE_OVERLAP_ALLOWANCE. Exceeding that budget means the API is returning pages that make less forward progress than expected (duplicated rows, shrinking pages) and the fetch would run unbounded, so it aborts.","triggerScenarios":"byId never reaches limit within maxPages requests — e.g. pages full of duplicates (each page adds few/no new ids) or paging.next keeps yielding valid but non-advancing pages without repeating a URL.","commonSituations":"Requesting a limit far larger than the number of existing comments while the API keeps returning overlapping pages; sort instability causing rows to shuffle between pages; or a very low limit with PAGE_OVERLAP_ALLOWANCE too small for a chatty endpoint.","solutions":["Lower the limit to something close to the answer's real comment count.","Retry with the other sort order to avoid unstable pagination windows on actively-commented answers.","Fetch during a quieter window or snapshot the answer before comments change rapidly.","If duplicates are the cause, increase PAGE_OVERLAP_ALLOWANCE in the library (or via config) to tolerate overlap."],"exampleFix":"// before\nconst comments = await fetchRootComments(page, answerId, 'latest', 10000);\n// after: size the limit to the actual count\nconst limit = Math.min(answer.comment_count, 500);\nconst comments = await fetchRootComments(page, answerId, 'latest', limit);","handlingStrategy":"validation","validationCode":"// request only as many comments as actually exist\nconst safeLimit = Math.min(answer.comment_count ?? 0, 500);\nconst comments = safeLimit > 0\n  ? await fetchRootComments(page, answerId, 'latest', safeLimit)\n  : [];","typeGuard":"function hasUsableCount(answer) {\n  return Number.isInteger(answer?.comment_count) && answer.comment_count >= 0;\n}","tryCatchPattern":"try {\n  const comments = await fetchRootComments(page, answerId, order, limit);\n} catch (err) {\n  if (String(err.message).includes('exceeded its fetch budget')) {\n    console.warn(`Reducing limit and retrying: ${limit}`);\n    return fetchRootComments(page, answerId, order, Math.ceil(limit / 2));\n  }\n  throw err;\n}","preventionTips":["Size limit from the answer's comment_count instead of a large fixed value.","Avoid fetching during rapid comment churn (sort instability causes overlap).","Raise PAGE_OVERLAP_ALLOWANCE only if you know pages legitimately overlap."],"tags":["pagination","api","limits","zhihu"],"backgroundTag":"pagination-budget-exceeded","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}