What causes "Zhihu ${label} pagination exceeded its fetch budget" in jackwener/OpenCLI?

Lower the limit to something close to the answer's real comment count.

jackwener/OpenCLI · error · CommandExecutionError

Zhihu ${label} pagination exceeded its fetch budget

Error message

Zhihu ${label} pagination exceeded its fetch budget

What it means

fetchPages caps total requests at ceil(limit / PAGE_SIZE) + PAGE_OVERLAP_ALLOWANCE. Exceeding that budget means the API is returning pages that make less forward progress than expected (duplicated rows, shrinking pages) and the fetch would run unbounded, so it aborts.

Source

Thrown at clis/zhihu/answer-comments-helpers.js:140

            throw new CommandExecutionError(`Zhihu answer comments returned conflicting data for comment ${descriptor.id}`);
        }
        const oldCount = previous.comment.child_comment_count;
        const newCount = comment.child_comment_count;
        if (Number.isInteger(newCount) && (!Number.isInteger(oldCount) || newCount > oldCount)) {
            previous.comment = { ...previous.comment, child_comment_count: newCount };
        }
    }
}
async function fetchPages(page, options) {
    const { firstUrl, limit, label, role, expectedRootId = '', normalizeNext, notFoundDetail = '' } = options;
    const byId = new Map();
    const visited = new Set();
    const maxPages = Math.ceil(limit / PAGE_SIZE) + PAGE_OVERLAP_ALLOWANCE;
    let pageCount = 0;
    let url = firstUrl;
    while (byId.size < limit) {
        if (visited.has(url)) throw new CommandExecutionError(`Zhihu ${label} pagination returned a repeated next URL`);
        if (pageCount >= maxPages) throw new CommandExecutionError(`Zhihu ${label} pagination exceeded its fetch budget`);
        visited.add(url);
        pageCount += 1;
        const payload = await fetchCommentPage(page, url, label, notFoundDetail);
        addPageComments(byId, payload.data, role, expectedRootId);
        if (payload.paging.is_end || byId.size >= limit) break;
        url = normalizeNext(payload.paging.next);
        if (!url) throw new CommandExecutionError(`Zhihu ${label} pagination returned a malformed next URL`);
    }
    return [...byId.values()].slice(0, limit).map(({ comment }) => comment);
}
export function fetchRootComments(page, answerId, order, limit) {
    const apiOrder = order === 'latest' ? 'ts' : 'score';
    const path = `/api/v4/comment_v5/answers/${answerId}/root_comment`;
    return fetchPages(page, {
        firstUrl: `https://www.zhihu.com${path}?order_by=${apiOrder}&limit=${PAGE_SIZE}&offset=`,
        limit,
        label: 'answer root comments',
        role: 'root',

View on GitHub (pinned to 49907e53dc)

Solutions

Lower the limit to something close to the answer's real comment count.
Retry with the other sort order to avoid unstable pagination windows on actively-commented answers.
Fetch during a quieter window or snapshot the answer before comments change rapidly.
If duplicates are the cause, increase PAGE_OVERLAP_ALLOWANCE in the library (or via config) to tolerate overlap.

Example fix

// before
const comments = await fetchRootComments(page, answerId, 'latest', 10000);
// after: size the limit to the actual count
const limit = Math.min(answer.comment_count, 500);
const comments = await fetchRootComments(page, answerId, 'latest', limit);

Defensive patterns

Strategy: validation

Validate before calling

// request only as many comments as actually exist
const safeLimit = Math.min(answer.comment_count ?? 0, 500);
const comments = safeLimit > 0
  ? await fetchRootComments(page, answerId, 'latest', safeLimit)
  : [];

Type guard

function hasUsableCount(answer) {
  return Number.isInteger(answer?.comment_count) && answer.comment_count >= 0;
}

Try / catch

try {
  const comments = await fetchRootComments(page, answerId, order, limit);
} catch (err) {
  if (String(err.message).includes('exceeded its fetch budget')) {
    console.warn(`Reducing limit and retrying: ${limit}`);
    return fetchRootComments(page, answerId, order, Math.ceil(limit / 2));
  }
  throw err;
}

Prevention

Size limit from the answer's comment_count instead of a large fixed value.
Avoid fetching during rapid comment churn (sort instability causes overlap).
Raise PAGE_OVERLAP_ALLOWANCE only if you know pages legitimately overlap.

When it happens

Trigger: byId never reaches limit within maxPages requests — e.g. pages full of duplicates (each page adds few/no new ids) or paging.next keeps yielding valid but non-advancing pages without repeating a URL.

Common situations: Requesting a limit far larger than the number of existing comments while the API keeps returning overlapping pages; sort instability causing rows to shuffle between pages; or a very low limit with PAGE_OVERLAP_ALLOWANCE too small for a chatty endpoint.

Related errors

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/5931f41d424a7cf2. Report an issue: GitHub.