jackwener/OpenCLI · error · CommandExecutionError

Zhihu ${label} pagination returned a malformed next URL

Error message

Zhihu ${label} pagination returned a malformed next URL

What it means

After a page that is not is_end, fetchPages normalizes payload.paging.next with normalizeNext; if the result is falsy the next link is absent or malformed, making continuation impossible. The library throws instead of silently truncating results.

Source

Thrown at clis/zhihu/answer-comments-helpers.js:147

    }
}
async function fetchPages(page, options) {
    const { firstUrl, limit, label, role, expectedRootId = '', normalizeNext, notFoundDetail = '' } = options;
    const byId = new Map();
    const visited = new Set();
    const maxPages = Math.ceil(limit / PAGE_SIZE) + PAGE_OVERLAP_ALLOWANCE;
    let pageCount = 0;
    let url = firstUrl;
    while (byId.size < limit) {
        if (visited.has(url)) throw new CommandExecutionError(`Zhihu ${label} pagination returned a repeated next URL`);
        if (pageCount >= maxPages) throw new CommandExecutionError(`Zhihu ${label} pagination exceeded its fetch budget`);
        visited.add(url);
        pageCount += 1;
        const payload = await fetchCommentPage(page, url, label, notFoundDetail);
        addPageComments(byId, payload.data, role, expectedRootId);
        if (payload.paging.is_end || byId.size >= limit) break;
        url = normalizeNext(payload.paging.next);
        if (!url) throw new CommandExecutionError(`Zhihu ${label} pagination returned a malformed next URL`);
    }
    return [...byId.values()].slice(0, limit).map(({ comment }) => comment);
}
export function fetchRootComments(page, answerId, order, limit) {
    const apiOrder = order === 'latest' ? 'ts' : 'score';
    const path = `/api/v4/comment_v5/answers/${answerId}/root_comment`;
    return fetchPages(page, {
        firstUrl: `https://www.zhihu.com${path}?order_by=${apiOrder}&limit=${PAGE_SIZE}&offset=`,
        limit,
        label: 'answer root comments',
        role: 'root',
        normalizeNext: (next) => normalizePageUrl(next, path, {
            order_by: apiOrder,
            limit: String(PAGE_SIZE),
            offset: null,
        }),
        notFoundDetail: `No Zhihu answer comments resource was found for ${answerId}.`,
    });

View on GitHub (pinned to 49907e53dc)

Solutions

1. Log payload.paging.next on the failing page and check what normalizeNext rejects about it.
2. Update/patch the library's normalizeNext to accept the new URL shape if Zhihu changed formats.
3. Ensure proxies pass the next URL through verbatim; disable URL rewriting middleware.
4. Retry the fetch in case the page was truncated by a transient network error.

Example fix

// before: proxy rewrites https to http and normalizeNext rejects
const page = await fetch(nextUrl.replace('https://', 'http://'));
// after: keep the original URL
const page = await fetch(nextUrl);

// pre-validate paging payloads before handing them to the walker
function nextUrlIsUsable(payload) {
  if (payload?.paging?.is_end) return true;
  const next = payload?.paging?.next;
  return typeof next === 'string' && /^https?:\/\//.test(next);
}

function isAbsoluteHttpUrl(v) {
  if (typeof v !== 'string' || v === '') return false;
  try { const u = new URL(v); return u.protocol === 'https:' || u.protocol === 'http:'; }
  catch { return false; }
}

try {
  const comments = await fetchRootComments(page, answerId, order, limit);
} catch (err) {
  if (String(err.message).includes('malformed next URL')) {
    console.warn('Paging stopped at a bad next link; keeping partial results');
    return partialResultsSoFar;
  }
  throw err;
}

When it happens

Trigger: A non-final page whose paging.next is null, an empty string, a relative URL normalizeNext cannot resolve, or a URL failing normalization (bad scheme/host).

Common situations: Zhihu changing the next-link format (e.g. protocol-relative or a new host), proxies rewriting the URL into something normalizeNext rejects, or API responses truncated mid-field by a flaky network layer.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Zhihu answer comments returned conflicting data for comment
• Zhihu ${label} pagination returned a repeated next URL
• Zhihu ${label} pagination exceeded its fetch budget
• Zhihu search pagination returned malformed next URL
• Bilibili view API did not return pages[] for --page selectio

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/449d937017b050f0. Report an issue: GitHub.