jackwener/OpenCLI · error · CommandExecutionError

Zhihu pagination returned a malformed next URL

Error message

Zhihu ${label} pagination returned a malformed next URL

What it means

After a page that is not is_end, fetchPages normalizes payload.paging.next with normalizeNext; if the result is falsy the next link is absent or malformed, making continuation impossible. The library throws instead of silently truncating results.

Solutions

1. Log payload.paging.next on the failing page and check what normalizeNext rejects about it.
2. Update/patch the library's normalizeNext to accept the new URL shape if Zhihu changed formats.
3. Ensure proxies pass the next URL through verbatim; disable URL rewriting middleware.
4. Retry the fetch in case the page was truncated by a transient network error.

Example fix

// before: proxy rewrites https to http and normalizeNext rejects
const page = await fetch(nextUrl.replace('https://', 'http://'));
// after: keep the original URL
const page = await fetch(nextUrl);

// pre-validate paging payloads before handing them to the walker
function nextUrlIsUsable(payload) {
  if (payload?.paging?.is_end) return true;
  const next = payload?.paging?.next;
  return typeof next === 'string' && /^https?:\/\//.test(next);
}

function isAbsoluteHttpUrl(v) {
  if (typeof v !== 'string' || v === '') return false;
  try { const u = new URL(v); return u.protocol === 'https:' || u.protocol === 'http:'; }
  catch { return false; }
}

try {
  const comments = await fetchRootComments(page, answerId, order, limit);
} catch (err) {
  if (String(err.message).includes('malformed next URL')) {
    console.warn('Paging stopped at a bad next link; keeping partial results');
    return partialResultsSoFar;
  }
  throw err;
}

When it happens

Trigger: A non-final page whose paging.next is null, an empty string, a relative URL normalizeNext cannot resolve, or a URL failing normalization (bad scheme/host).

Common situations: Zhihu changing the next-link format (e.g. protocol-relative or a new host), proxies rewriting the URL into something normalizeNext rejects, or API responses truncated mid-field by a flaky network layer.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Zhihu answer comments returned conflicting data for comment
• Zhihu pagination exceeded its fetch budget
• Zhihu pagination returned a repeated next URL
• Zhihu search pagination returned malformed next URL
• Bilibili view API did not return pages[] for --page…

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/449d937017b050f0. Report an issue: GitHub.

Appendix: source

Thrown at clis/zhihu/answer-comments-helpers.js:147

    }
}
async function fetchPages(page, options) {
    const { firstUrl, limit, label, role, expectedRootId = '', normalizeNext, notFoundDetail = '' } = options;
    const byId = new Map();
    const visited = new Set();
    const maxPages = Math.ceil(limit / PAGE_SIZE) + PAGE_OVERLAP_ALLOWANCE;
    let pageCount = 0;
    let url = firstUrl;
    while (byId.size < limit) {
        if (visited.has(url)) throw new CommandExecutionError(`Zhihu ${label} pagination returned a repeated next URL`);
        if (pageCount >= maxPages) throw new CommandExecutionError(`Zhihu ${label} pagination exceeded its fetch budget`);
        visited.add(url);
        pageCount += 1;
        const payload = await fetchCommentPage(page, url, label, notFoundDetail);
        addPageComments(byId, payload.data, role, expectedRootId);
        if (payload.paging.is_end || byId.size >= limit) break;
        url = normalizeNext(payload.paging.next);
        if (!url) throw new CommandExecutionError(`Zhihu ${label} pagination returned a malformed next URL`);
    }
    return [...byId.values()].slice(0, limit).map(({ comment }) => comment);
}
export function fetchRootComments(page, answerId, order, limit) {
    const apiOrder = order === 'latest' ? 'ts' : 'score';
    const path = `/api/v4/comment_v5/answers/${answerId}/root_comment`;
    return fetchPages(page, {
        firstUrl: `https://www.zhihu.com${path}?order_by=${apiOrder}&limit=${PAGE_SIZE}&offset=`,
        limit,
        label: 'answer root comments',
        role: 'root',
        normalizeNext: (next) => normalizePageUrl(next, path, {
            order_by: apiOrder,
            limit: String(PAGE_SIZE),
            offset: null,
        }),
        notFoundDetail: `No Zhihu answer comments resource was found for ${answerId}.`,
    });

View on GitHub (pinned to 49907e53dc)