jackwener/OpenCLI · error · CommandExecutionError

Unexpected Xiaohongshu search harvest payload shape; expecte

Error message

Unexpected Xiaohongshu search harvest payload shape; expected rows plus typed diagnostics.

What it means

requireHarvestPayload validates the shape of the payload produced by the browser-side harvest script: rows array plus a typed diag object whose fields (securityBlock, stopReason, scrollHeight, clientHeight, cardCount, feedClientHeight, distinctCardTops) must all have exact types/ranges. If any field is missing or mistyped it throws this CommandExecutionError, meaning the in-page script returned something other than the agreed contract.

Source

Thrown at clis/xiaohongshu/search.js:319

        throw new CommandExecutionError(`Unexpected Xiaohongshu search harvest row ${index + 1} URL; expected a trusted note URL.`);
    }
    if (!isTrustedAuthorUrl(row.author_url, webHost)) {
        throw new CommandExecutionError(`Unexpected Xiaohongshu search harvest row ${index + 1} author URL; expected a trusted profile URL.`);
    }
    return row;
}
function requireHarvestPayload(payload, webHost) {
    const result = unwrapEvaluateResult(payload);
    const diag = result?.diag;
    if (!result || typeof result !== 'object' || Array.isArray(result) || !Array.isArray(result.rows) ||
        !diag || typeof diag !== 'object' || Array.isArray(diag) ||
        typeof diag.securityBlock !== 'boolean' || typeof diag.stopReason !== 'string' ||
        !Number.isFinite(diag.scrollHeight) || diag.scrollHeight < 0 ||
        !Number.isFinite(diag.clientHeight) || diag.clientHeight < 0 ||
        !Number.isSafeInteger(diag.cardCount) || diag.cardCount < 0 ||
        !(diag.feedClientHeight === null || (Number.isFinite(diag.feedClientHeight) && diag.feedClientHeight >= 0)) ||
        !Number.isSafeInteger(diag.distinctCardTops) || diag.distinctCardTops < 0) {
        throw new CommandExecutionError('Unexpected Xiaohongshu search harvest payload shape; expected rows plus typed diagnostics.');
    }
    result.rows = result.rows.map((row, index) => requireTrustedHarvestRow(row, index, webHost));
    return result;
}
export function parseLimit(raw) {
    const parsed = Number(raw ?? 20);
    if (!Number.isFinite(parsed) || !Number.isInteger(parsed)) {
        throw new ArgumentError(`--limit must be an integer between 1 and 100, got ${JSON.stringify(raw)}`);
    }
    if (parsed < 1 || parsed > 100) {
        throw new ArgumentError(`--limit must be between 1 and 100, got ${parsed}`);
    }
    return parsed;
}

function resolveSearchFilters(kwargs) {
    return SEARCH_FILTERS.map((definition) => {
        const value = kwargs[definition.arg] ?? definition.defaultValue;

View on GitHub (pinned to 49907e53dc)

Solutions

1. Dump the raw payload (JSON.stringify of the evaluate result) to see which diag field or structure deviates from the contract
2. Ensure the page is fully loaded and not showing a login/captcha/anti-bot interstitial before running the harvest script
3. Rebuild/sync the injected page script and the Node-side requireHarvestPayload validator so both agree on the {rows, diag} contract
4. Check that unwrapEvaluateResult is handling the driver's return format (some drivers wrap results in {value} or serialize undefined to null)

Example fix

// before
const result = unwrapEvaluateResult(payload);
// after
const result = unwrapEvaluateResult(payload);
if (!result || !Array.isArray(result.rows) || !result.diag) {
  console.error('raw harvest payload:', JSON.stringify(payload));
  throw new CommandExecutionError('Xiaohongshu harvest returned no data; page may be showing a login/captcha screen.');
}

function looksLikeHarvestPayload(p) {
  const r = p?.value ?? p;
  return !!r && typeof r === 'object' && Array.isArray(r.rows) &&
    typeof r.diag === 'object' && r.diag !== null &&
    ['securityBlock','stopReason','scrollHeight','clientHeight','cardCount','feedClientHeight','distinctCardTops'].every(k => k in r.diag);
}

const isHarvestPayload = (r) =>
  r !== null && typeof r === 'object' && !Array.isArray(r) &&
  Array.isArray(r.rows) && r.diag !== null && typeof r.diag === 'object' &&
  typeof r.diag.securityBlock === 'boolean' &&
  typeof r.diag.stopReason === 'string' &&
  Number.isSafeInteger(r.diag.cardCount) && r.diag.cardCount >= 0;

try {
  const { rows, diag } = requireHarvestPayload(raw, webHost);
} catch (e) {
  if (e.message.includes('harvest payload shape')) {
    console.error('raw payload:', JSON.stringify(raw));
    // treat as page-level failure (login wall / DOM change) and re-run or alert
  } else throw e;
}

When it happens

Trigger: The page.evaluate harvest script returns null/undefined, an array instead of {rows, diag}, or a diag object missing/with mistyped fields (e.g. cardCount as string, feedClientHeight as undefined instead of null).

Common situations: Xiaohongshu served a captcha/login page so the in-page script bailed early and returned a partial object; a browser or userscript wrapped the return value (e.g. via JSON round-trip turning numbers into strings); a stale bundled page script no longer matches the Node-side validation contract after an upgrade.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• LinkedIn messengerMessages API returned a malformed normaliz
• LinkedIn messengerMessages payload is missing the included e
• Unexpected Xiaohongshu search filter result shape.
• ${label} returned an unexpected payload shape; expected an o
• Barchart greeks returned an unreadable options payload

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/3484cc9e37425737. Report an issue: GitHub.