{"record":{"id":"3484cc9e37425737","repo":"jackwener/OpenCLI","slug":"unexpected-xiaohongshu-search-harvest-payload-shap","errorCode":null,"errorMessage":"Unexpected Xiaohongshu search harvest payload shape; expected rows plus typed diagnostics.","messagePattern":"Unexpected Xiaohongshu search harvest payload shape; expected rows plus typed diagnostics\\.","errorType":"exception","errorClass":"CommandExecutionError","httpStatus":null,"severity":"error","filePath":"clis/xiaohongshu/search.js","lineNumber":319,"sourceCode":"        throw new CommandExecutionError(`Unexpected Xiaohongshu search harvest row ${index + 1} URL; expected a trusted note URL.`);\n    }\n    if (!isTrustedAuthorUrl(row.author_url, webHost)) {\n        throw new CommandExecutionError(`Unexpected Xiaohongshu search harvest row ${index + 1} author URL; expected a trusted profile URL.`);\n    }\n    return row;\n}\nfunction requireHarvestPayload(payload, webHost) {\n    const result = unwrapEvaluateResult(payload);\n    const diag = result?.diag;\n    if (!result || typeof result !== 'object' || Array.isArray(result) || !Array.isArray(result.rows) ||\n        !diag || typeof diag !== 'object' || Array.isArray(diag) ||\n        typeof diag.securityBlock !== 'boolean' || typeof diag.stopReason !== 'string' ||\n        !Number.isFinite(diag.scrollHeight) || diag.scrollHeight < 0 ||\n        !Number.isFinite(diag.clientHeight) || diag.clientHeight < 0 ||\n        !Number.isSafeInteger(diag.cardCount) || diag.cardCount < 0 ||\n        !(diag.feedClientHeight === null || (Number.isFinite(diag.feedClientHeight) && diag.feedClientHeight >= 0)) ||\n        !Number.isSafeInteger(diag.distinctCardTops) || diag.distinctCardTops < 0) {\n        throw new CommandExecutionError('Unexpected Xiaohongshu search harvest payload shape; expected rows plus typed diagnostics.');\n    }\n    result.rows = result.rows.map((row, index) => requireTrustedHarvestRow(row, index, webHost));\n    return result;\n}\nexport function parseLimit(raw) {\n    const parsed = Number(raw ?? 20);\n    if (!Number.isFinite(parsed) || !Number.isInteger(parsed)) {\n        throw new ArgumentError(`--limit must be an integer between 1 and 100, got ${JSON.stringify(raw)}`);\n    }\n    if (parsed < 1 || parsed > 100) {\n        throw new ArgumentError(`--limit must be between 1 and 100, got ${parsed}`);\n    }\n    return parsed;\n}\n\nfunction resolveSearchFilters(kwargs) {\n    return SEARCH_FILTERS.map((definition) => {\n        const value = kwargs[definition.arg] ?? definition.defaultValue;","sourceCodeStart":301,"sourceCodeEnd":337,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/xiaohongshu/search.js#L301-L337","documentation":"requireHarvestPayload validates the shape of the payload produced by the browser-side harvest script: rows array plus a typed diag object whose fields (securityBlock, stopReason, scrollHeight, clientHeight, cardCount, feedClientHeight, distinctCardTops) must all have exact types/ranges. If any field is missing or mistyped it throws this CommandExecutionError, meaning the in-page script returned something other than the agreed contract.","triggerScenarios":"The page.evaluate harvest script returns null/undefined, an array instead of {rows, diag}, or a diag object missing/with mistyped fields (e.g. cardCount as string, feedClientHeight as undefined instead of null).","commonSituations":"Xiaohongshu served a captcha/login page so the in-page script bailed early and returned a partial object; a browser or userscript wrapped the return value (e.g. via JSON round-trip turning numbers into strings); a stale bundled page script no longer matches the Node-side validation contract after an upgrade.","solutions":["Dump the raw payload (JSON.stringify of the evaluate result) to see which diag field or structure deviates from the contract","Ensure the page is fully loaded and not showing a login/captcha/anti-bot interstitial before running the harvest script","Rebuild/sync the injected page script and the Node-side requireHarvestPayload validator so both agree on the {rows, diag} contract","Check that unwrapEvaluateResult is handling the driver's return format (some drivers wrap results in {value} or serialize undefined to null)"],"exampleFix":"// before\nconst result = unwrapEvaluateResult(payload);\n// after\nconst result = unwrapEvaluateResult(payload);\nif (!result || !Array.isArray(result.rows) || !result.diag) {\n  console.error('raw harvest payload:', JSON.stringify(payload));\n  throw new CommandExecutionError('Xiaohongshu harvest returned no data; page may be showing a login/captcha screen.');\n}","handlingStrategy":"type-guard","validationCode":"function looksLikeHarvestPayload(p) {\n  const r = p?.value ?? p;\n  return !!r && typeof r === 'object' && Array.isArray(r.rows) &&\n    typeof r.diag === 'object' && r.diag !== null &&\n    ['securityBlock','stopReason','scrollHeight','clientHeight','cardCount','feedClientHeight','distinctCardTops'].every(k => k in r.diag);\n}","typeGuard":"const isHarvestPayload = (r) =>\n  r !== null && typeof r === 'object' && !Array.isArray(r) &&\n  Array.isArray(r.rows) && r.diag !== null && typeof r.diag === 'object' &&\n  typeof r.diag.securityBlock === 'boolean' &&\n  typeof r.diag.stopReason === 'string' &&\n  Number.isSafeInteger(r.diag.cardCount) && r.diag.cardCount >= 0;","tryCatchPattern":"try {\n  const { rows, diag } = requireHarvestPayload(raw, webHost);\n} catch (e) {\n  if (e.message.includes('harvest payload shape')) {\n    console.error('raw payload:', JSON.stringify(raw));\n    // treat as page-level failure (login wall / DOM change) and re-run or alert\n  } else throw e;\n}","preventionTips":["Assert the {rows, diag} contract in tests with representative payloads","Ensure the page is fully loaded and free of interstitials before evaluating the harvest script","Keep the injected page script and the Node validator versioned together","Beware JSON round-trips that convert numbers to strings or undefined to null in diag fields"],"tags":["schema-validation","scraping","payload-shape","xiaohongshu"],"backgroundTag":"schema-validation-failed","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}