{"record":{"id":"b496204d91f5460e","repo":"jackwener/OpenCLI","slug":"archive-snapshots-returned-malformed-cdx-payload","errorCode":null,"errorMessage":"archive snapshots returned malformed CDX payload: missing \"${name}\" column","messagePattern":"archive snapshots returned malformed CDX payload: missing \"(.+?)\" column","errorType":"exception","errorClass":"CommandExecutionError","httpStatus":null,"severity":"error","filePath":"clis/archive/snapshots.js","lineNumber":17,"sourceCode":"// archive snapshots: Wayback Machine CDX history for a URL.\nimport { cli, Strategy } from '@jackwener/opencli/registry';\nimport {\n    ArgumentError,\n    CommandExecutionError,\n    EmptyResultError,\n} from '@jackwener/opencli/errors';\n\nfunction buildWaybackUrl(timestamp, original) {\n    if (!timestamp || !original) return '';\n    return `https://web.archive.org/web/${timestamp}/${original}`;\n}\n\nfunction requireCdxColumn(cols, name) {\n    const index = cols[name];\n    if (!Number.isInteger(index)) {\n        throw new CommandExecutionError(`archive snapshots returned malformed CDX payload: missing \"${name}\" column`);\n    }\n    return index;\n}\n\ncli({\n    site: 'archive',\n    name: 'snapshots',\n    access: 'read',\n    description: 'List Wayback Machine snapshots over time for a URL via the CDX API.',\n    domain: 'archive.org',\n    strategy: Strategy.PUBLIC,\n    browser: false,\n    args: [\n        { name: 'url', positional: true, required: true, help: 'URL to look up (with or without scheme).' },\n        { name: 'from', type: 'string', required: false, help: 'Earliest year/timestamp (YYYY[MM[DD[hh[mm[ss]]]]])' },\n        { name: 'to', type: 'string', required: false, help: 'Latest year/timestamp (YYYY[MM[DD[hh[mm[ss]]]]])' },\n        { name: 'limit', type: 'int', default: 20, help: 'Max snapshots to return (max 1000).' },\n    ],","sourceCodeStart":1,"sourceCodeEnd":35,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/archive/snapshots.js#L1-L35","documentation":"CommandExecutionError from `requireCdxColumn` in `opencli archive snapshots`. The Wayback CDX API's JSON output includes a `columns` map naming each field's index; this helper asserts a needed column (timestamp, original, statuscode, mimetype) is present before reading rows. Missing column means the CDX payload's schema changed or is not the expected JSON format.","triggerScenarios":"Calling `opencli archive snapshots <url>` where the CDX response's data[0].columns object lacks the requested key (e.g. 'timestamp', 'original', 'statuscode', 'mimetype'), so cols[name] is not an integer.","commonSituations":"Wayback CDX API schema changes; response being an HTML error page or a differently-shaped JSON (e.g. plain text CDX parsed incorrectly); proxy/CDN mangling the response; hitting a deprecated endpoint version.","solutions":["Fetch the same CDX URL (http://web.archive.org/cdx/search/cdx?url=...&output=json) in a browser/curl and inspect data[0].columns to see the actual schema.","Retry later if Wayback is returning degraded responses (the schema may be intermittently wrong under load).","Update the opencli library if the CDX API renamed columns (check for a newer version).","Ensure no proxy is rewriting the response body."],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"// Inspect the CDX JSON shape before relying on column lookups\nconst resp = await fetch('http://web.archive.org/cdx/search/cdx?url=example.org&output=json');\nconst data = await resp.json();\nconst cols = data?.[0];\nconst required = ['timestamp', 'original', 'statuscode', 'mimetype'];\nconst missing = required.filter(k => !Number.isInteger(cols?.columns?.[k]));\nif (missing.length) console.warn(`CDX schema missing columns: ${missing}`);","typeGuard":"function hasCdxColumns(payload, names) {\n  const cols = payload?.[0]?.columns;\n  return !!cols && names.every(n => Number.isInteger(cols[n]));\n}","tryCatchPattern":"try {\n  const snaps = await run(['archive', 'snapshots', url]);\n} catch (e) {\n  if (/malformed CDX payload/.test(e.message)) {\n    // Wayback schema drift or degraded response: retry later or fetch CDX manually\n    return fallbackFetchCdxManually(url);\n  }\n  throw e;\n}","preventionTips":["Inspect data[0].columns from the CDX API when upgrading the library.","Detect HTML/error pages masquerading as JSON before parsing columns.","Keep the library updated — column names can change upstream."],"tags":["malformed-payload","api-schema","wayback","cdx","archive-org"],"backgroundTag":"schema-validation-failed","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}