jackwener/OpenCLI · error · CommandExecutionError

archive snapshots returned malformed CDX payload: missing "$

Error message

archive snapshots returned malformed CDX payload: missing "${name}" column

What it means

CommandExecutionError from `requireCdxColumn` in `opencli archive snapshots`. The Wayback CDX API's JSON output includes a `columns` map naming each field's index; this helper asserts a needed column (timestamp, original, statuscode, mimetype) is present before reading rows. Missing column means the CDX payload's schema changed or is not the expected JSON format.

Source

Thrown at clis/archive/snapshots.js:17

// archive snapshots: Wayback Machine CDX history for a URL.
import { cli, Strategy } from '@jackwener/opencli/registry';
import {
    ArgumentError,
    CommandExecutionError,
    EmptyResultError,
} from '@jackwener/opencli/errors';

function buildWaybackUrl(timestamp, original) {
    if (!timestamp || !original) return '';
    return `https://web.archive.org/web/${timestamp}/${original}`;
}

function requireCdxColumn(cols, name) {
    const index = cols[name];
    if (!Number.isInteger(index)) {
        throw new CommandExecutionError(`archive snapshots returned malformed CDX payload: missing "${name}" column`);
    }
    return index;
}

cli({
    site: 'archive',
    name: 'snapshots',
    access: 'read',
    description: 'List Wayback Machine snapshots over time for a URL via the CDX API.',
    domain: 'archive.org',
    strategy: Strategy.PUBLIC,
    browser: false,
    args: [
        { name: 'url', positional: true, required: true, help: 'URL to look up (with or without scheme).' },
        { name: 'from', type: 'string', required: false, help: 'Earliest year/timestamp (YYYY[MM[DD[hh[mm[ss]]]]])' },
        { name: 'to', type: 'string', required: false, help: 'Latest year/timestamp (YYYY[MM[DD[hh[mm[ss]]]]])' },
        { name: 'limit', type: 'int', default: 20, help: 'Max snapshots to return (max 1000).' },
    ],

View on GitHub (pinned to 49907e53dc)

Solutions

1. Fetch the same CDX URL (http://web.archive.org/cdx/search/cdx?url=...&output=json) in a browser/curl and inspect data[0].columns to see the actual schema.
2. Retry later if Wayback is returning degraded responses (the schema may be intermittently wrong under load).
3. Update the opencli library if the CDX API renamed columns (check for a newer version).
4. Ensure no proxy is rewriting the response body.

// Inspect the CDX JSON shape before relying on column lookups
const resp = await fetch('http://web.archive.org/cdx/search/cdx?url=example.org&output=json');
const data = await resp.json();
const cols = data?.[0];
const required = ['timestamp', 'original', 'statuscode', 'mimetype'];
const missing = required.filter(k => !Number.isInteger(cols?.columns?.[k]));
if (missing.length) console.warn(`CDX schema missing columns: ${missing}`);

function hasCdxColumns(payload, names) {
  const cols = payload?.[0]?.columns;
  return !!cols && names.every(n => Number.isInteger(cols[n]));
}

try {
  const snaps = await run(['archive', 'snapshots', url]);
} catch (e) {
  if (/malformed CDX payload/.test(e.message)) {
    // Wayback schema drift or degraded response: retry later or fetch CDX manually
    return fallbackFetchCdxManually(url);
  }
  throw e;
}

When it happens

Trigger: Calling `opencli archive snapshots <url>` where the CDX response's data[0].columns object lacks the requested key (e.g. 'timestamp', 'original', 'statuscode', 'mimetype'), so cols[name] is not an integer.

Common situations: Wayback CDX API schema changes; response being an HTML error page or a differently-shaped JSON (e.g. plain text CDX parsed incorrectly); proxy/CDN mangling the response; hitting a deprecated endpoint version.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• archive search returned malformed payload: result row is mis
• archive search returned malformed payload for "${id}": downl
• archive snapshots request failed: ${error?.message || error}
• archive snapshots failed: HTTP ${resp.status}
• archive item returned malformed payload: files must be an ar

AI-assisted analysis of jackwener/OpenCLI@49907e53dc (2026-08-29). Data as JSON: /api/errors/b496204d91f5460e. Report an issue: GitHub.