What causes "${field} is not canonical base64" in JuliusBrussee/caveman?

Re-encode the value canonically: Buffer.from(value,'base64') then .toString('base64'), and put the result in the journal

JuliusBrussee/caveman · error · Error

is not canonical base64

Error message

${field} is not canonical base64

What it means

decodePendingBytes also requires base64 values to be canonical: re-encoding the decoded bytes must reproduce the original string exactly. Non-canonical forms (wrong padding, embedded whitespace, different alphabet usage) are rejected so journal integrity hashes compare deterministically.

Solutions

Re-encode the value canonically: Buffer.from(value,'base64') then .toString('base64'), and put the result in the journal
Remove whitespace/line breaks and fix '=' padding to the correct multiple of 4
Regenerate the journal by re-running the transaction rather than repairing by hand

Example fix

// before
{ "config_before_base64": "eyJhIjoxfQ" } // missing padding
// after
{ "config_before_base64": "eyJhIjoxfQ==" }

Defensive patterns

Strategy: validation

Validate before calling

function isCanonicalBase64(v: string): boolean {
  return Buffer.from(v, 'base64').toString('base64') === v;
}
// check before writing/repairing journal fields
['config_before_base64','marker_before_base64','marker_after_base64']
  .forEach(k => { if (typeof j[k] === 'string' && !isCanonicalBase64(j[k])) throw new Error(k); });

Type guard

function isCanonicalBase64Value(v: unknown): v is string {
  return typeof v === 'string' && Buffer.from(v, 'base64').toString('base64') === v;
}

Try / catch

try {
  decodePendingBytes(value, field);
} catch (e) {
  if (e instanceof Error && e.message.includes('not canonical base64')) {
    value = Buffer.from(value as string, 'base64').toString('base64'); // re-canonicalize
  } else throw e;
}

Prevention

Always produce base64 with standard encoding, no line wrapping
Fix padding: length must be a multiple of 4 with correct '='
Never paste base64 from sources that add whitespace; strip it first
Round-trip check (decode then re-encode) before persisting base64

When it happens

Trigger: A journal file whose *_base64 field uses non-canonical base64: missing '=' padding, extra padding, line breaks or spaces inside the value, or base64url encoding instead of standard base64.

Common situations: Hand-copying a base64 blob from docs/logs into the journal; an external editor wrapping long lines; using a base64url encoder when standard base64 is expected.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/d7de81e2aff56591. Report an issue: GitHub.

Appendix: source

Thrown at packages/cli/src/index.ts:12697

function mcpMarkerBytes(mcp: { command: string; args: string[] }, tool: string, configPath?: string): Buffer {
  return Buffer.from(JSON.stringify({
    ...(configPath ? { schema_version: 1 } : {}),
    tool,
    command: mcp.command,
    args: mcp.args,
    ...(configPath ? { config_path: canonicalMcpConfigPath(configPath) } : {}),
  }, null, 2) + "\n");
}

function validMcpMarkerBytes(bytes: Buffer, agent: "kilo" | "qwen", serverName: string): boolean {
  return parseMcpServerMarkerBytes(agent, serverName, bytes) !== null;
}

function decodePendingBytes(value: unknown, field: string): Buffer | null {
  if (value === null) return null;
  if (typeof value !== "string") throw new Error(`${field} must be base64 or null`);
  const bytes = Buffer.from(value, "base64");
  if (bytes.toString("base64") !== value) throw new Error(`${field} is not canonical base64`);
  return bytes;
}

function validOptionalHash(value: unknown): value is string | null {
  return value === null || (typeof value === "string" && /^sha256:[0-9a-f]{64}$/.test(value));
}

type ReadOwnedMcpPendingJournal = {
  journal: OwnedMcpPendingJournal;
  configBefore: Buffer | null;
  markerBefore: Buffer | null;
  markerAfter: Buffer | null;
  path: string;
  bytes: Buffer;
};

function ownedMcpPendingLabel(agent?: string, serverName?: string): string {
  return agent && serverName ? `${agent} ${serverName} MCP` : "owned MCP";

View on GitHub (pinned to 3ee70a1026)