thedotmack/claude-mem · error · Error

push response did not ack the canary op

Error message

push response did not ack the canary op

What it means

This error is thrown by the canary's `push` method when the sync hub acknowledges the HTTP request but the response body does not contain the just-pushed operation's SHA-256 in its `acked` array. It means the hub either dropped, rejected, or silently discarded the operation despite returning a successful response. The canary uses this strict ack check to detect data loss between client and server that a status-code-only check would miss.

Solutions

  1. Compare the client's operationSha256 against the raw op stored on the hub (recompute the canonical hash both sides) to find where the hashing diverges.
  2. Log the full `parsed.acked` array on mismatch to see whether the hub acked under a different SHA or acked nothing at all.
  3. Verify client and server use the same canonicalJson/canonicalization version before hashing the operation payload.
  4. Retry the push once with backoff, then fail loudly — a missing ack after retry indicates real data loss and should page an operator.

Example fix

// before
const ack = parsed.acked.find((a) => a.operation_sha256 === operationSha256);
if (!ack) throw new Error("push response did not ack the canary op");

// after
const ack = parsed.acked.find((a) => a.operation_sha256 === operationSha256);
if (!ack) {
  console.error(JSON.stringify({ sent: operationSha256, acked: parsed.acked, headSeq: parsed.head_seq }));
  throw new Error(`push response did not ack canary op ${operationSha256}; acked=${JSON.stringify(parsed.acked)}`);
}
Defensive patterns

Strategy: validation

Validate before calling

// pre-check the op shape before hashing/pushing
function isPushableOp(op: unknown): op is { payload: Record<string, unknown> } {
  return typeof op === 'object' && op !== null && 'payload' in op &&
    typeof (op as any).payload === 'object';
}
// and after push, verify ack shape before .find():
function hasAckShape(res: unknown): res is { acked: Array<{ operation_sha256: string; seq: string }>; head_seq: string } {
  const r = res as any;
  return r && Array.isArray(r.acked) && typeof r.head_seq === 'string';
}

Type guard

function isAckFor(ack: unknown, sha: string): ack is { operation_sha256: string; seq: string } {
  return typeof ack === 'object' && ack !== null &&
    (ack as any).operation_sha256 === sha && typeof (ack as any).seq === 'string';
}

Try / catch

try {
  const result = await canary.pushed(op);
} catch (err) {
  if (err instanceof Error && err.message.includes('did not ack the canary op')) {
    log({ event: 'ack_missing', detail: err.message });
    // treat as potential data loss: alert, do not silently retry forever
  } else throw err;
}

Prevention

When it happens

Trigger: The hub returned HTTP 2xx but `parsed.acked` contains no entry whose `operation_sha256` equals the client-computed `operationSha256` — e.g. the hub deduplicated/rejected the op server-side, the client hashed the op with a different canonicalization than the server, or the hub truncated its ack list.

Common situations: A server deploy changed the op-hashing algorithm or canonical JSON rules so server-computed SHAs no longer match the client's; the hub is behind a proxy that strips or rewrites the request body; a partial ack limit drops newer ops; or the canary runs against a staging hub with an older schema where `acked` uses a different field name.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of thedotmack/claude-mem@d8bc9755e7 (2026-09-17). Data as JSON: /api/errors/453eec4f060b6b00. Report an issue: GitHub.

Appendix: source

Thrown at workers/sync-hub/canary/canary.ts:247

		const res = await fetch(`${this.args.hub}/v1/sync/ops`, {
			method: "POST",
			headers: { ...this.headers(deviceId), "Content-Type": "application/json" },
			body: JSON.stringify({
				protocol_version: 2,
				ops: [{ body, operation_sha256: operationSha256 }],
			}),
			signal: AbortSignal.timeout(this.args.timeoutMs),
		});
		this.noteMode(res);
		if (!res.ok) {
			throw new Error(`push ${res.status}: ${(await res.text().catch(() => "")).slice(0, 200)}`);
		}
		const parsed = (await res.json()) as {
			acked: Array<{ operation_sha256: string; seq: string }>;
			head_seq: string;
		};
		const ack = parsed.acked.find((a) => a.operation_sha256 === operationSha256);
		if (!ack) throw new Error("push response did not ack the canary op");
		return { seq: ack.seq, headSeq: parsed.head_seq, opSha: operationSha256 };
	}

	private async pull(
		deviceId: string,
		since: string,
	): Promise<{ ops: Array<{ seq: string; operation_sha256: string }>; headSeq: string }> {
		const res = await fetch(`${this.args.hub}/v1/sync/changes?since=${since}&limit=500`, {
			headers: this.headers(deviceId),
			signal: AbortSignal.timeout(this.args.timeoutMs),
		});
		this.noteMode(res);
		if (!res.ok) {
			throw new Error(`pull ${res.status}: ${(await res.text().catch(() => "")).slice(0, 200)}`);
		}
		const parsed = (await res.json()) as {
			ops: Array<{ seq: string; operation_sha256: string }>;
			head_seq: string;

View on GitHub (pinned to d8bc9755e7)