{"record":{"id":"453eec4f060b6b00","repo":"thedotmack/claude-mem","slug":"push-response-did-not-ack-the-canary-op","errorCode":null,"errorMessage":"push response did not ack the canary op","messagePattern":"push response did not ack the canary op","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"workers/sync-hub/canary/canary.ts","lineNumber":247,"sourceCode":"\t\tconst res = await fetch(`${this.args.hub}/v1/sync/ops`, {\n\t\t\tmethod: \"POST\",\n\t\t\theaders: { ...this.headers(deviceId), \"Content-Type\": \"application/json\" },\n\t\t\tbody: JSON.stringify({\n\t\t\t\tprotocol_version: 2,\n\t\t\t\tops: [{ body, operation_sha256: operationSha256 }],\n\t\t\t}),\n\t\t\tsignal: AbortSignal.timeout(this.args.timeoutMs),\n\t\t});\n\t\tthis.noteMode(res);\n\t\tif (!res.ok) {\n\t\t\tthrow new Error(`push ${res.status}: ${(await res.text().catch(() => \"\")).slice(0, 200)}`);\n\t\t}\n\t\tconst parsed = (await res.json()) as {\n\t\t\tacked: Array<{ operation_sha256: string; seq: string }>;\n\t\t\thead_seq: string;\n\t\t};\n\t\tconst ack = parsed.acked.find((a) => a.operation_sha256 === operationSha256);\n\t\tif (!ack) throw new Error(\"push response did not ack the canary op\");\n\t\treturn { seq: ack.seq, headSeq: parsed.head_seq, opSha: operationSha256 };\n\t}\n\n\tprivate async pull(\n\t\tdeviceId: string,\n\t\tsince: string,\n\t): Promise<{ ops: Array<{ seq: string; operation_sha256: string }>; headSeq: string }> {\n\t\tconst res = await fetch(`${this.args.hub}/v1/sync/changes?since=${since}&limit=500`, {\n\t\t\theaders: this.headers(deviceId),\n\t\t\tsignal: AbortSignal.timeout(this.args.timeoutMs),\n\t\t});\n\t\tthis.noteMode(res);\n\t\tif (!res.ok) {\n\t\t\tthrow new Error(`pull ${res.status}: ${(await res.text().catch(() => \"\")).slice(0, 200)}`);\n\t\t}\n\t\tconst parsed = (await res.json()) as {\n\t\t\tops: Array<{ seq: string; operation_sha256: string }>;\n\t\t\thead_seq: string;","sourceCodeStart":229,"sourceCodeEnd":265,"githubUrl":"https://github.com/thedotmack/claude-mem/blob/d8bc9755e74915e5c3b999181e10a67c889bce2a/workers/sync-hub/canary/canary.ts#L229-L265","documentation":"This error is thrown by the canary's `push` method when the sync hub acknowledges the HTTP request but the response body does not contain the just-pushed operation's SHA-256 in its `acked` array. It means the hub either dropped, rejected, or silently discarded the operation despite returning a successful response. The canary uses this strict ack check to detect data loss between client and server that a status-code-only check would miss.","triggerScenarios":"The hub returned HTTP 2xx but `parsed.acked` contains no entry whose `operation_sha256` equals the client-computed `operationSha256` — e.g. the hub deduplicated/rejected the op server-side, the client hashed the op with a different canonicalization than the server, or the hub truncated its ack list.","commonSituations":"A server deploy changed the op-hashing algorithm or canonical JSON rules so server-computed SHAs no longer match the client's; the hub is behind a proxy that strips or rewrites the request body; a partial ack limit drops newer ops; or the canary runs against a staging hub with an older schema where `acked` uses a different field name.","solutions":["Compare the client's operationSha256 against the raw op stored on the hub (recompute the canonical hash both sides) to find where the hashing diverges.","Log the full `parsed.acked` array on mismatch to see whether the hub acked under a different SHA or acked nothing at all.","Verify client and server use the same canonicalJson/canonicalization version before hashing the operation payload.","Retry the push once with backoff, then fail loudly — a missing ack after retry indicates real data loss and should page an operator."],"exampleFix":"// before\nconst ack = parsed.acked.find((a) => a.operation_sha256 === operationSha256);\nif (!ack) throw new Error(\"push response did not ack the canary op\");\n\n// after\nconst ack = parsed.acked.find((a) => a.operation_sha256 === operationSha256);\nif (!ack) {\n  console.error(JSON.stringify({ sent: operationSha256, acked: parsed.acked, headSeq: parsed.head_seq }));\n  throw new Error(`push response did not ack canary op ${operationSha256}; acked=${JSON.stringify(parsed.acked)}`);\n}","handlingStrategy":"validation","validationCode":"// pre-check the op shape before hashing/pushing\nfunction isPushableOp(op: unknown): op is { payload: Record<string, unknown> } {\n  return typeof op === 'object' && op !== null && 'payload' in op &&\n    typeof (op as any).payload === 'object';\n}\n// and after push, verify ack shape before .find():\nfunction hasAckShape(res: unknown): res is { acked: Array<{ operation_sha256: string; seq: string }>; head_seq: string } {\n  const r = res as any;\n  return r && Array.isArray(r.acked) && typeof r.head_seq === 'string';\n}","typeGuard":"function isAckFor(ack: unknown, sha: string): ack is { operation_sha256: string; seq: string } {\n  return typeof ack === 'object' && ack !== null &&\n    (ack as any).operation_sha256 === sha && typeof (ack as any).seq === 'string';\n}","tryCatchPattern":"try {\n  const result = await canary.pushed(op);\n} catch (err) {\n  if (err instanceof Error && err.message.includes('did not ack the canary op')) {\n    log({ event: 'ack_missing', detail: err.message });\n    // treat as potential data loss: alert, do not silently retry forever\n  } else throw err;\n}","preventionTips":["Keep client and server canonicalization/hashing code generated from one shared module.","Log the full acked array on every mismatch for postmortems.","Alert (don't ignore) when an ack is missing after a retry — it usually means real data loss.","Add a contract test asserting the hub acks every op it accepts."],"tags":["sync","data-loss","acknowledgement-missing","network"],"backgroundTag":"unexpected-api-response-shape","analyzedSha":"d8bc9755e74915e5c3b999181e10a67c889bce2a","analyzedAt":"2026-09-17T16:40:26.182Z","contentChangedAt":"2026-09-17T16:40:26.182Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}