{"record":{"id":"8669e8b903779922","repo":"paperclipai/paperclip","slug":"photon-state-changed-concurrently-retry","errorCode":null,"errorMessage":"Photon state changed concurrently; retry","messagePattern":"Photon state changed concurrently; retry","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"server/src/services/photon/state.ts","lineNumber":37,"sourceCode":"  }\n  async update<T>(key: string, update: (current: T | null) => T): Promise<T> {\n    for (let attempt = 0; attempt < 32; attempt++) {\n      const row = await this.persistence.read(this.scope, this.key(key));\n      const value = update(row ? (row.value as T) : null);\n      if (Buffer.byteLength(JSON.stringify(value)) > 512 * 1024)\n        throw new Error(\"Photon state record is too large\");\n      if (\n        await this.persistence.compareAndSet({\n          ...this.scope,\n          key: this.key(key),\n          expectedVersion: row?.version ?? null,\n          expiresAt: null,\n          value,\n        })\n      )\n        return value;\n    }\n    throw new Error(\"Photon state changed concurrently; retry\");\n  }\n}\n","sourceCodeStart":19,"sourceCodeEnd":40,"githubUrl":"https://github.com/paperclipai/paperclip/blob/3f1d897a7c018d76563a21c6e39c3c9b03933622/server/src/services/photon/state.ts#L19-L40","documentation":"PhotonState.update() uses optimistic concurrency: it reads the row, applies the updater, then compareAndSet()s against the version it read. If CAS fails, it retries; after 32 failed attempts it gives up and throws this Error. It means another writer kept modifying the same state key faster than this caller could commit — the update was never applied.","triggerScenarios":"Calling update() on a state key while many concurrent writers (other agents, heartbeats, parallel issue runs in the same company/endpoint scope) repeatedly mutate the same key, so all 32 CAS attempts observe a changed version.","commonSituations":"Hot keys written by every run in a company (counters, shared queues); several agents retrying the same key after an earlier failure storm; long updater work combined with frequent writers, so this writer always loses the race.","solutions":["Catch the error and retry the whole update() call at the application level with backoff","Reduce contention: shard the hot key into per-worker keys and merge lazily, or narrow the update scope","Make the updater fast and pure (no awaits, no heavy work) to shrink the CAS race window","Serialize writes through a single queue/owner per key instead of many concurrent updaters"],"exampleFix":"// before: single call, unhandled contention\nconst v = await state.update(\"queue\", enqueue(item));\n// after: app-level retry with backoff\nlet v;\nfor (let i = 0; i < 5; i++) {\n  try { v = await state.update(\"queue\", enqueue(item)); break; }\n  catch (e) {\n    if (String(e.message).includes(\"changed concurrently\")) {\n      await sleep(50 * 2 ** i + Math.random() * 50);\n      continue;\n    }\n    throw e;\n  }\n}","handlingStrategy":"retry","validationCode":"// none: contention is a runtime race, not a pre-checkable condition; minimize it instead\n// keep updater synchronous and cheap — no awaits inside the update callback\nconst updater = (cur) => ({ count: (cur?.count ?? 0) + 1 }); // pure, fast => small CAS window","typeGuard":"function isConcurrentStateError(e: unknown): boolean {\n  return e instanceof Error && e.message === \"Photon state changed concurrently; retry\";\n}","tryCatchPattern":"async function updateWithRetry<T>(state, key, updater, tries = 5) {\n  for (let i = 0; ; i++) {\n    try { return await state.update(key, updater); }\n    catch (e) {\n      if (isConcurrentStateError(e) && i < tries) {\n        await sleep(2 ** i * 25 + Math.random() * 25);\n        continue;\n      }\n      throw e;\n    }\n  }\n}","preventionTips":["Keep the updater function synchronous and free of I/O — long updaters widen the CAS race window","Avoid many writers on one hot key; shard keys per worker or serialize writes through one owner","Wrap state.update() in a small backoff-retry helper by default in app code","Alert on repeated concurrent-state errors — they signal key-level hot spotting, not random flake"],"tags":["concurrency","optimistic-locking","state","retry"],"backgroundTag":"invalid-state-transition","analyzedSha":"3f1d897a7c018d76563a21c6e39c3c9b03933622","analyzedAt":"2026-09-18T08:03:59.046Z","contentChangedAt":"2026-09-18T08:03:59.046Z","schemaVersion":2},"datasetVersion":"2026-09-22T10:30:35.592Z"}