paperclipai/paperclip · error · Error

Photon state changed concurrently; retry

Error message

Photon state changed concurrently; retry

What it means

PhotonState.update() uses optimistic concurrency: it reads the row, applies the updater, then compareAndSet()s against the version it read. If CAS fails, it retries; after 32 failed attempts it gives up and throws this Error. It means another writer kept modifying the same state key faster than this caller could commit — the update was never applied.

Solutions

1. Catch the error and retry the whole update() call at the application level with backoff
2. Reduce contention: shard the hot key into per-worker keys and merge lazily, or narrow the update scope
3. Make the updater fast and pure (no awaits, no heavy work) to shrink the CAS race window
4. Serialize writes through a single queue/owner per key instead of many concurrent updaters

Example fix

// before: single call, unhandled contention
const v = await state.update("queue", enqueue(item));
// after: app-level retry with backoff
let v;
for (let i = 0; i < 5; i++) {
  try { v = await state.update("queue", enqueue(item)); break; }
  catch (e) {
    if (String(e.message).includes("changed concurrently")) {
      await sleep(50 * 2 ** i + Math.random() * 50);
      continue;
    }
    throw e;
  }
}

// none: contention is a runtime race, not a pre-checkable condition; minimize it instead
// keep updater synchronous and cheap — no awaits inside the update callback
const updater = (cur) => ({ count: (cur?.count ?? 0) + 1 }); // pure, fast => small CAS window

function isConcurrentStateError(e: unknown): boolean {
  return e instanceof Error && e.message === "Photon state changed concurrently; retry";
}

async function updateWithRetry<T>(state, key, updater, tries = 5) {
  for (let i = 0; ; i++) {
    try { return await state.update(key, updater); }
    catch (e) {
      if (isConcurrentStateError(e) && i < tries) {
        await sleep(2 ** i * 25 + Math.random() * 25);
        continue;
      }
      throw e;
    }
  }
}

When it happens

Trigger: Calling update() on a state key while many concurrent writers (other agents, heartbeats, parallel issue runs in the same company/endpoint scope) repeatedly mutate the same key, so all 32 CAS attempts observe a changed version.

Common situations: Hot keys written by every run in a company (counters, shared queues); several agents retrying the same key after an earlier failure storm; long updater work combined with frequent writers, so this writer always loses the race.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• codex_run_attach_busy
• Discord correction draft ownership changed
• opencode_run_attach_busy
• queued_comment_already_dispatching
• queued_comment_not_pending

AI-assisted analysis of paperclipai/paperclip@3f1d897a7c (2026-09-18). Data as JSON: /api/errors/8669e8b903779922. Report an issue: GitHub.

Appendix: source

Thrown at server/src/services/photon/state.ts:37

  }
  async update<T>(key: string, update: (current: T | null) => T): Promise<T> {
    for (let attempt = 0; attempt < 32; attempt++) {
      const row = await this.persistence.read(this.scope, this.key(key));
      const value = update(row ? (row.value as T) : null);
      if (Buffer.byteLength(JSON.stringify(value)) > 512 * 1024)
        throw new Error("Photon state record is too large");
      if (
        await this.persistence.compareAndSet({
          ...this.scope,
          key: this.key(key),
          expectedVersion: row?.version ?? null,
          expiresAt: null,
          value,
        })
      )
        return value;
    }
    throw new Error("Photon state changed concurrently; retry");
  }
}

View on GitHub (pinned to 3f1d897a7c)