observation generation job status transition was not applied

Error message

observation generation job status transition was not applied

What it means

PostgresObservationGenerationJobRepository.updateStatus() performs a guarded UPDATE whose WHERE clause encodes the legal state-machine transitions, then re-reads the row and calls assertValidJobStatusTransition() to distinguish failure causes. If the row exists and the requested transition is legal, yet the guarded UPDATE returned no row, the only remaining explanation is that the row changed between the UPDATE and the follow-up SELECT — this error signals a lost race against another worker processing the same job.

Solutions

Treat this as a benign lost race in the caller: re-read the job (getById/listByStatusForScope) and skip processing if another worker already moved it.
Ensure only one worker pool claims jobs per project/team, or stagger poll intervals so claim windows don't overlap.
If it fires constantly with a single worker, check for triggers or external writers mutating observation_generation_jobs, and for non-transactional reads on a hot job.
Wrap the whole claim+update in a single transaction so the guarded UPDATE's outcome is authoritative.

Example fix

// before
await repo.updateStatus({ id, projectId, teamId, status: 'processing', lockedBy: workerId });
// after
const applied = await repo.updateStatus({ id, projectId, teamId, status: 'processing', lockedBy: workerId });
if (!applied) {
  const fresh = await repo.getByIdForScope({ id, projectId, teamId });
  if (fresh && fresh.status === 'processing' && fresh.lockedBy !== workerId) {
    continue; // another worker won the claim race
  }
  throw new Error('job update failed for unknown reason');
}

Defensive patterns

Strategy: retry

Try / catch

try {
  const updated = await repo.updateStatus({ id, projectId, teamId, status, lockedBy });
  if (!updated) {
    const fresh = await repo.getByIdForScope({ id, projectId, teamId });
    if (fresh && fresh.lockedBy !== workerId) return; // lost claim race — another worker won
  }
} catch (err) {
  if (err instanceof Error && err.message === 'observation generation job status transition was not applied') {
    return; // benign concurrent-update race; re-read and move on
  }
  throw err;
}

Prevention

Treat updateStatus() returning null as the normal race outcome and always re-read the row.
Run one worker (or one claim window) per project/team to shrink race probability.
Claim and process jobs inside a single transaction so the guarded UPDATE is authoritative.

When it happens

Trigger: Two workers call updateStatus() on the same job near-simultaneously (e.g. both claim a queued job: the first flips it to processing; the second's guarded UPDATE matches nothing, but by the time it re-reads the row the state still looks legal to the validator). Also triggered when the attempts < max_attempts guard in SQL races with an attempt increment committed in between.

Common situations: Running multiple claude-mem workers against one Postgres database with overlapping claim intervals; a redeploy spinning up a new worker while the old one is draining; retries after network hiccups causing duplicate processing of the same job.

Related errors

AI-assisted analysis of thedotmack/claude-mem@e2d1df569a (2026-08-20). Data as JSON: /api/errors/5d7816521666a72b. Report an issue: GitHub.

Appendix: source

Thrown at src/storage/postgres/generation-jobs.ts:223

        input.lastError == null ? null : JSON.stringify(input.lastError),
        input.projectId,
        input.teamId
      ]
    );
    if (row) {
      return mapJobRow(row);
    }

    const current = await queryOne<JobRow>(
      this.client,
      'SELECT * FROM observation_generation_jobs WHERE id = $1 AND project_id = $2 AND team_id = $3',
      [input.id, input.projectId, input.teamId]
    );
    if (!current) {
      return null;
    }
    assertValidJobStatusTransition(mapJobRow(current), input.status);
    throw new Error('observation generation job status transition was not applied');
  }

  async listByStatusForScope(input: {
    status: ObservationGenerationJobStatus;
    projectId: string;
    teamId: string;
    limit?: number;
  }): Promise<PostgresObservationGenerationJob[]> {
    const result = await this.client.query<JobRow>(
      `
        SELECT * FROM observation_generation_jobs
        WHERE status = $1 AND project_id = $2 AND team_id = $3
        ORDER BY created_at ASC
        LIMIT $4
      `,
      [input.status, input.projectId, input.teamId, input.limit ?? 100]
    );
    return result.rows.map(mapJobRow);

View on GitHub (pinned to e2d1df569a)