{"record":{"id":"5d7816521666a72b","repo":"thedotmack/claude-mem","slug":"observation-generation-job-status-transition-was-n","errorCode":null,"errorMessage":"observation generation job status transition was not applied","messagePattern":"observation generation job status transition was not applied","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/storage/postgres/generation-jobs.ts","lineNumber":223,"sourceCode":"        input.lastError == null ? null : JSON.stringify(input.lastError),\n        input.projectId,\n        input.teamId\n      ]\n    );\n    if (row) {\n      return mapJobRow(row);\n    }\n\n    const current = await queryOne<JobRow>(\n      this.client,\n      'SELECT * FROM observation_generation_jobs WHERE id = $1 AND project_id = $2 AND team_id = $3',\n      [input.id, input.projectId, input.teamId]\n    );\n    if (!current) {\n      return null;\n    }\n    assertValidJobStatusTransition(mapJobRow(current), input.status);\n    throw new Error('observation generation job status transition was not applied');\n  }\n\n  async listByStatusForScope(input: {\n    status: ObservationGenerationJobStatus;\n    projectId: string;\n    teamId: string;\n    limit?: number;\n  }): Promise<PostgresObservationGenerationJob[]> {\n    const result = await this.client.query<JobRow>(\n      `\n        SELECT * FROM observation_generation_jobs\n        WHERE status = $1 AND project_id = $2 AND team_id = $3\n        ORDER BY created_at ASC\n        LIMIT $4\n      `,\n      [input.status, input.projectId, input.teamId, input.limit ?? 100]\n    );\n    return result.rows.map(mapJobRow);","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/thedotmack/claude-mem/blob/e2d1df569a8f04075d40e92461128ece7cf04c82/src/storage/postgres/generation-jobs.ts#L205-L241","documentation":"PostgresObservationGenerationJobRepository.updateStatus() performs a guarded UPDATE whose WHERE clause encodes the legal state-machine transitions, then re-reads the row and calls assertValidJobStatusTransition() to distinguish failure causes. If the row exists and the requested transition is legal, yet the guarded UPDATE returned no row, the only remaining explanation is that the row changed between the UPDATE and the follow-up SELECT — this error signals a lost race against another worker processing the same job.","triggerScenarios":"Two workers call updateStatus() on the same job near-simultaneously (e.g. both claim a queued job: the first flips it to processing; the second's guarded UPDATE matches nothing, but by the time it re-reads the row the state still looks legal to the validator). Also triggered when the attempts < max_attempts guard in SQL races with an attempt increment committed in between.","commonSituations":"Running multiple claude-mem workers against one Postgres database with overlapping claim intervals; a redeploy spinning up a new worker while the old one is draining; retries after network hiccups causing duplicate processing of the same job.","solutions":["Treat this as a benign lost race in the caller: re-read the job (getById/listByStatusForScope) and skip processing if another worker already moved it.","Ensure only one worker pool claims jobs per project/team, or stagger poll intervals so claim windows don't overlap.","If it fires constantly with a single worker, check for triggers or external writers mutating observation_generation_jobs, and for non-transactional reads on a hot job.","Wrap the whole claim+update in a single transaction so the guarded UPDATE's outcome is authoritative."],"exampleFix":"// before\nawait repo.updateStatus({ id, projectId, teamId, status: 'processing', lockedBy: workerId });\n// after\nconst applied = await repo.updateStatus({ id, projectId, teamId, status: 'processing', lockedBy: workerId });\nif (!applied) {\n  const fresh = await repo.getByIdForScope({ id, projectId, teamId });\n  if (fresh && fresh.status === 'processing' && fresh.lockedBy !== workerId) {\n    continue; // another worker won the claim race\n  }\n  throw new Error('job update failed for unknown reason');\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  const updated = await repo.updateStatus({ id, projectId, teamId, status, lockedBy });\n  if (!updated) {\n    const fresh = await repo.getByIdForScope({ id, projectId, teamId });\n    if (fresh && fresh.lockedBy !== workerId) return; // lost claim race — another worker won\n  }\n} catch (err) {\n  if (err instanceof Error && err.message === 'observation generation job status transition was not applied') {\n    return; // benign concurrent-update race; re-read and move on\n  }\n  throw err;\n}","preventionTips":["Treat updateStatus() returning null as the normal race outcome and always re-read the row.","Run one worker (or one claim window) per project/team to shrink race probability.","Claim and process jobs inside a single transaction so the guarded UPDATE is authoritative."],"tags":["postgres","race-condition","job-queue","concurrency","state-machine"],"backgroundTag":"optimistic-locking-conflict","analyzedSha":"e2d1df569a8f04075d40e92461128ece7cf04c82","analyzedAt":"2026-08-20T23:58:13.836Z","contentChangedAt":"2026-08-20T23:58:13.836Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}