{"record":{"id":"57da44f1a3683c5a","repo":"thedotmack/claude-mem","slug":"failed-to-re-enqueue-generation-job-on-operator-re","errorCode":null,"errorMessage":"failed to re-enqueue generation job on operator retry","messagePattern":"failed to re-enqueue generation job on operator retry","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/server/routes/v1/ServerV1PostgresRoutes.ts","lineNumber":1567,"sourceCode":"        requestId: req.requestId ?? null,\n        retriedCount,\n      },\n    });\n\n    // Re-enqueue to BullMQ. If the queue is unavailable we leave the row in\n    // queued state and reconciliation will publish it on next startup —\n    // never lie about \"enqueued\" when we couldn't publish.\n    const queue = this.resolveEventQueueForRetry(updatedRow as { source_type: string });\n    if (queue && updatedRow) {\n      try {\n        const bullmqJobId = (updatedRow as { bullmq_job_id: string | null }).bullmq_job_id;\n        if (bullmqJobId) {\n          // Best effort remove first so a terminal-state slot doesn't block.\n          try { await queue.remove(bullmqJobId); } catch { /* terminal slot may be missing — ok */ }\n          await queue.add(bullmqJobId, retryBullmqPayload as never);\n        }\n      } catch (error) {\n        logger.warn('SYSTEM', 'failed to re-enqueue generation job on operator retry', {\n          jobId: id,\n          requestId: req.requestId ?? null,\n          error: error instanceof Error ? error.message : String(error),\n        });\n      }\n    }\n\n    const refreshed = await repo.getByIdForScope({ id, projectId: current.projectId, teamId });\n    if (!refreshed) {\n      res.status(404).json({ error: 'NotFound', message: 'Generation job not found' });\n      return null;\n    }\n\n    await this.auditWrite(req, 'generation_job.retried_by_operator', refreshed.id, refreshed.projectId, {\n      previousStatus: current.status,\n      currentStatus: refreshed.status,\n      retriedCount,\n      requestId: req.requestId ?? null,","sourceCodeStart":1549,"sourceCodeEnd":1585,"githubUrl":"https://github.com/thedotmack/claude-mem/blob/e2d1df569a8f04075d40e92461128ece7cf04c82/src/server/routes/v1/ServerV1PostgresRoutes.ts#L1549-L1585","documentation":"The operator retry endpoint updates the job row to queued and then re-publishes it to the BullMQ event queue (a best-effort remove of the old bullmq_job_id, then queue.add with the retry payload). If publishing throws, this warning logs and the row is left queued-but-unpublished: by design the API never claims 'enqueued' when it could not publish, and startup reconciliation publishes such rows on next boot.","triggerScenarios":"POSTing an operator retry for a generation job while Redis is down or unreachable (BullMQ add fails), the REDIS_URL or queue name changed between deploys, or the retry payload cannot be serialized for the queue.","commonSituations":"Redis restart or failover mid-retry; queue prefix or name drift after a config change; Redis maxmemory pressure; network policy blocking the Redis port from new pods.","solutions":["Verify Redis from the server host: redis-cli -u \"$REDIS_URL\" ping","Read the warn's error field: ioredis connection errors point at Redis, serialization errors point at the retry payload shape","Restart the API: startup reconciliation publishes queued-but-unpublished rows, completing the retry","Align queue names, prefixes, and REDIS_URL with the deployment if they drifted"],"exampleFix":"// before\nawait queue.add(bullmqJobId, retryBullmqPayload as never);\n\n// after: retry the publish, then rely on startup reconciliation if it still fails\nlet published = false;\nfor (let attempt = 1; attempt <= 3 && !published; attempt++) {\n  try {\n    await queue.add(bullmqJobId, retryBullmqPayload as never);\n    published = true;\n  } catch (error) {\n    if (attempt === 3) {\n      logger.warn('SYSTEM', 'failed to re-enqueue generation job on operator retry', {\n        jobId: id,\n        error: error instanceof Error ? error.message : String(error),\n      });\n    } else {\n      await new Promise((resolve) => setTimeout(resolve, 100 * attempt));\n    }\n  }\n}","handlingStrategy":"retry","validationCode":"// Before triggering operator retries, confirm the queue backend is reachable:\nawait queue.client.ping(); // the ioredis client BullMQ uses\n// If this throws, fix Redis first: retries would only queue rows, not jobs.","typeGuard":"function isRedisConnectionError(error: unknown): boolean {\n  const name = (error as { name?: string } | null)?.name;\n  const message = error instanceof Error ? error.message : String(error);\n  return name === 'MaxRetriesPerRequestError' || /connection|ECONNREFUSED|ETIMEDOUT/i.test(message);\n}","tryCatchPattern":"try {\n  if (bullmqJobId) {\n    try { await queue.remove(bullmqJobId); } catch { /* terminal slot may be missing */ }\n    await queue.add(bullmqJobId, retryBullmqPayload as never);\n  }\n} catch (error) {\n  if (isRedisConnectionError(error)) {\n    // leave the row queued; startup reconciliation re-publishes it — do not report 'enqueued'\n  }\n  logger.warn('SYSTEM', 'failed to re-enqueue generation job on operator retry', {\n    jobId: id, requestId: req.requestId ?? null, error: error instanceof Error ? error.message : String(error),\n  });\n}","preventionTips":["Add a Redis health check to the API readiness probe so retries are rejected while the queue is down","Pin queue names and prefixes in shared config so producer and consumer cannot drift","Alert on queued-but-unpublished rows to see reconciliation lag before users do","Version BullMQ job payloads backward-compatibly so serialization never fails on old workers"],"tags":["bullmq","redis","job-retry","reconciliation"],"backgroundTag":"job-enqueue-failed","analyzedSha":"e2d1df569a8f04075d40e92461128ece7cf04c82","analyzedAt":"2026-08-20T23:58:13.836Z","contentChangedAt":"2026-08-20T23:58:13.836Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}