{"record":{"id":"f368b4b3017876df","repo":"paperclipai/paperclip","slug":"pending-retry-store-full-bound-bound-evicted","errorCode":null,"errorMessage":"pending-retry store full (bound=${bound}); evicted oldest batch ${evicted?.batchId}; ${evicted?.events.length ?? 0} event(s) lost","messagePattern":"pending-retry store full \\(bound=(.+?)\\); evicted oldest batch (.+?); (.+?) event\\(s\\) lost","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"packages/shared/src/telemetry/client.ts","lineNumber":340,"sourceCode":"    return Math.max(0, Math.min(maxDelayMs, Math.round(base + jitter)));\n  }\n\n  /**\n   * Pushes a batch onto the pending store (bounded), then schedules its retry\n   * wake-up. On overflow the OLDEST batches (front) are evicted first — newest\n   * prioritized — and each eviction is logged so no batch is lost silently.\n   */\n  private enqueuePending(batch: PendingBatch): void {\n    this.pending.push(batch);\n    const bound = Math.max(0, this.caps.maxPendingRetryBatches);\n    while (this.pending.length > bound) {\n      const evicted = this.pending.shift();\n      // Cancel the evicted batch's scheduled retry so its timer doesn't keep the\n      // wake-up alive for a batch that is no longer in the store. Without this a\n      // flush that overflows the bound would strand one live timer per evicted\n      // batch even though `pending` itself stays bounded.\n      if (evicted) this.cancelRetryTimer(evicted);\n      this.warn(\n        `pending-retry store full (bound=${bound}); evicted oldest batch ${evicted?.batchId}; ${evicted?.events.length ?? 0} event(s) lost`,\n      );\n    }\n    // Only schedule a wake-up if this batch actually survived eviction. When the\n    // batch is immediately evicted by the bound (e.g. a large failed flush, or\n    // `maxPendingRetryBatches: 0`) it has no pending work, so scheduling a timer\n    // for it would strand thousands of no-op timers behind a small bound.\n    if (this.pending.includes(batch)) {\n      this.scheduleDrain(batch);\n    }\n  }\n\n  /** Cancels a pending batch's scheduled retry wake-up, if it has one. */\n  private cancelRetryTimer(batch: PendingBatch): void {\n    if (batch.timerId === undefined) return;\n    clearTimeout(batch.timerId);\n    this.retryTimers.delete(batch.timerId);\n    batch.timerId = undefined;","sourceCodeStart":322,"sourceCodeEnd":358,"githubUrl":"https://github.com/paperclipai/paperclip/blob/120ae5428fa29bee300bcf806491cd4d965fbb7c/packages/shared/src/telemetry/client.ts#L322-L358","documentation":"Failed flushes queue in an in-memory pending-retry store bounded by caps.maxPendingRetryBatches (default 20). When a re-queued batch pushes the store past the bound, the OLDEST pending batch is shifted out, its retry timer cancelled, and this eviction logged — FIFO loss under sustained collector failure. The store stays memory-bounded by design.","triggerScenarios":"An outage long enough that pending batches pile past the bound; a single large failed flush enqueueing many batches at once; or maxPendingRetryBatches deliberately set to 0 (every failed batch evicted immediately).","commonSituations":"Collector down while agents emit heavily; caps tuned small for memory; bound=0 chosen to make telemetry strictly fire-and-forget but drop warns still alerting.","solutions":["Restore collector availability so the pending queue drains before the bound is hit.","Raise caps.maxPendingRetryBatches to buffer longer outages (trade memory: batches are held in-process).","Reduce telemetry volume (sampling, fewer events, larger maxEventsPerBatch) if the bound must stay small.","If bound=0 is intentional, mark this warn as expected in alerting instead of paging on it."],"exampleFix":"// before\ntelemetry: { maxPendingRetryBatches: 5 }\n\n// after\ntelemetry: { maxPendingRetryBatches: 50 }","handlingStrategy":"fallback","validationCode":"// Size the bound at config time: bound >= peak events/sec * max outage sec / batch size.\nconst expectedOutageSec = 600;\nconst peakEventsPerSec = 50;\nconst eventsPerBatch = 100;\nconst needed = Math.ceil((peakEventsPerSec * expectedOutageSec) / eventsPerBatch);\nif (telemetryCaps.maxPendingRetryBatches < needed) {\n  logger.warn(`pending bound ${telemetryCaps.maxPendingRetryBatches} under-sized for ${needed} batches`);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat telemetry as best-effort; keep business-critical data out of it entirely.","Alert on the FIRST eviction warn — every subsequent one is compounding data loss.","Raise maxPendingRetryBatches on high-traffic hosts (mind memory; batches are in-process).","If running with bound 0 deliberately, mark this warn as expected noise in alerting."],"tags":["telemetry","queue-overflow","eviction","backpressure"],"backgroundTag":"retry-queue-overflow","analyzedSha":"120ae5428fa29bee300bcf806491cd4d965fbb7c","analyzedAt":"2026-08-18T22:49:45.177Z","schemaVersion":2},"datasetVersion":"2026-08-22T09:17:25.309Z"}