pending-retry store full (bound=${bound}); evicted oldest ba

Error message

pending-retry store full (bound=${bound}); evicted oldest batch ${evicted?.batchId}; ${evicted?.events.length ?? 0} event(s) lost

What it means

Failed flushes queue in an in-memory pending-retry store bounded by caps.maxPendingRetryBatches (default 20). When a re-queued batch pushes the store past the bound, the OLDEST pending batch is shifted out, its retry timer cancelled, and this eviction logged — FIFO loss under sustained collector failure. The store stays memory-bounded by design.

Source

Thrown at packages/shared/src/telemetry/client.ts:340

    return Math.max(0, Math.min(maxDelayMs, Math.round(base + jitter)));
  }

  /**
   * Pushes a batch onto the pending store (bounded), then schedules its retry
   * wake-up. On overflow the OLDEST batches (front) are evicted first — newest
   * prioritized — and each eviction is logged so no batch is lost silently.
   */
  private enqueuePending(batch: PendingBatch): void {
    this.pending.push(batch);
    const bound = Math.max(0, this.caps.maxPendingRetryBatches);
    while (this.pending.length > bound) {
      const evicted = this.pending.shift();
      // Cancel the evicted batch's scheduled retry so its timer doesn't keep the
      // wake-up alive for a batch that is no longer in the store. Without this a
      // flush that overflows the bound would strand one live timer per evicted
      // batch even though `pending` itself stays bounded.
      if (evicted) this.cancelRetryTimer(evicted);
      this.warn(
        `pending-retry store full (bound=${bound}); evicted oldest batch ${evicted?.batchId}; ${evicted?.events.length ?? 0} event(s) lost`,
      );
    }
    // Only schedule a wake-up if this batch actually survived eviction. When the
    // batch is immediately evicted by the bound (e.g. a large failed flush, or
    // `maxPendingRetryBatches: 0`) it has no pending work, so scheduling a timer
    // for it would strand thousands of no-op timers behind a small bound.
    if (this.pending.includes(batch)) {
      this.scheduleDrain(batch);
    }
  }

  /** Cancels a pending batch's scheduled retry wake-up, if it has one. */
  private cancelRetryTimer(batch: PendingBatch): void {
    if (batch.timerId === undefined) return;
    clearTimeout(batch.timerId);
    this.retryTimers.delete(batch.timerId);
    batch.timerId = undefined;

View on GitHub (pinned to 120ae5428f)

Solutions

Restore collector availability so the pending queue drains before the bound is hit.
Raise caps.maxPendingRetryBatches to buffer longer outages (trade memory: batches are held in-process).
Reduce telemetry volume (sampling, fewer events, larger maxEventsPerBatch) if the bound must stay small.
If bound=0 is intentional, mark this warn as expected in alerting instead of paging on it.

Example fix

// before
telemetry: { maxPendingRetryBatches: 5 }

// after
telemetry: { maxPendingRetryBatches: 50 }

Defensive patterns

Strategy: fallback

Validate before calling

// Size the bound at config time: bound >= peak events/sec * max outage sec / batch size.
const expectedOutageSec = 600;
const peakEventsPerSec = 50;
const eventsPerBatch = 100;
const needed = Math.ceil((peakEventsPerSec * expectedOutageSec) / eventsPerBatch);
if (telemetryCaps.maxPendingRetryBatches < needed) {
  logger.warn(`pending bound ${telemetryCaps.maxPendingRetryBatches} under-sized for ${needed} batches`);
}

Prevention

Treat telemetry as best-effort; keep business-critical data out of it entirely.
Alert on the FIRST eviction warn — every subsequent one is compounding data loss.
Raise maxPendingRetryBatches on high-traffic hosts (mind memory; batches are in-process).
If running with bound 0 deliberately, mark this warn as expected noise in alerting.

When it happens

Trigger: An outage long enough that pending batches pile past the bound; a single large failed flush enqueueing many batches at once; or maxPendingRetryBatches deliberately set to 0 (every failed batch evicted immediately).

Common situations: Collector down while agents emit heavily; caps tuned small for memory; bound=0 chosen to make telemetry strictly fire-and-forget but drop warns still alerting.

Related errors

AI-assisted analysis of paperclipai/paperclip@120ae5428f (2026-08-18). Data as JSON: /api/errors/f368b4b3017876df. Report an issue: GitHub.