provider_notification_window_exceeded

provider_notification_window_exceeded

Error message

provider_notification_window_exceeded: expected source sequence ${expectedSourceSeq}, received ${event.sourceSeq}

What it means

This helper filters an event batch down to events the consumer has not yet seen, based on monotonically increasing sourceSeq values. It requires the unseen events to form a contiguous sequence starting at lastSourceSeq + 1; if there is a gap or overlap (the provider's notification window was exceeded and events were dropped or reordered), it throws this error naming the expected and received sequence numbers.

Source

Thrown at packages/paperclip-runner/src/live/runnerd-codex-transport.ts:1302

    if (
      event.eventType !== "harness.ready" &&
      event.eventType !== "session.started" &&
      event.eventType !== "session.resumed"
    ) continue;
    return record(record(record(event.envelope).payload).payload);
  }
  return null;
}

export function unseenRunnerdCommittedEvents<
  T extends { sourceSeq: number },
>(events: readonly T[], lastSourceSeq: number): T[] {
  const unseen = events.filter((event) => event.sourceSeq > lastSourceSeq);
  if (unseen.length === 0) return [];
  let expectedSourceSeq = lastSourceSeq + 1;
  for (const event of unseen) {
    if (event.sourceSeq !== expectedSourceSeq) {
      throw new Error(
        `provider_notification_window_exceeded: expected source sequence ${expectedSourceSeq}, received ${event.sourceSeq}`,
      );
    }
    expectedSourceSeq += 1;
  }
  return unseen;
}

export function expandRunnerdCanonicalNotifications(
  method: string,
  input: unknown,
): Array<{ method: string; params: Record<string, unknown> }> {
  const payload = record(input);
  if (!Array.isArray(payload.events)) return [{ method, params: payload }];
  return payload.events.map((event) => ({ method, params: record(event) }));
}

export function runnerdCanonicalNotificationMethod(

View on GitHub (pinned to 01ad858492)

Solutions

Resync by fetching a full state snapshot from the provider and resetting lastSourceSeq, instead of replaying the gapped event stream
Reduce polling interval or enable a provider-side replay/catch-up mechanism so events are not dropped
Log the expected vs received sequence numbers and re-establish the session/window with the provider
If the provider restarted, re-baseline the sequence tracker from the provider's current sequence

Example fix

// before
const unseen = filterUnseen(events, lastSourceSeq); // throws on sequence gap
// after
let unseen: T[];
try {
  unseen = filterUnseen(events, lastSourceSeq);
} catch (err) {
  if (!(err as Error).message.startsWith("provider_notification_window_exceeded")) throw err;
  await resyncFromProviderSnapshot(); // full snapshot re-baselines lastSourceSeq
  unseen = [];
}

Defensive patterns

Strategy: try-catch

Validate before calling

const unseen = events.filter(e => e.sourceSeq > lastSourceSeq);
const contiguous = unseen.every((e, i) => e.sourceSeq === lastSourceSeq + 1 + i);
if (!contiguous) throw new Error("sequence gap detected; resync from snapshot before draining");

Type guard

const isContiguousWindow = <T extends { sourceSeq: number }>(events: readonly T[], last: number): boolean =>
  events.filter(e => e.sourceSeq > last)
    .every((e, i) => e.sourceSeq === last + 1 + i);

Try / catch

try {
  unseen = filterUnseenWindow(events, lastSourceSeq);
} catch (err) {
  if (!(err as Error).message.startsWith("provider_notification_window_exceeded")) throw err;
  await resyncFromProviderSnapshot();
  unseen = [];
}

Prevention

Poll frequently enough to stay inside the provider's notification retention window
Track and persist lastSourceSeq durably across restarts
On provider restart, re-baseline the sequence tracker from a full snapshot

When it happens

Trigger: Calling this filter with events where the first unseen event's sourceSeq is greater than lastSourceSeq + 1, or where consecutive unseen events skip a sequence number — indicating lost or dropped provider notifications between polls.

Common situations: Polling intervals too long so the provider's notification retention window expired and dropped events; a provider restart resetting or skipping sequence numbers; network interruption losing events; concurrent consumers draining the same window so events are consumed out of band.

Related errors

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/563b4eb75597b96a. Report an issue: GitHub.