vectordotdev/vector · error · ChunkedGelfDecoderError

Pending messages limit of {pending_messages_limit} reached w

Error message

Pending messages limit of {pending_messages_limit} reached while processing chunk with message id {message_id} and sequence number {sequence_number}

What it means

The chunked GELF decoder keeps one reassembly state per in-flight message id and bounds that map with the optional `pending_messages_limit` framing option. When a new chunk arrives while the map already holds `pending_messages_limit` entries, `PendingMessagesLimitReachedSnafu` rejects it — a memory/DoS guard so unfinished messages cannot grow the buffer unboundedly. States expire on their own after `timeout_secs` (default from `ChunkedGelfDecoderConfig`).

Source

Thrown at lib/codecs/src/decoding/framing/chunked_gelf.rs:378

        ensure!(
            sequence_number < total_chunks,
            InvalidSequenceNumberSnafu {
                message_id,
                sequence_number,
                total_chunks
            }
        );

        let mut state_lock = self.state.lock().expect("poisoned lock");

        // Only a new message grows the table, so the limit applies on insert. Checking it
        // before the lookup rejected chunks of messages already pending, which could then
        // never complete and expired instead.
        if !state_lock.contains_key(&message_id)
            && let Some(pending_messages_limit) = self.pending_messages_limit
        {
            ensure!(
                state_lock.len() < pending_messages_limit,
                PendingMessagesLimitReachedSnafu {
                    message_id,
                    sequence_number,
                    pending_messages_limit
                }
            );
        }

        let message_state = state_lock.entry(message_id).or_insert_with(|| {
            // We need to spawn a task that will clear the message state after a certain time
            // otherwise we will have a memory leak due to messages that never complete
            let state = Arc::clone(&self.state);
            let timeout = self.timeout;
            let timeout_handle = tokio::spawn(async move {
                tokio::time::sleep(timeout).await;
                let mut state_lock = state.lock().expect("poisoned lock");
                if state_lock.remove(&message_id).is_some() {

View on GitHub (pinned to 99894c8d88)

Solutions

Raise `pending_messages_limit` to comfortably above peak concurrent in-flight messages (it is only a safety cap, sized to `rate × timeout_secs`).
Reduce `timeout_secs` (default 5s) so abandoned message states are evicted sooner.
Fix the underlying loss/scattering: increase sender-side chunk reliability, tune NIC/UDP buffers, or ensure affinity so all chunks of a message hit the same Vector instance.
Check whether a hostile or misbehaving sender is spraying unique message ids and rate-limit it at the network edge.

Example fix

# before
decoding:
  codec: gelf
  framing:
    method: chunked_gelf
    chunked_gelf:
      pending_messages_limit: 1000

# after
decoding:
  codec: gelf
  framing:
    method: chunked_gelf
    chunked_gelf:
      timeout_secs: 2
      pending_messages_limit: 50000

Defensive patterns

Strategy: retry

Validate before calling

# Size the limit before deploy:
# pending ≈ messages_per_sec × timeout_secs × (loss factor)
# e.g. 5k msg/s × 5s default timeout × 2 headroom → 50_000
framing:
  method: chunked_gelf
  chunked_gelf:
    timeout_secs: 5
    pending_messages_limit: 50000

Try / catch

Err(e) if e.to_string().contains("Pending messages limit") => {
    // transient overload: drop the chunk, backpressure the source metrics
    warn!(error = %e, "GELF reassembly at capacity; increasing timeout/limit advised");
}

Prevention

Set pending_messages_limit ≈ 2 × peak rate × timeout_secs rather than a round small number.
Keep timeout_secs as low as your senders' worst retransmit window.
Monitor UDP receive errors and Vector internal metrics to catch loss before the limit trips.

When it happens

Trigger: Setting `framing.chunked_gelf.pending_messages_limit` (e.g. via GELF/UDP source codec config) to N and then having N message ids pending reassembly — typically heavy chunk loss, a `timeout_secs` longer than the loss recovery window, or a flood of unique message ids (malicious or buggy sender) — while one more chunk arrives.

Common situations: Packet loss on high-volume GELF UDP traffic leaving half-assembled messages pinned until timeout; NAT/load-balancer hash changes scattering chunks across Vector replicas so each sees partial messages; raising the limit too low after a traffic increase.

Related errors

AI-assisted analysis of vectordotdev/vector@99894c8d88 (2026-08-25). Data as JSON: /api/errors/b9e4c6d5d3537af0. Report an issue: GitHub.