vectordotdev/vector · error

Out of memory

Error message

Out of memory

What it means

The Prometheus remote_write request builder encodes a protobuf TimeSeries into a BytesMut with prost's encode() and unwraps with expect("Out of memory"). Per prost, encode only fails when the underlying buffer cannot grow (allocation failure), so the message asserts memory exhaustion, not malformed metrics. Metrics were already validated by encode_defaults above.

Solutions

1. Reduce batch size (batch.max_events / batch_timeout) in the prometheus remote_write sink config to shrink each encoded payload.
2. Increase the container/host memory limit available to Vector.
3. Lower metric cardinality before the sink (drop/relabel high-cardinality series via VRL or the source).
4. Check for memory leaks or contention in the process (RSS near the limit) and upgrade Vector.

Example fix

// before (vector.yaml)
sinks:
  prometheus_remote:
    batch:
      max_events: 100000

// after
sinks:
  prometheus_remote:
    batch:
      max_events: 1000

// vector.yaml — cap batch size so encoded protobuf payloads stay small
sinks:
  prometheus_remote:
    batch:
      max_events: 1000
      timeout_secs: 1

When it happens

Trigger: prost encode() fails during remote_write request finalization when BytesMut cannot reserve encoded_len bytes — i.e., the process cannot allocate the needed memory for a large batch of time series.

Common situations: Memory-constrained containers hitting cgroup limits with very large metric batches; extremely high-cardinality Prometheus metrics producing huge remote_write payloads; OOM-adjacent conditions.

Related errors

• error_tps cannot be Nan
• invalid timestamp
• target_tps cannot be Nan
• a record with a next ID must have an event count
• a valid HTTP/1 URI is valid as an HTTP URI

AI-assisted analysis of vectordotdev/vector@bdb87aeaa4 (2026-09-16). Data as JSON: /api/errors/cc19b25775f8e1af. Report an issue: GitHub.

Appendix: source

Thrown at src/sinks/prometheus/remote_write/request_builder.rs:42

    ) -> io::Result<(usize, GroupedCountByteSize)> {
        let mut byte_size = telemetry().create_request_count_byte_size();

        let mut time_series = collector::TimeSeries::new();
        let len = input.len();
        for metric in input {
            byte_size.add_event(&metric, metric.estimated_json_encoded_size_of());

            time_series.encode_metric(
                self.default_namespace.as_deref(),
                &self.buckets,
                &self.quantiles,
                &metric,
            );
        }
        let request = time_series.finish();

        let mut out = BytesMut::with_capacity(request.encoded_len());
        request.encode(&mut out).expect("Out of memory");
        let body = out.freeze();

        write_all(writer, len, body.as_ref())?;

        Ok((body.len(), byte_size))
    }
}

#[derive(Clone)]
pub(super) struct RemoteWriteRequest {
    pub(super) request: Bytes,
    pub(super) tenant_id: Option<String>,
    finalizers: EventFinalizers,
    metadata: RequestMetadata,
}

impl Finalizable for RemoteWriteRequest {
    fn take_finalizers(&mut self) -> EventFinalizers {

View on GitHub (pinned to bdb87aeaa4)