vectordotdev/vector · error

Zstd encoder should not fail on init.

Error message

Zstd encoder should not fail on init.

What it means

This is a Rust panic assertion (`.expect`) on `ZstdEncoder::new(writer, level)`, which returns a Result only because zstd allocation/compression-parameter validation can fail in theory. Vector wraps it in an expect because with an in-memory `BytesMut` writer and a validated `Compression` level, initialization is treated as an infallible internal invariant. If this panic fires, the zstd crate rejected the encoder creation itself (e.g. allocation failure or an out-of-range compression level), meaning a broken invariant rather than a runtime condition the caller can handle.

Solutions

  1. Check the sink's compression configuration and set zstd level to a valid range (1-22, typically 3 as default).
  2. Reduce memory pressure in the process/container (raise memory limits) since the failure mode is allocation failure inside zstd.
  3. If the panic reproduces reliably, verify the zstd crate version in Cargo.lock for regressions and pin/update it.
  4. Switch compression to gzip/zlib/snappy as a workaround while investigating.

Example fix

// before (config vector.yaml)
compression: zstd
compression_level: 100
// after
compression: zstd
compression_level: 3
Defensive patterns

Strategy: validation

Validate before calling

// Validate zstd compression level before configuring the sink
fn validate_zstd_level(level: i32) -> Result<(), String> {
    if (1..=22).contains(&level) {
        Ok(())
    } else {
        Err(format!("zstd compression_level must be 1-22, got {level}"))
    }
}

Try / catch

// Rust panics are not catchable via Result; abort the task.
// Use std::panic::catch_unwind only at an isolation boundary:
let result = std::panic::catch_unwind(|| buffer_with_zstd_config());
match result {
    Ok(inner) => inner,
    Err(_) => restart_sink_task_with_default_compression(),
}

Prevention

When it happens

Trigger: Calling `BatchSerializerBuffer::buffer()`/`push()` with `compression = Compression::Zstd(level)` when `ZstdEncoder::new` returns Err — practically only on memory allocation failure or an invalid zstd compression level being passed into `level.into()`.

Common situations: Configuring a sink with `compression = "zstd"` and an extreme/unsupported compression level (e.g. level above zstd's max of 22 or a negative level after conversion); running under severe memory pressure where the zstd internal allocation fails; using a custom-built zstd build with restricted limits.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of vectordotdev/vector@bdb87aeaa4 (2026-09-16). Data as JSON: /api/errors/48ac926e01898186. Report an issue: GitHub.

Appendix: source

Thrown at src/sinks/util/buffer/mod.rs:65

        }
    }

    fn buffer(&mut self) -> &mut InnerBuffer {
        let bytes = self.settings.bytes;
        let compression = self.compression;
        self.inner.get_or_insert_with(|| {
            let writer = BytesMut::with_capacity(bytes).writer();
            match compression {
                Compression::None => InnerBuffer::Plain(writer),
                Compression::Gzip(level) => {
                    InnerBuffer::Gzip(GzEncoder::new(writer, level.as_flate2()))
                }
                Compression::Zlib(level) => {
                    InnerBuffer::Zlib(ZlibEncoder::new(writer, level.as_flate2()))
                }
                Compression::Zstd(level) => InnerBuffer::Zstd(
                    ZstdEncoder::new(writer, level.into())
                        .expect("Zstd encoder should not fail on init."),
                ),
                Compression::Snappy => InnerBuffer::Snappy(SnappyEncoder::new(writer)),
            }
        })
    }

    pub fn push(&mut self, input: &[u8]) {
        self.num_items += 1;
        match self.buffer() {
            InnerBuffer::Plain(inner) => {
                inner.write_all(input).unwrap();
            }
            InnerBuffer::Gzip(inner) => {
                inner.write_all(input).unwrap();
            }
            InnerBuffer::Zlib(inner) => {
                inner.write_all(input).unwrap();
            }

View on GitHub (pinned to bdb87aeaa4)