What causes "zstd writer should not fail to finish" in vectordotdev/vector?

Increase memory available to the Vector process/container.

zstd writer should not fail to finish

Error message

zstd writer should not fail to finish

What it means

A panic assertion on `ZstdEncoder::finish()` in `Writer::into_inner` (compressor.rs). Finishing a zstd encoder writes the final frame into an in-memory `BytesMut` writer, which cannot fail, so Vector treats it as infallible. A panic here indicates the finalization write failed (e.g. allocation failure) — an internal invariant violation rather than a condition callers can handle.

Solutions

Increase memory available to the Vector process/container.
Temporarily switch sink compression to a different codec while investigating.
Report the panic to Vector maintainers as an invariant violation if it recurs.

Defensive patterns

Strategy: try-catch

Try / catch

let buf = writer.finish()?; // fallible variant returns io::Result<BytesMut>
match buf {
    Ok(bytes) => send(bytes),
    Err(e) => log::warn!("zstd finish failed: {e}"),
}

Prevention

Use the fallible `Writer::finish()` API instead of the asserting `into_inner()`.
Keep zstd compression levels moderate to bound finalization memory use.
Provision memory headroom proportional to batch.max_bytes for zstd sinks.

When it happens

Trigger: Calling `Writer::into_inner()` with `Compression::Zstd` when `ZstdEncoder::finish` returns Err writing the terminating bytes of the zstd frame.

Common situations: Rare; only plausible under severe memory exhaustion while finalizing a zstd-compressed payload before sending it to a sink.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of vectordotdev/vector@bdb87aeaa4 (2026-09-16). Data as JSON: /api/errors/c078e8ef2e2387f0. Report an issue: GitHub.

Appendix: source

Thrown at src/sinks/util/compressor.rs:47

        }
    }

    pub fn into_inner(self) -> BytesMut {
        match self {
            Writer::Plain(writer) => writer,
            Writer::Gzip(writer) => writer
                .into_inner()
                .expect("BufWriter writer should not fail to finish")
                .finish()
                .expect("gzip writer should not fail to finish"),
            Writer::Zlib(writer) => writer
                .into_inner()
                .expect("BufWriter writer should not fail to finish")
                .finish()
                .expect("zlib writer should not fail to finish"),
            Writer::Zstd(writer) => writer
                .finish()
                .expect("zstd writer should not fail to finish"),
            Writer::Snappy(writer) => writer
                .finish()
                .expect("snappy writer should not fail to finish"),
        }
        .into_inner()
    }

    pub fn finish(self) -> io::Result<BytesMut> {
        let buf = match self {
            Writer::Plain(writer) => writer,
            Writer::Gzip(writer) => writer.into_inner()?.finish()?,
            Writer::Zlib(writer) => writer.into_inner()?.finish()?,
            Writer::Zstd(writer) => writer.finish()?,
            Writer::Snappy(writer) => writer.finish()?,
        }
        .into_inner();

        Ok(buf)

View on GitHub (pinned to bdb87aeaa4)