thanos-io/thanos · error

writing streamed snappy header

Error message

writing streamed snappy header

What it means

diffVarintSnappyStreamedEncode fails when writing the streamed-snappy magic header into the output buffer errors. bytes.Buffer.WriteString only errors on allocation failure, so this almost always indicates memory exhaustion.

Solutions

1. Increase available memory / container memory limits
2. Reduce cache encoding concurrency
3. Check cgroup OOM events around the failure

if debug.FreeOSMemory != nil && runtime.MemStats{} /* check headroom */ {
    var ms runtime.MemStats; runtime.ReadMemStats(&ms)
    if ms.Sys > ms.Sys/10*9 { return errors.New("insufficient memory headroom for encoding") }
}

b, err := diffVarintSnappyStreamedEncode(p, length)
if err != nil {
    if strings.Contains(err.Error(), "writing streamed snappy header") {
        return retryWithBackoff(func() ([]byte, error) { return diffVarintSnappyStreamedEncode(p, length) })
    }
    return nil, err
}

When it happens

Trigger: Calling encodePostingsToCache -> diffVarintSnappyStreamedEncode on a system where bytes.NewBuffer cannot allocate/append due to OOM conditions.

Common situations: Very large postings sets during cache encoding combined with memory pressure; container memory limits being hit.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• short-write streamed snappy header
• creating snappy compressor
• writing uvarint encoded byte
• short-write for uvarint encoded byte
• closing snappy stream writer

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5ea87604a0f7b17e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/postings_codec.go:96

	const maxBlockSize = 65536

	length = 5 * length / 4 // estimate 1.25B per posting.

	blocks := length / maxBlockSize

	ret += blocks * snappy.MaxEncodedLen(maxBlockSize)
	length -= blocks * maxBlockSize
	if length > 0 {
		ret += snappy.MaxEncodedLen(length)
	}

	return ret
}

func diffVarintSnappyStreamedEncode(p index.Postings, length int) ([]byte, error) {
	compressedBuf := bytes.NewBuffer(make([]byte, 0, estimateSnappyStreamSize(length)))
	if n, err := compressedBuf.WriteString(codecHeaderStreamedSnappy); err != nil {
		return nil, fmt.Errorf("writing streamed snappy header")
	} else if n != len(codecHeaderStreamedSnappy) {
		return nil, fmt.Errorf("short-write streamed snappy header")
	}

	uvarintEncodeBuf := make([]byte, binary.MaxVarintLen64)

	sw, err := extsnappy.Compressor.Compress(compressedBuf)
	if err != nil {
		return nil, fmt.Errorf("creating snappy compressor: %w", err)
	}

	prev := storage.SeriesRef(0)
	for p.Next() {
		v := p.At()
		if v < prev {
			return nil, errors.Errorf("postings entries must be in increasing order, current: %d, previous: %d", v, prev)
		}

View on GitHub (pinned to 35b8b99117)