thanos-io/thanos · warning

short-write streamed snappy header

Error message

short-write streamed snappy header

What it means

The streamed snappy encoder verifies the header write reported the full length of codecHeaderStreamedSnappy. A short write means the output buffer did not accept all header bytes, which bytes.Buffer should never do under normal conditions — it signals a broken runtime/invariant.

Solutions

1. Report/investigate the runtime condition; stock bytes.Buffer cannot short-write
2. Retry cache encoding after freeing resources

// pre-allocate adequate capacity so bytes.Buffer writes cannot degrade
buf := bytes.NewBuffer(make([]byte, 0, estimateSnappyStreamSize(length)+len(codecHeaderStreamedSnappy)))

b, err := diffVarintSnappyStreamedEncode(p, length)
if err != nil {
    if strings.Contains(err.Error(), "short-write streamed snappy header") {
        return nil, fmt.Errorf("encoder invariant breached: %w", err)
    }
    return nil, err
}

When it happens

Trigger: encodePostingsToCache -> diffVarintSnappyStreamedEncode when WriteString returns n != len(codecHeaderStreamedSnappy); practically only under abnormal buffer/runtime states.

Common situations: Buggy custom buffer implementations or exotic memory conditions; virtually never seen in stock Prometheus.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• short-write for uvarint encoded byte
• writing streamed snappy header
• creating snappy compressor
• writing uvarint encoded byte
• closing snappy stream writer

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/db052e48b888b6cc. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/postings_codec.go:98

	length = 5 * length / 4 // estimate 1.25B per posting.

	blocks := length / maxBlockSize

	ret += blocks * snappy.MaxEncodedLen(maxBlockSize)
	length -= blocks * maxBlockSize
	if length > 0 {
		ret += snappy.MaxEncodedLen(length)
	}

	return ret
}

func diffVarintSnappyStreamedEncode(p index.Postings, length int) ([]byte, error) {
	compressedBuf := bytes.NewBuffer(make([]byte, 0, estimateSnappyStreamSize(length)))
	if n, err := compressedBuf.WriteString(codecHeaderStreamedSnappy); err != nil {
		return nil, fmt.Errorf("writing streamed snappy header")
	} else if n != len(codecHeaderStreamedSnappy) {
		return nil, fmt.Errorf("short-write streamed snappy header")
	}

	uvarintEncodeBuf := make([]byte, binary.MaxVarintLen64)

	sw, err := extsnappy.Compressor.Compress(compressedBuf)
	if err != nil {
		return nil, fmt.Errorf("creating snappy compressor: %w", err)
	}

	prev := storage.SeriesRef(0)
	for p.Next() {
		v := p.At()
		if v < prev {
			return nil, errors.Errorf("postings entries must be in increasing order, current: %d, previous: %d", v, prev)
		}

		uvarintSize := binary.PutUvarint(uvarintEncodeBuf, uint64(v-prev))
		if written, err := sw.Write(uvarintEncodeBuf[:uvarintSize]); err != nil {

View on GitHub (pinned to 35b8b99117)