What causes "copy posting offsets" in thanos-io/thanos?

Inspect the wrapped error: writer-side ENOSPC requires freeing space or growing the cache volume.

copy posting offsets

Error message

copy posting offsets

What it means

CopyPostingsOffsets fetched the postings offset table range but io.CopyBuffer failed streaming it to the destination writer, wrapped with 'copy posting offsets'. Like the symbols copy failure, this can originate from the local writer or the remote body.

Solutions

Inspect the wrapped error: writer-side ENOSPC requires freeing space or growing the cache volume.
Retry on read/network failures; configure the retrying objstore client.
Reduce concurrent header builds or increase buffer sizes to improve reliability of large copies.
Check the context deadline causing cancellation if errors occur at a consistent interval.

Defensive patterns

Strategy: try-catch

Validate before calling

if free, _ := diskFree(cacheDir); free < estPostingsSize {
    return fmt.Errorf("not enough space for postings offsets of %s", path)
}

Try / catch

var perr *fs.PathError
if errors.As(err, &perr) {
    // disk-full or permission issue on the cache volume
} else if isRetryableNetErr(err) {
    retry.WithBackoff(func() error { return copyPostingsOffsets(w, buf) })
}

Prevention

Provision cache volumes with headroom for postings tables of high-cardinality blocks.
Alert on cache disk utilization thresholds.
Use retrying bucket clients to absorb transient body-read failures.
Set copy deadlines proportional to block size; avoid fixed short timeouts.

When it happens

Trigger: CopyPostingsOffsets when io.CopyBuffer(w, rc, buf) errors: local disk full or I/O error while writing the index-header file, or the object-storage body read failed mid-stream (connection reset, timeout, context cancellation).

Common situations: Index-header cache volume full on store gateways; network blips during large postings tables (high-cardinality blocks have big postings sections); context timeouts during startup fan-out.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/b6fa261d948ec806. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/indexheader/binary_reader.go:274

	}
	defer runutil.CloseWithErrCapture(&err, rc, "close symbol reader")

	if _, err := io.CopyBuffer(w, rc, buf); err != nil {
		return errors.Wrap(err, "copy symbols")
	}

	return nil
}

func (r *chunkedIndexReader) CopyPostingsOffsets(w io.Writer, buf []byte) (err error) {
	rc, err := r.bkt.GetRange(r.ctx, r.path, int64(r.toc.PostingsTable), int64(r.size-r.toc.PostingsTable))
	if err != nil {
		return errors.Wrapf(err, "get posting offset table from object storage of %s", r.path)
	}
	defer runutil.CloseWithErrCapture(&err, rc, "close posting offsets reader")

	if _, err := io.CopyBuffer(w, rc, buf); err != nil {
		return errors.Wrap(err, "copy posting offsets")
	}

	return nil
}

// TODO(bwplotka): Add padding for efficient read.
type binaryWriter struct {
	writer PosWriter

	toc BinaryTOC

	// Reusable memory.
	buf encoding.Encbuf

	crc32 hash.Hash
}

func newBinaryWriter(id ulid.ULID, cacheFilename string, buf []byte) (w *binaryWriter, err error) {

View on GitHub (pinned to 35b8b99117)