thanos-io/thanos · error

get header from object storage of

Error message

get header from object storage of %s

What it means

Once the ranged reader is obtained, io.ReadAll(rc) streams the header bytes from the object store. This error wraps a failure while reading the response body: the connection dropped mid-read, a timeout hit, or the provider returned a truncated/failed stream. The reader is closed with error capture before returning.

Solutions

1. Retry the operation — mid-body read failures are usually transient network issues.
2. Increase the objstore HTTP client's idle-connection timeout / configure sensible read deadlines.
3. Check for proxy/NAT devices resetting long connections; keep-alive tuning in the objstore config may help.
4. Verify stable connectivity to the object-store endpoint (curl the endpoint, check packet loss).

Example fix

// before: single-shot read
b, err := io.ReadAll(rc)
// after: caller ensures retry at a higher level
if err := backoff.Retry(func() error { _, retryErr := rebuildIndexHeader(ctx, bkt, id); return retryErr }, backoff.NewExponentialBackOff()); err != nil {
    return errors.Wrap(err, "index-header rebuild failed after retries")
}

// Go: measure round-trip stability before heavy bucket traffic
func reachable(ctx context.Context, bkt objstore.Bucket) error {
    c, cancel := context.WithTimeout(ctx, 5*time.Second)
    defer cancel()
    _, err := bkt.Attributes(c, "probe-nonexistent")
    if err != nil && !isNotFound(err) { return errors.Wrap(err, "bucket unreachable") }
    return nil
}

// Go
if err := backoff.Retry(func() error {
    _, err := indexheader.NewLazyBinaryReader(ctx, bkt, id, dst, pool)
    if err != nil && strings.Contains(err.Error(), "get header from object storage") {
        return err // transient mid-body read failure: retry
    }
    return backoff.Permanent(err)
}, backoff.WithMaxTries(backoff.NewExponentialBackOff(), 4)); err != nil {
    return errors.Wrap(err, "index-header read failed after retries")
}

When it happens

Trigger: WriteBinary -> newChunkedIndexReader -> io.ReadAll(rc) failing while reading the first HeaderLen bytes of the index object: network interruption mid-body, read deadline exceeded, proxy terminating the connection, or provider-side stream error.

Common situations: Flaky network between the cluster and S3/GCS/Azure; very tight HTTP client timeouts in the objstore config; NAT/idle-connection resets on long-lived http.Client connections; VPN/proxy interference.

Related errors

• read meta file
• read meta.json for block
• check exists in bucket
• get meta file
• get TOC from object storage of

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/82aed459f2c7bf5e. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/indexheader/binary_reader.go:196

	toc  *index.TOC
}

func newChunkedIndexReader(ctx context.Context, bkt objstore.BucketReader, id ulid.ULID) (*chunkedIndexReader, int, error) {
	indexFilepath := filepath.Join(id.String(), block.IndexFilename)
	attrs, err := bkt.Attributes(ctx, indexFilepath)
	if err != nil {
		return nil, 0, errors.Wrapf(err, "get object attributes of %s", indexFilepath)
	}

	rc, err := bkt.GetRange(ctx, indexFilepath, 0, index.HeaderLen)
	if err != nil {
		return nil, 0, errors.Wrapf(err, "get TOC from object storage of %s", indexFilepath)
	}

	b, err := io.ReadAll(rc)
	if err != nil {
		runutil.CloseWithErrCapture(&err, rc, "close reader")
		return nil, 0, errors.Wrapf(err, "get header from object storage of %s", indexFilepath)
	}

	if err := rc.Close(); err != nil {
		return nil, 0, errors.Wrap(err, "close reader")
	}

	if m := binary.BigEndian.Uint32(b[0:4]); m != index.MagicIndex {
		return nil, 0, errors.Errorf("invalid magic number %x for %s", m, indexFilepath)
	}

	version := int(b[4:5][0])

	if version != index.FormatV1 && version != index.FormatV2 {
		return nil, 0, errors.Errorf("not supported index file version %d of %s", version, indexFilepath)
	}

	ir := &chunkedIndexReader{
		ctx:  ctx,

View on GitHub (pinned to 35b8b99117)