thanos-io/thanos · error

get TOC from object storage of

Error message

get TOC from object storage of %s

What it means

After reading attributes, newChunkedIndexReader issues bkt.GetRange(ctx, indexFilepath, 0, index.HeaderLen) to read the first HeaderLen bytes of the block's index object (magic + version + TOC pointer). This error wraps a failure of that ranged GET: the object-store request itself failed (network, 403, 404, throttling).

Solutions

1. Retry; ranged GETs to S3/GCS are frequently transient (throttling, blips).
2. Verify the object still exists — lifecycle rules or concurrent compaction deletion can race with reads.
3. Check credentials/permissions for GetObject on the bucket.
4. Reduce request rate (scale down store-gateways, enable objstore request logging) if throttling (SlowDown/429) appears.
5. Validate endpoint/region/TLS settings in the objstore config.

Example fix

// before
rc, err := bkt.GetRange(ctx, indexFilepath, 0, index.HeaderLen)
if err != nil { return nil, 0, errors.Wrapf(err, ...) }
// after: wrap with a bounded retry for transient provider errors
rc, err := getRangeWithRetry(ctx, bkt, indexFilepath, 0, index.HeaderLen, 3)

// Go: probe a tiny ranged GET to validate access before the real call
func probeIndexAccess(ctx context.Context, bkt objstore.BucketReader, id ulid.ULID) error {
    rc, err := bkt.GetRange(ctx, path.Join(id.String(), block.IndexFilename), 0, 4)
    if err != nil { return err }
    defer rc.Close()
    _, err = io.ReadFull(rc, make([]byte, 4))
    return err
}

// Go
err := retry.Do(func() error {
    _, err := indexheader.NewBinaryReader(ctx, bkt, id, dst, pool)
    if err == nil { return nil }
    if isTransientObjstoreErr(err) { return err } // retried
    return retry.Unrecoverable(err)
}, retry.Attempts(4), retry.BackOff(backoff.NewExponentialBackOff()))

When it happens

Trigger: WriteBinary -> newChunkedIndexReader -> bkt.GetRange for the first index.HeaderLen bytes failing: object deleted between Attributes and GetRange, permission denied on GET, request timeout, rate limiting (S3 SlowDown), or TLS/endpoint misconfiguration.

Common situations: S3 throttling under heavy store-gateway fan-out; block removed by lifecycle rules mid-read; expired/rotated credentials; wrong region endpoint causing request failures; corporate proxy blocking HTTPS to the object store.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• read meta.json for block
• check exists in bucket
• get meta file
• read meta file
• get header from object storage of

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/8c6b7298cca1a69c. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/indexheader/binary_reader.go:190

type chunkedIndexReader struct {
	ctx  context.Context
	path string
	size uint64
	bkt  objstore.BucketReader
	toc  *index.TOC
}

func newChunkedIndexReader(ctx context.Context, bkt objstore.BucketReader, id ulid.ULID) (*chunkedIndexReader, int, error) {
	indexFilepath := filepath.Join(id.String(), block.IndexFilename)
	attrs, err := bkt.Attributes(ctx, indexFilepath)
	if err != nil {
		return nil, 0, errors.Wrapf(err, "get object attributes of %s", indexFilepath)
	}

	rc, err := bkt.GetRange(ctx, indexFilepath, 0, index.HeaderLen)
	if err != nil {
		return nil, 0, errors.Wrapf(err, "get TOC from object storage of %s", indexFilepath)
	}

	b, err := io.ReadAll(rc)
	if err != nil {
		runutil.CloseWithErrCapture(&err, rc, "close reader")
		return nil, 0, errors.Wrapf(err, "get header from object storage of %s", indexFilepath)
	}

	if err := rc.Close(); err != nil {
		return nil, 0, errors.Wrap(err, "close reader")
	}

	if m := binary.BigEndian.Uint32(b[0:4]); m != index.MagicIndex {
		return nil, 0, errors.Errorf("invalid magic number %x for %s", m, indexFilepath)
	}

	version := int(b[4:5][0])

View on GitHub (pinned to 35b8b99117)