{"record":{"id":"8c6b7298cca1a69c","repo":"thanos-io/thanos","slug":"get-toc-from-object-storage-of-s","errorCode":null,"errorMessage":"get TOC from object storage of %s","messagePattern":"get TOC from object storage of (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/indexheader/binary_reader.go","lineNumber":190,"sourceCode":"\ntype chunkedIndexReader struct {\n\tctx  context.Context\n\tpath string\n\tsize uint64\n\tbkt  objstore.BucketReader\n\ttoc  *index.TOC\n}\n\nfunc newChunkedIndexReader(ctx context.Context, bkt objstore.BucketReader, id ulid.ULID) (*chunkedIndexReader, int, error) {\n\tindexFilepath := filepath.Join(id.String(), block.IndexFilename)\n\tattrs, err := bkt.Attributes(ctx, indexFilepath)\n\tif err != nil {\n\t\treturn nil, 0, errors.Wrapf(err, \"get object attributes of %s\", indexFilepath)\n\t}\n\n\trc, err := bkt.GetRange(ctx, indexFilepath, 0, index.HeaderLen)\n\tif err != nil {\n\t\treturn nil, 0, errors.Wrapf(err, \"get TOC from object storage of %s\", indexFilepath)\n\t}\n\n\tb, err := io.ReadAll(rc)\n\tif err != nil {\n\t\trunutil.CloseWithErrCapture(&err, rc, \"close reader\")\n\t\treturn nil, 0, errors.Wrapf(err, \"get header from object storage of %s\", indexFilepath)\n\t}\n\n\tif err := rc.Close(); err != nil {\n\t\treturn nil, 0, errors.Wrap(err, \"close reader\")\n\t}\n\n\tif m := binary.BigEndian.Uint32(b[0:4]); m != index.MagicIndex {\n\t\treturn nil, 0, errors.Errorf(\"invalid magic number %x for %s\", m, indexFilepath)\n\t}\n\n\tversion := int(b[4:5][0])\n","sourceCodeStart":172,"sourceCodeEnd":208,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/indexheader/binary_reader.go#L172-L208","documentation":"After reading attributes, newChunkedIndexReader issues bkt.GetRange(ctx, indexFilepath, 0, index.HeaderLen) to read the first HeaderLen bytes of the block's index object (magic + version + TOC pointer). This error wraps a failure of that ranged GET: the object-store request itself failed (network, 403, 404, throttling).","triggerScenarios":"WriteBinary -> newChunkedIndexReader -> bkt.GetRange for the first index.HeaderLen bytes failing: object deleted between Attributes and GetRange, permission denied on GET, request timeout, rate limiting (S3 SlowDown), or TLS/endpoint misconfiguration.","commonSituations":"S3 throttling under heavy store-gateway fan-out; block removed by lifecycle rules mid-read; expired/rotated credentials; wrong region endpoint causing request failures; corporate proxy blocking HTTPS to the object store.","solutions":["Retry; ranged GETs to S3/GCS are frequently transient (throttling, blips).","Verify the object still exists — lifecycle rules or concurrent compaction deletion can race with reads.","Check credentials/permissions for GetObject on the bucket.","Reduce request rate (scale down store-gateways, enable objstore request logging) if throttling (SlowDown/429) appears.","Validate endpoint/region/TLS settings in the objstore config."],"exampleFix":"// before\nrc, err := bkt.GetRange(ctx, indexFilepath, 0, index.HeaderLen)\nif err != nil { return nil, 0, errors.Wrapf(err, ...) }\n// after: wrap with a bounded retry for transient provider errors\nrc, err := getRangeWithRetry(ctx, bkt, indexFilepath, 0, index.HeaderLen, 3)","handlingStrategy":"retry","validationCode":"// Go: probe a tiny ranged GET to validate access before the real call\nfunc probeIndexAccess(ctx context.Context, bkt objstore.BucketReader, id ulid.ULID) error {\n    rc, err := bkt.GetRange(ctx, path.Join(id.String(), block.IndexFilename), 0, 4)\n    if err != nil { return err }\n    defer rc.Close()\n    _, err = io.ReadFull(rc, make([]byte, 4))\n    return err\n}","typeGuard":null,"tryCatchPattern":"// Go\nerr := retry.Do(func() error {\n    _, err := indexheader.NewBinaryReader(ctx, bkt, id, dst, pool)\n    if err == nil { return nil }\n    if isTransientObjstoreErr(err) { return err } // retried\n    return retry.Unrecoverable(err)\n}, retry.Attempts(4), retry.BackOff(backoff.NewExponentialBackOff()))","preventionTips":["Configure S3 request-rate limits / concurrency caps to avoid SlowDown throttling.","Pin correct region/endpoint in the objstore config; wrong-region requests fail intermittently.","Rotate credentials before expiry and use IRSA/workload identity where possible.","Check object lifecycle rules so index objects are not expired while blocks are still queried."],"tags":["object-storage","network","thanos","range-request"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}