{"record":{"id":"82aed459f2c7bf5e","repo":"thanos-io/thanos","slug":"get-header-from-object-storage-of-s","errorCode":null,"errorMessage":"get header from object storage of %s","messagePattern":"get header from object storage of (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/indexheader/binary_reader.go","lineNumber":196,"sourceCode":"\ttoc  *index.TOC\n}\n\nfunc newChunkedIndexReader(ctx context.Context, bkt objstore.BucketReader, id ulid.ULID) (*chunkedIndexReader, int, error) {\n\tindexFilepath := filepath.Join(id.String(), block.IndexFilename)\n\tattrs, err := bkt.Attributes(ctx, indexFilepath)\n\tif err != nil {\n\t\treturn nil, 0, errors.Wrapf(err, \"get object attributes of %s\", indexFilepath)\n\t}\n\n\trc, err := bkt.GetRange(ctx, indexFilepath, 0, index.HeaderLen)\n\tif err != nil {\n\t\treturn nil, 0, errors.Wrapf(err, \"get TOC from object storage of %s\", indexFilepath)\n\t}\n\n\tb, err := io.ReadAll(rc)\n\tif err != nil {\n\t\trunutil.CloseWithErrCapture(&err, rc, \"close reader\")\n\t\treturn nil, 0, errors.Wrapf(err, \"get header from object storage of %s\", indexFilepath)\n\t}\n\n\tif err := rc.Close(); err != nil {\n\t\treturn nil, 0, errors.Wrap(err, \"close reader\")\n\t}\n\n\tif m := binary.BigEndian.Uint32(b[0:4]); m != index.MagicIndex {\n\t\treturn nil, 0, errors.Errorf(\"invalid magic number %x for %s\", m, indexFilepath)\n\t}\n\n\tversion := int(b[4:5][0])\n\n\tif version != index.FormatV1 && version != index.FormatV2 {\n\t\treturn nil, 0, errors.Errorf(\"not supported index file version %d of %s\", version, indexFilepath)\n\t}\n\n\tir := &chunkedIndexReader{\n\t\tctx:  ctx,","sourceCodeStart":178,"sourceCodeEnd":214,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/indexheader/binary_reader.go#L178-L214","documentation":"Once the ranged reader is obtained, io.ReadAll(rc) streams the header bytes from the object store. This error wraps a failure while reading the response body: the connection dropped mid-read, a timeout hit, or the provider returned a truncated/failed stream. The reader is closed with error capture before returning.","triggerScenarios":"WriteBinary -> newChunkedIndexReader -> io.ReadAll(rc) failing while reading the first HeaderLen bytes of the index object: network interruption mid-body, read deadline exceeded, proxy terminating the connection, or provider-side stream error.","commonSituations":"Flaky network between the cluster and S3/GCS/Azure; very tight HTTP client timeouts in the objstore config; NAT/idle-connection resets on long-lived http.Client connections; VPN/proxy interference.","solutions":["Retry the operation — mid-body read failures are usually transient network issues.","Increase the objstore HTTP client's idle-connection timeout / configure sensible read deadlines.","Check for proxy/NAT devices resetting long connections; keep-alive tuning in the objstore config may help.","Verify stable connectivity to the object-store endpoint (curl the endpoint, check packet loss)."],"exampleFix":"// before: single-shot read\nb, err := io.ReadAll(rc)\n// after: caller ensures retry at a higher level\nif err := backoff.Retry(func() error { _, retryErr := rebuildIndexHeader(ctx, bkt, id); return retryErr }, backoff.NewExponentialBackOff()); err != nil {\n    return errors.Wrap(err, \"index-header rebuild failed after retries\")\n}","handlingStrategy":"retry","validationCode":"// Go: measure round-trip stability before heavy bucket traffic\nfunc reachable(ctx context.Context, bkt objstore.Bucket) error {\n    c, cancel := context.WithTimeout(ctx, 5*time.Second)\n    defer cancel()\n    _, err := bkt.Attributes(c, \"probe-nonexistent\")\n    if err != nil && !isNotFound(err) { return errors.Wrap(err, \"bucket unreachable\") }\n    return nil\n}","typeGuard":null,"tryCatchPattern":"// Go\nif err := backoff.Retry(func() error {\n    _, err := indexheader.NewLazyBinaryReader(ctx, bkt, id, dst, pool)\n    if err != nil && strings.Contains(err.Error(), \"get header from object storage\") {\n        return err // transient mid-body read failure: retry\n    }\n    return backoff.Permanent(err)\n}, backoff.WithMaxTries(backoff.NewExponentialBackOff(), 4)); err != nil {\n    return errors.Wrap(err, \"index-header read failed after retries\")\n}","preventionTips":["Set sane HTTP timeouts and MaxIdleConnsPerHost in the objstore HTTP client config.","Watch for NAT gateways / proxies dropping idle connections; enable TCP keep-alives.","Deploy Thanos with request logging to identify provider-side 5xx/truncation patterns.","Prefer same-region object-store access to reduce cross-region connection churn."],"tags":["object-storage","network","io","thanos"],"backgroundTag":"network-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}