{"record":{"id":"75a18a6d35019b65","repo":"thanos-io/thanos","slug":"get-block-metas-from-oldest","errorCode":null,"errorMessage":"get block metas from oldest","messagePattern":"get block metas from oldest","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":312,"sourceCode":"\t// TODO(bwplotka) so confusing! we need to sort it first. Add comment to TSDB code.\n\tmetas := append([]tsdb.BlockMeta{newMeta}, c.metas...)\n\tsort.Slice(metas, func(i, j int) bool {\n\t\treturn metas[i].MinTime < metas[j].MinTime\n\t})\n\tif o := tsdb.OverlappingBlocks(metas); len(o) > 0 {\n\t\t// TODO(bwplotka): Consider checking if overlaps relates to block in concern?\n\t\treturn errors.Errorf(\"shipping compacted block %s is blocked; overlap spotted: %s\", newMeta.ULID, o.String())\n\t}\n\treturn nil\n}\n\nfunc (s *Shipper) AreAllBlocksUploaded() (bool, error) {\n\ts.mtx.RLock()\n\tdefer s.mtx.RUnlock()\n\n\tmetas, _, err := s.blockMetasFromOldest()\n\tif err != nil {\n\t\treturn false, errors.Wrap(err, \"get block metas from oldest\")\n\t}\n\n\tif len(metas) == 0 {\n\t\treturn true, nil\n\t}\n\n\tmeta, err := ReadMetaFile(s.metadataFilePath)\n\tif err != nil {\n\t\treturn false, errors.Wrap(err, \"read meta file\")\n\t}\n\n\tuploaded := make(map[ulid.ULID]struct{}, len(meta.Uploaded))\n\tfor _, id := range meta.Uploaded {\n\t\tuploaded[id] = struct{}{}\n\t}\n\n\tfor _, m := range metas {\n\t\tif _, ok := uploaded[m.ULID]; !ok {","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L294-L330","documentation":"Wraps errors from blockMetasFromOldest() during AreAllBlocksUploaded, which walks the local directory and re-reads every block's meta.json sorted from oldest to newest. The shipper needs the full local block list to compare against the uploaded manifest, so any read/parse failure of a block meta file aborts the check.","triggerScenarios":"Calling AreAllBlocksUploaded (used by sidecar readiness/gateway) when any block directory under s.dir has a corrupt, missing, or unreadable meta.json, or directory traversal fails with an I/O error.","commonSituations":"Corrupted block after unclean restart; NFS/对象 storage mount flakiness; blocks being deleted by Prometheus retention while the check walks the directory; wrong data directory permissions.","solutions":["Check the wrapped cause; restore or remove the offending block's meta.json / block directory.","Retry the check — transient directory races with Prometheus retention usually resolve on the next call.","Verify the data directory is the actual TSDB dir (not a partially-mounted path).","Run bucket verification tooling to confirm the affected block is safe in object storage before deleting it locally."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"info, err := os.Stat(dataDir)\nif err != nil || !info.IsDir() {\n    return fmt.Errorf(\"data dir unreadable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"ok, err := shipper.AreAllBlocksUploaded()\nif err != nil && strings.Contains(err.Error(), \"get block metas from oldest\") {\n    time.Sleep(retryInterval)\n    ok, err = shipper.AreAllBlocksUploaded() // transient dir-walk races often clear\n}","preventionTips":["Run the readiness check on the same filesystem as the TSDB dir (avoid flaky NFS)","Pin data-dir permissions for the sidecar user","Expect races with Prometheus retention; retry rather than alert immediately"],"tags":["filesystem","block-meta","readiness"],"backgroundTag":"file-read-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}