{"record":{"id":"e56d73b0f782e1c5","repo":"thanos-io/thanos","slug":"read-metadata-for-block-v","errorCode":null,"errorMessage":"read metadata for block %v","messagePattern":"read metadata for block (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":565,"sourceCode":"\t\tif err != nil {\n\t\t\tif s.skipCorruptedBlocks {\n\t\t\t\tlevel.Error(s.logger).Log(\"msg\", \"stat block\", \"err\", err, \"block\", dir)\n\t\t\t\tfailedBlocks = append(failedBlocks, n)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\treturn nil, nil, errors.Wrapf(err, \"stat block %v\", dir)\n\t\t}\n\t\tif !fi.IsDir() {\n\t\t\tcontinue\n\t\t}\n\t\tm, err := metadata.ReadFromDir(dir)\n\t\tif err != nil {\n\t\t\tif s.skipCorruptedBlocks {\n\t\t\t\tlevel.Error(s.logger).Log(\"msg\", \"read metadata for block\", \"err\", err, \"block\", dir)\n\t\t\t\tfailedBlocks = append(failedBlocks, n)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\treturn nil, nil, errors.Wrapf(err, \"read metadata for block %v\", dir)\n\t\t}\n\t\tmetas = append(metas, m)\n\t}\n\tsort.Slice(metas, func(i, j int) bool {\n\t\treturn metas[i].MinTime < metas[j].MinTime\n\t})\n\n\tif len(failedBlocks) > 0 {\n\t\terr = ErrorSyncBlockCorrupted\n\t}\n\treturn metas, failedBlocks, err\n}\n\nfunc hardlinkBlock(src, dst string) error {\n\tchunkDir := filepath.Join(dst, block.ChunksDirname)\n\n\tif err := os.MkdirAll(chunkDir, 0750); err != nil {\n\t\treturn errors.Wrap(err, \"create chunks dir\")","sourceCodeStart":547,"sourceCodeEnd":583,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L547-L583","documentation":"blockMetasFromOldest iterates the shipper's local data directory and calls metadata.ReadFromDir on every block directory to load its meta.json. When reading or parsing a block's metadata fails and skipCorruptedBlocks is false, the whole sync/upload iteration is aborted with this wrapped error naming the block directory. It signals that a local TSDB block is unreadable or its meta.json is missing/corrupt, so its upload state cannot be determined.","triggerScenarios":"Calling Sync() or AreAllBlocksUploaded() on a Shipper whose local dir contains a block directory whose meta.json is missing, truncated, fails JSON validation, or cannot be stat/read, with skipCorruptedBlocks=false.","commonSituations":"Crash or OOM-kill mid-compaction leaving a partially written block; disk corruption on the Prometheus/Thanos sidecar data volume; manually deleted or partially copied block directories; NFS/EFS volumes dropping files; running an older Prometheus producing meta files the current metadata parser rejects.","solutions":["Set Shipper.WithSkipCorruptedBlocks(true) so bad blocks are logged and skipped (Sync then returns ErrorSyncBlockCorrupted) instead of failing outright.","Inspect the named block directory: verify <dir>/meta.json exists and is valid JSON with the expected fields (ulid, minTime, maxTime, version).","Delete the corrupted block directory from the data dir and let Prometheus/Thanos re-sync it, or restore it from a healthy replica.","Check filesystem health (dmesg, fsck) if multiple blocks fail to read; a failing volume usually corrupts several blocks at once."],"exampleFix":"// before: shipper aborts on any unreadable block meta\nshipper.NewPrometheusShipper(...)\n// after: tolerate corrupted blocks and let sync report them\nshipper.NewPrometheusShipper(...).WithSkipCorruptedBlocks(true)","handlingStrategy":"fallback","validationCode":"meta, err := metadata.ReadFromDir(blockDir)\nif err != nil {\n\tlevel.Error(logger).Log(\"msg\", \"skipping corrupted block\", \"block\", blockDir, \"err\", err)\n\treturn nil // skip instead of failing the whole sync\n}","typeGuard":"func blockReadable(dir string) bool {\n\t_, err := os.Stat(filepath.Join(dir, \"meta.json\"))\n\treturn err == nil\n}","tryCatchPattern":"metas, failed, err := sh.Sync(ctx)\nif errors.Is(err, shipper.ErrorSyncBlockCorrupted) {\n\tlevel.Warn(logger).Log(\"msg\", \"some blocks corrupted, skipped\", \"failed\", failed)\n\treturn nil\n} else if err != nil {\n\treturn errors.Wrap(err, \"sync\")\n}","preventionTips":["Enable WithSkipCorruptedBlocks(true) so one bad block does not stop all uploads.","Run with clean shutdown handling so compaction is never interrupted mid-block-write.","Monitor for partially written block dirs (meta.json missing) and alert.","Use reliable, non-NFS local storage for the TSDB data dir."],"tags":["filesystem","corrupt-data","block-metadata","thanos"],"backgroundTag":"file-read-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}