{"record":{"id":"539d19769862870a","repo":"thanos-io/thanos","slug":"stat-block-v","errorCode":null,"errorMessage":"stat block %v","messagePattern":"stat block (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":553,"sourceCode":"\n\tnames := make([]string, 0, len(fis))\n\tfor _, fi := range fis {\n\t\tnames = append(names, fi.Name())\n\t}\n\tfor _, n := range names {\n\t\tif _, ok := block.IsBlockDir(n); !ok {\n\t\t\tcontinue\n\t\t}\n\t\tdir := filepath.Join(s.dir.Name(), n)\n\n\t\tfi, err := s.dir.Stat(n)\n\t\tif err != nil {\n\t\t\tif s.skipCorruptedBlocks {\n\t\t\t\tlevel.Error(s.logger).Log(\"msg\", \"stat block\", \"err\", err, \"block\", dir)\n\t\t\t\tfailedBlocks = append(failedBlocks, n)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\treturn nil, nil, errors.Wrapf(err, \"stat block %v\", dir)\n\t\t}\n\t\tif !fi.IsDir() {\n\t\t\tcontinue\n\t\t}\n\t\tm, err := metadata.ReadFromDir(dir)\n\t\tif err != nil {\n\t\t\tif s.skipCorruptedBlocks {\n\t\t\t\tlevel.Error(s.logger).Log(\"msg\", \"read metadata for block\", \"err\", err, \"block\", dir)\n\t\t\t\tfailedBlocks = append(failedBlocks, n)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\treturn nil, nil, errors.Wrapf(err, \"read metadata for block %v\", dir)\n\t\t}\n\t\tmetas = append(metas, m)\n\t}\n\tsort.Slice(metas, func(i, j int) bool {\n\t\treturn metas[i].MinTime < metas[j].MinTime\n\t})","sourceCodeStart":535,"sourceCodeEnd":571,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L535-L571","documentation":"This error wraps os.Stat failures on an individual block directory path during blockMetasFromOldest. When s.skipCorruptedBlocks is false (default), any entry in the shipper dir whose stat fails aborts the whole Sync. When skipping is enabled, the block is logged and added to failedBlocks instead.","triggerScenarios":"Calling Shipper.Sync or AreAllBlocksUploaded when an entry inside the shipper dir cannot be stat'ed: a dangling symlink, an entry removed concurrently between ReadDir and Stat, a broken filesystem entry, or permission denial on a subdirectory.","commonSituations":"Partial block deletion by retention cleanup leaving dangling entries; permission mismatch between Prometheus (creator) and Thanos sidecar (reader) users; corrupted directory entries after unclean shutdown/power loss.","solutions":["Remove the broken entry (dangling symlink/corrupt dir) from the shipper dir, then re-run Sync.","Enable skipCorruptedBlocks (if using the Shipper API directly) so bad blocks are logged and skipped instead of failing the sync.","Fix permissions on the offending subdirectory so stat succeeds.","Run fsck / verify the volume if entries are corrupted after a crash."],"exampleFix":"// before: one dangling symlink fails the whole sync\nreturn nil, nil, errors.Wrapf(err, \"stat block %v\", dir)\n\n// after: construct shipper with skipping enabled\nshipper.New(..., shipper.WithSkipCorruptedBlocks())","handlingStrategy":"try-catch","validationCode":"entries, err := os.ReadDir(shipperDir)\nif err != nil {\n    return err\n}\nfor _, e := range entries {\n    p := filepath.Join(shipperDir, e.Name())\n    if _, err := os.Stat(p); err != nil {\n        log.Printf(\"removing dangling entry %s: %v\", p, err)\n        os.RemoveAll(p)\n    }\n}","typeGuard":"func entryAccessible(base, name string) bool {\n    _, err := os.Stat(filepath.Join(base, name))\n    return err == nil\n}","tryCatchPattern":"if err != nil {\n    if s.skipCorruptedBlocks {\n        level.Error(s.logger).Log(\"msg\", \"stat block\", \"err\", err, \"block\", dir)\n        failedBlocks = append(failedBlocks, n)\n        continue\n    }\n    return nil, nil, errors.Wrapf(err, \"stat block %v\", dir)\n}","preventionTips":["Enable skipCorruptedBlocks so one bad block directory cannot stall the entire sync.","Ensure the retention cleaner and the shipper do not race on block dirs.","Fix uid/gid consistency between processes touching the blocks dir.","After unclean shutdowns, scan for and remove dangling symlinks/broken entries before restart."],"tags":["go","filesystem","stat","shipper","blocks"],"backgroundTag":"file-not-found","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}