{"record":{"id":"792b08ab938a7e95","repo":"thanos-io/thanos","slug":"read-dir-792b08","errorCode":null,"errorMessage":"read dir","messagePattern":"read dir","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":533,"sourceCode":"\tif s.uploadConcurrency > 0 {\n\t\tuploadOptions = append(uploadOptions, objstore.WithUploadConcurrency(s.uploadConcurrency))\n\t}\n\treturn block.Upload(ctx, s.logger, s.bucket, absUpdir, s.hashFunc, uploadOptions...)\n}\n\n// blockMetasFromOldest returns the block meta of each block found in dir\n// sorted by minTime asc.\nfunc (s *Shipper) blockMetasFromOldest() (metas []*metadata.Meta, failedBlocks []string, _ error) {\n\n\tdir, err := s.dir.Open(\".\")\n\tif err != nil {\n\t\treturn nil, nil, errors.Wrap(err, \"open dir\")\n\t}\n\tdefer runutil.CloseWithLogOnErr(s.logger, dir, \"shipper dir\")\n\n\tfis, err := dir.ReadDir(-1)\n\tif err != nil {\n\t\treturn nil, nil, errors.Wrap(err, \"read dir\")\n\t}\n\n\tnames := make([]string, 0, len(fis))\n\tfor _, fi := range fis {\n\t\tnames = append(names, fi.Name())\n\t}\n\tfor _, n := range names {\n\t\tif _, ok := block.IsBlockDir(n); !ok {\n\t\t\tcontinue\n\t\t}\n\t\tdir := filepath.Join(s.dir.Name(), n)\n\n\t\tfi, err := s.dir.Stat(n)\n\t\tif err != nil {\n\t\t\tif s.skipCorruptedBlocks {\n\t\t\t\tlevel.Error(s.logger).Log(\"msg\", \"stat block\", \"err\", err, \"block\", dir)\n\t\t\t\tfailedBlocks = append(failedBlocks, n)\n\t\t\t\tcontinue","sourceCodeStart":515,"sourceCodeEnd":551,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L515-L551","documentation":"This error wraps a failure from dir.ReadDir(-1) in blockMetasFromOldest, which lists all entries of the opened shipper directory to discover block directories. The dir handle was opened successfully but the directory enumeration I/O failed.","triggerScenarios":"Calling Shipper.Sync or AreAllBlocksUploaded when the underlying directory is removed while the handle is open, the filesystem returns I/O errors (disk failure, network filesystem hiccup), or the fd becomes invalid after the directory is replaced on a volatile mount.","commonSituations":"NFS/EFS stale handle errors after server-side volume changes; hardware disk errors surfacing as EIO during readdir; concurrent retention/deletion tooling removing the directory between Open and ReadDir.","solutions":["Retry Sync once the transient filesystem error clears (network mount reconnect, I/O recovery).","Check dmesg/journal for disk I/O errors; replace or repair the failing storage.","Stop tools that delete/replace the blocks dir concurrently with shipping.","Restart the process to reopen the directory on a healthy mount."],"exampleFix":"// before: NFS stale handle during ReadDir\nread dir: readdir /prometheus: stale NFS file handle\n\n// after: remount the volume with hard mounts\n$ sudo mount -o remount,hard,intr nfsserver:/export/prometheus /prometheus","handlingStrategy":"retry","validationCode":"d, err := os.Open(dirPath)\nif err != nil {\n    return err\n}\nif _, err := d.Readdirnames(1); err != nil {\n    d.Close()\n    return fmt.Errorf(\"directory not listable: %w\", err)\n}\nd.Close()","typeGuard":"func dirListable(path string) bool {\n    d, err := os.Open(path)\n    if err != nil {\n        return false\n    }\n    defer d.Close()\n    _, err = d.Readdirnames(1)\n    return err == nil || errors.Is(err, io.EOF)\n}","tryCatchPattern":"fis, err := dir.ReadDir(-1)\nif err != nil {\n    if errors.Is(err, syscall.ESTALE) || errors.Is(err, syscall.EIO) {\n        return nil, nil, retryable(errors.Wrap(err, \"read dir\"))\n    }\n    return nil, nil, errors.Wrap(err, \"read dir\")\n}","preventionTips":["Avoid NFS for TSDB data; if unavoidable, use hard/intr mount options.","Do not delete or replace the blocks directory while the shipper process holds it open.","Monitor dmesg for disk I/O errors on the storage backing the data dir.","Wrap Sync in a bounded backoff retry for transient filesystem errors."],"tags":["go","filesystem","readdir","shipper","io"],"backgroundTag":"file-read-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}