thanos-io/thanos · error

read dir

Error message

read dir

What it means

This error wraps a failure from dir.ReadDir(-1) in blockMetasFromOldest, which lists all entries of the opened shipper directory to discover block directories. The dir handle was opened successfully but the directory enumeration I/O failed.

Solutions

1. Retry Sync once the transient filesystem error clears (network mount reconnect, I/O recovery).
2. Check dmesg/journal for disk I/O errors; replace or repair the failing storage.
3. Stop tools that delete/replace the blocks dir concurrently with shipping.
4. Restart the process to reopen the directory on a healthy mount.

Example fix

// before: NFS stale handle during ReadDir
read dir: readdir /prometheus: stale NFS file handle

// after: remount the volume with hard mounts
$ sudo mount -o remount,hard,intr nfsserver:/export/prometheus /prometheus

d, err := os.Open(dirPath)
if err != nil {
    return err
}
if _, err := d.Readdirnames(1); err != nil {
    d.Close()
    return fmt.Errorf("directory not listable: %w", err)
}
d.Close()

func dirListable(path string) bool {
    d, err := os.Open(path)
    if err != nil {
        return false
    }
    defer d.Close()
    _, err = d.Readdirnames(1)
    return err == nil || errors.Is(err, io.EOF)
}

fis, err := dir.ReadDir(-1)
if err != nil {
    if errors.Is(err, syscall.ESTALE) || errors.Is(err, syscall.EIO) {
        return nil, nil, retryable(errors.Wrap(err, "read dir"))
    }
    return nil, nil, errors.Wrap(err, "read dir")
}

When it happens

Trigger: Calling Shipper.Sync or AreAllBlocksUploaded when the underlying directory is removed while the handle is open, the filesystem returns I/O errors (disk failure, network filesystem hiccup), or the fd becomes invalid after the directory is replaced on a volatile mount.

Common situations: NFS/EFS stale handle errors after server-side volume changes; hardware disk errors surfacing as EIO during readdir; concurrent retention/deletion tooling removing the directory between Open and ReadDir.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• open dir
• remove any existing index at path
• hard link block
• write meta file
• stat block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/792b08ab938a7e95. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:533

	if s.uploadConcurrency > 0 {
		uploadOptions = append(uploadOptions, objstore.WithUploadConcurrency(s.uploadConcurrency))
	}
	return block.Upload(ctx, s.logger, s.bucket, absUpdir, s.hashFunc, uploadOptions...)
}

// blockMetasFromOldest returns the block meta of each block found in dir
// sorted by minTime asc.
func (s *Shipper) blockMetasFromOldest() (metas []*metadata.Meta, failedBlocks []string, _ error) {

	dir, err := s.dir.Open(".")
	if err != nil {
		return nil, nil, errors.Wrap(err, "open dir")
	}
	defer runutil.CloseWithLogOnErr(s.logger, dir, "shipper dir")

	fis, err := dir.ReadDir(-1)
	if err != nil {
		return nil, nil, errors.Wrap(err, "read dir")
	}

	names := make([]string, 0, len(fis))
	for _, fi := range fis {
		names = append(names, fi.Name())
	}
	for _, n := range names {
		if _, ok := block.IsBlockDir(n); !ok {
			continue
		}
		dir := filepath.Join(s.dir.Name(), n)

		fi, err := s.dir.Stat(n)
		if err != nil {
			if s.skipCorruptedBlocks {
				level.Error(s.logger).Log("msg", "stat block", "err", err, "block", dir)
				failedBlocks = append(failedBlocks, n)
				continue

View on GitHub (pinned to 35b8b99117)