thanos-io/thanos · error

write meta file

Error message

write meta file

What it means

This error wraps a failure from metadata.Meta.WriteToDir, which serializes the block meta (with Thanos extensions: labels, source, segment files) as meta.json into the block directory before upload. It is thrown when writing/flushing meta.json to the block dir fails, typically due to filesystem permission, disk-full, or directory-state issues.

Solutions

1. Check free disk space on the data volume (df -h) and free space or grow the volume.
2. Fix permissions/ownership of the blocks directory so the shipping process can write meta.json.
3. Ensure the volume is mounted read-write (check /proc/mounts for 'ro').
4. Verify the block directory still exists (no concurrent cleanup); re-run Sync after the block is restored.

Example fix

// before: container runs as uid 65534 against root-owned data
$ kubectl exec sidecar -- ls -ld /prometheus
// after: set matching securityContext
securityContext:
  runAsUser: 0        # or chown data dir to the sidecar uid
  fsGroup: 2000

info, err := os.Stat(absUpdir)
if err != nil {
    return fmt.Errorf("block dir missing: %w", err)
}
if !info.IsDir() {
    return errors.New("block path is not a directory")
}
// probe writability
probe := filepath.Join(absUpdir, ".write-test")
if err := os.WriteFile(probe, nil, 0o600); err != nil {
    return fmt.Errorf("block dir not writable: %w", err)
}
os.Remove(probe)

func dirWritable(path string) bool {
    probe := filepath.Join(path, ".writetest")
    if err := os.WriteFile(probe, nil, 0o600); err != nil {
        return false
    }
    os.Remove(probe)
    return true
}

if err := meta.WriteToDir(s.logger, absUpdir); err != nil {
    if errors.Is(err, syscall.ENOSPC) {
        return errors.Wrap(err, "write meta file: disk full, free space and retry")
    }
    if os.IsPermission(err) {
        return errors.Wrap(err, "write meta file: check volume permissions/ownership")
    }
    return errors.Wrap(err, "write meta file")
}

When it happens

Trigger: Calling Shipper.Sync when the block directory (hardlinked ULID dir) is not writable by the process, the filesystem is out of space or read-only, or the directory disappeared between hardlink and meta write (concurrent deletion).

Common situations: Read-only volume after pod rescheduling; disk-full on the Prometheus data volume; running the process as non-root against a data dir owned by another user; antivirus/backup tooling locking files on the volume.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• hard link block
• open dir
• read dir
• stat block
• open data dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/cd2d08f84987bf86. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:512

			level.Error(s.logger).Log("msg", "failed to clean upload directory", "err", err)
		}
	}()

	absUpdir := filepath.Join(s.dir.Name(), updir)
	dir := filepath.Join(s.dir.Name(), meta.ULID.String())
	if err := hardlinkBlock(dir, absUpdir); err != nil {
		return errors.Wrap(err, "hard link block")
	}
	// Attach current labels and write a new meta file with Thanos extensions.
	if lset := s.labels(); !lset.IsEmpty() {
		lset.Range(func(l labels.Label) {
			meta.Thanos.Labels[l.Name] = l.Value
		})
	}
	meta.Thanos.Source = s.source
	meta.Thanos.SegmentFiles = block.GetSegmentFiles(absUpdir)
	if err := meta.WriteToDir(s.logger, absUpdir); err != nil {
		return errors.Wrap(err, "write meta file")
	}
	var uploadOptions []objstore.UploadOption
	if s.uploadConcurrency > 0 {
		uploadOptions = append(uploadOptions, objstore.WithUploadConcurrency(s.uploadConcurrency))
	}
	return block.Upload(ctx, s.logger, s.bucket, absUpdir, s.hashFunc, uploadOptions...)
}

// blockMetasFromOldest returns the block meta of each block found in dir
// sorted by minTime asc.
func (s *Shipper) blockMetasFromOldest() (metas []*metadata.Meta, failedBlocks []string, _ error) {

	dir, err := s.dir.Open(".")
	if err != nil {
		return nil, nil, errors.Wrap(err, "open dir")
	}
	defer runutil.CloseWithLogOnErr(s.logger, dir, "shipper dir")

View on GitHub (pinned to 35b8b99117)