thanos-io/thanos · error

clean upload directory

Error message

clean upload directory

What it means

Wraps an error from s.dir.RemoveAll(updir) in upload(): before hard-linking block files into the temporary thanos/upload/<ULID> directory, the shipper removes any stale copy. Failure to remove it (permissions, read-only fs, EBUSY on a mount) aborts the upload of that block.

Solutions

1. Delete thanos/upload/<ULID> manually with sufficient privileges and re-run Sync.
2. Chown the data dir to the sidecar's runtime user (or fix securityContext/fsGroup in Kubernetes).
3. Check that the filesystem is mounted read-write (mount | grep, dmesg for I/O errors).
4. Ensure no other process (backup agent, old sidecar) holds files open inside the upload dir.

Example fix

// before
// clean upload directory: remove /data/thanos/upload/01ARZ...: permission denied
// after
// $ docker run --rm -v /data:/data busybox rm -rf /data/thanos/upload
// $ chown -R 65534:65534 /data/thanos

updir := filepath.Join(dataDir, "thanos", "upload")
if info, err := os.Stat(updir); err == nil && !info.IsDir() {
    return fmt.Errorf("%s exists and is not a directory", updir)
}

if err != nil && strings.Contains(err.Error(), "clean upload directory") {
    // stale root-owned upload dir; run one-time cleanup with elevated perms then restart
    log.Printf("requires manual cleanup of thanos/upload: %v", err)
}

When it happens

Trigger: Calling Sync → upload when the directory thanos/upload/<ULID> exists and cannot be deleted — e.g. permissions differ (previous run as root), filesystem is read-only, or a stale bind mount holds it.

Common situations: Sidecar previously run as a different user leaving root-owned upload dirs; the data dir shared as a Kubernetes volume with wrong fsGroup; disk remounted read-only after errors.

Understand the failure class

Background: "Permission denied" / "Failed to write" file errors: why a library can't write its files to disk (EACCES, EPERM, ENOSPC) and how to fix them — this error's family across 43 libraries.

Related errors

• create upload dir
• create chunks dir
• create working compact directory
• create working downsample directory
• create dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/19fcdd680de89988. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:487

	for _, id := range meta.Uploaded {
		ret[id] = struct{}{}
	}

	return ret
}

// sync uploads the block if not exists in remote storage.
// TODO(khyatisoneji): Double check if block does not have deletion-mark.json for some reason, otherwise log it or return error.
func (s *Shipper) upload(ctx context.Context, meta *metadata.Meta) error {
	level.Info(s.logger).Log("msg", "upload new block", "id", meta.ULID)

	// We hard-link the files into a temporary upload directory so we are not affected
	// by other operations happening against the TSDB directory.
	updir := filepath.Join("thanos", "upload", meta.ULID.String())

	// Remove updir just in case.
	if err := s.dir.RemoveAll(updir); err != nil {
		return errors.Wrap(err, "clean upload directory")
	}
	if err := s.dir.MkdirAll(updir, 0750); err != nil {
		return errors.Wrap(err, "create upload dir")
	}
	defer func() {
		if err := s.dir.RemoveAll(updir); err != nil {
			level.Error(s.logger).Log("msg", "failed to clean upload directory", "err", err)
		}
	}()

	absUpdir := filepath.Join(s.dir.Name(), updir)
	dir := filepath.Join(s.dir.Name(), meta.ULID.String())
	if err := hardlinkBlock(dir, absUpdir); err != nil {
		return errors.Wrap(err, "hard link block")
	}
	// Attach current labels and write a new meta file with Thanos extensions.
	if lset := s.labels(); !lset.IsEmpty() {
		lset.Range(func(l labels.Label) {

View on GitHub (pinned to 35b8b99117)