thanos-io/thanos · error

create upload dir

Error message

create upload dir

What it means

Wraps an error from s.dir.MkdirAll(updir, 0750) in upload(): after cleaning, the shipper creates the temporary thanos/upload/<ULID> directory to hard-link block files into. Failure means the upload of that block cannot proceed.

Solutions

1. Check the wrapped cause; fix disk space/inodes (df -h /data; df -i /data).
2. Grant the sidecar write permission on <data-dir>/thanos (chown/chmod or K8s fsGroup).
3. Ensure the volume is mounted read-write (remove :ro mount flag).
4. Verify the data directory path in --data-dir points to a writable filesystem.

Example fix

// before
// create upload dir: mkdir /data/thanos/upload/01ARZ...: read-only file system
// after (kubernetes)
// volumes:
//   - name: data
//     emptyDir: {}        # or remove readOnly: true from the PVC mount
// securityContext:
//   fsGroup: 65534

thanosDir := filepath.Join(dataDir, "thanos")
if err := os.MkdirAll(thanosDir, 0750); err != nil {
    return fmt.Errorf("data dir not writable for shipper: %w", err)
}

if err != nil && strings.Contains(err.Error(), "create upload dir") {
    switch {
    case strings.Contains(err.Error(), "ENOSPC"):
        log.Printf("disk full — free space before next sync")
    case strings.Contains(err.Error(), "permission denied", "read-only"):
        log.Printf("fix data-dir permissions/mount: %v", err)
    }
}

When it happens

Trigger: Calling Sync → upload when creating thanos/upload/<ULID> fails: parent dir missing/unwritable, permission denied, disk full (ENOSPC), or filesystem read-only.

Common situations: Read-only root filesystem in Kubernetes; data volume out of inodes or space; restrictive fsGroup/securityContext; data dir path changed without the thanos/ parent existing and being writable.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• create dir
• create
• clean upload directory
• create chunks dir
• create dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/5fa15ea7ecce9ff7. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:490

	return ret
}

// sync uploads the block if not exists in remote storage.
// TODO(khyatisoneji): Double check if block does not have deletion-mark.json for some reason, otherwise log it or return error.
func (s *Shipper) upload(ctx context.Context, meta *metadata.Meta) error {
	level.Info(s.logger).Log("msg", "upload new block", "id", meta.ULID)

	// We hard-link the files into a temporary upload directory so we are not affected
	// by other operations happening against the TSDB directory.
	updir := filepath.Join("thanos", "upload", meta.ULID.String())

	// Remove updir just in case.
	if err := s.dir.RemoveAll(updir); err != nil {
		return errors.Wrap(err, "clean upload directory")
	}
	if err := s.dir.MkdirAll(updir, 0750); err != nil {
		return errors.Wrap(err, "create upload dir")
	}
	defer func() {
		if err := s.dir.RemoveAll(updir); err != nil {
			level.Error(s.logger).Log("msg", "failed to clean upload directory", "err", err)
		}
	}()

	absUpdir := filepath.Join(s.dir.Name(), updir)
	dir := filepath.Join(s.dir.Name(), meta.ULID.String())
	if err := hardlinkBlock(dir, absUpdir); err != nil {
		return errors.Wrap(err, "hard link block")
	}
	// Attach current labels and write a new meta file with Thanos extensions.
	if lset := s.labels(); !lset.IsEmpty() {
		lset.Range(func(l labels.Label) {
			meta.Thanos.Labels[l.Name] = l.Value
		})
	}

View on GitHub (pinned to 35b8b99117)