{"record":{"id":"19fcdd680de89988","repo":"thanos-io/thanos","slug":"clean-upload-directory","errorCode":null,"errorMessage":"clean upload directory","messagePattern":"clean upload directory","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":487,"sourceCode":"\tfor _, id := range meta.Uploaded {\n\t\tret[id] = struct{}{}\n\t}\n\n\treturn ret\n}\n\n// sync uploads the block if not exists in remote storage.\n// TODO(khyatisoneji): Double check if block does not have deletion-mark.json for some reason, otherwise log it or return error.\nfunc (s *Shipper) upload(ctx context.Context, meta *metadata.Meta) error {\n\tlevel.Info(s.logger).Log(\"msg\", \"upload new block\", \"id\", meta.ULID)\n\n\t// We hard-link the files into a temporary upload directory so we are not affected\n\t// by other operations happening against the TSDB directory.\n\tupdir := filepath.Join(\"thanos\", \"upload\", meta.ULID.String())\n\n\t// Remove updir just in case.\n\tif err := s.dir.RemoveAll(updir); err != nil {\n\t\treturn errors.Wrap(err, \"clean upload directory\")\n\t}\n\tif err := s.dir.MkdirAll(updir, 0750); err != nil {\n\t\treturn errors.Wrap(err, \"create upload dir\")\n\t}\n\tdefer func() {\n\t\tif err := s.dir.RemoveAll(updir); err != nil {\n\t\t\tlevel.Error(s.logger).Log(\"msg\", \"failed to clean upload directory\", \"err\", err)\n\t\t}\n\t}()\n\n\tabsUpdir := filepath.Join(s.dir.Name(), updir)\n\tdir := filepath.Join(s.dir.Name(), meta.ULID.String())\n\tif err := hardlinkBlock(dir, absUpdir); err != nil {\n\t\treturn errors.Wrap(err, \"hard link block\")\n\t}\n\t// Attach current labels and write a new meta file with Thanos extensions.\n\tif lset := s.labels(); !lset.IsEmpty() {\n\t\tlset.Range(func(l labels.Label) {","sourceCodeStart":469,"sourceCodeEnd":505,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L469-L505","documentation":"Wraps an error from s.dir.RemoveAll(updir) in upload(): before hard-linking block files into the temporary thanos/upload/<ULID> directory, the shipper removes any stale copy. Failure to remove it (permissions, read-only fs, EBUSY on a mount) aborts the upload of that block.","triggerScenarios":"Calling Sync → upload when the directory thanos/upload/<ULID> exists and cannot be deleted — e.g. permissions differ (previous run as root), filesystem is read-only, or a stale bind mount holds it.","commonSituations":"Sidecar previously run as a different user leaving root-owned upload dirs; the data dir shared as a Kubernetes volume with wrong fsGroup; disk remounted read-only after errors.","solutions":["Delete thanos/upload/<ULID> manually with sufficient privileges and re-run Sync.","Chown the data dir to the sidecar's runtime user (or fix securityContext/fsGroup in Kubernetes).","Check that the filesystem is mounted read-write (mount | grep, dmesg for I/O errors).","Ensure no other process (backup agent, old sidecar) holds files open inside the upload dir."],"exampleFix":"// before\n// clean upload directory: remove /data/thanos/upload/01ARZ...: permission denied\n// after\n// $ docker run --rm -v /data:/data busybox rm -rf /data/thanos/upload\n// $ chown -R 65534:65534 /data/thanos","handlingStrategy":"validation","validationCode":"updir := filepath.Join(dataDir, \"thanos\", \"upload\")\nif info, err := os.Stat(updir); err == nil && !info.IsDir() {\n    return fmt.Errorf(\"%s exists and is not a directory\", updir)\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"clean upload directory\") {\n    // stale root-owned upload dir; run one-time cleanup with elevated perms then restart\n    log.Printf(\"requires manual cleanup of thanos/upload: %v\", err)\n}","preventionTips":["Run the sidecar with a consistent UID across restarts/upgrades","Add a startup pre-clean of <data-dir>/thanos/upload","Avoid sharing the data dir with processes that lock files (backup agents)"],"tags":["filesystem","permissions","upload"],"backgroundTag":"file-write-permission-denied","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}