{"record":{"id":"5fa15ea7ecce9ff7","repo":"thanos-io/thanos","slug":"create-upload-dir","errorCode":null,"errorMessage":"create upload dir","messagePattern":"create upload dir","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":490,"sourceCode":"\n\treturn ret\n}\n\n// sync uploads the block if not exists in remote storage.\n// TODO(khyatisoneji): Double check if block does not have deletion-mark.json for some reason, otherwise log it or return error.\nfunc (s *Shipper) upload(ctx context.Context, meta *metadata.Meta) error {\n\tlevel.Info(s.logger).Log(\"msg\", \"upload new block\", \"id\", meta.ULID)\n\n\t// We hard-link the files into a temporary upload directory so we are not affected\n\t// by other operations happening against the TSDB directory.\n\tupdir := filepath.Join(\"thanos\", \"upload\", meta.ULID.String())\n\n\t// Remove updir just in case.\n\tif err := s.dir.RemoveAll(updir); err != nil {\n\t\treturn errors.Wrap(err, \"clean upload directory\")\n\t}\n\tif err := s.dir.MkdirAll(updir, 0750); err != nil {\n\t\treturn errors.Wrap(err, \"create upload dir\")\n\t}\n\tdefer func() {\n\t\tif err := s.dir.RemoveAll(updir); err != nil {\n\t\t\tlevel.Error(s.logger).Log(\"msg\", \"failed to clean upload directory\", \"err\", err)\n\t\t}\n\t}()\n\n\tabsUpdir := filepath.Join(s.dir.Name(), updir)\n\tdir := filepath.Join(s.dir.Name(), meta.ULID.String())\n\tif err := hardlinkBlock(dir, absUpdir); err != nil {\n\t\treturn errors.Wrap(err, \"hard link block\")\n\t}\n\t// Attach current labels and write a new meta file with Thanos extensions.\n\tif lset := s.labels(); !lset.IsEmpty() {\n\t\tlset.Range(func(l labels.Label) {\n\t\t\tmeta.Thanos.Labels[l.Name] = l.Value\n\t\t})\n\t}","sourceCodeStart":472,"sourceCodeEnd":508,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L472-L508","documentation":"Wraps an error from s.dir.MkdirAll(updir, 0750) in upload(): after cleaning, the shipper creates the temporary thanos/upload/<ULID> directory to hard-link block files into. Failure means the upload of that block cannot proceed.","triggerScenarios":"Calling Sync → upload when creating thanos/upload/<ULID> fails: parent dir missing/unwritable, permission denied, disk full (ENOSPC), or filesystem read-only.","commonSituations":"Read-only root filesystem in Kubernetes; data volume out of inodes or space; restrictive fsGroup/securityContext; data dir path changed without the thanos/ parent existing and being writable.","solutions":["Check the wrapped cause; fix disk space/inodes (df -h /data; df -i /data).","Grant the sidecar write permission on <data-dir>/thanos (chown/chmod or K8s fsGroup).","Ensure the volume is mounted read-write (remove :ro mount flag).","Verify the data directory path in --data-dir points to a writable filesystem."],"exampleFix":"// before\n// create upload dir: mkdir /data/thanos/upload/01ARZ...: read-only file system\n// after (kubernetes)\n// volumes:\n//   - name: data\n//     emptyDir: {}        # or remove readOnly: true from the PVC mount\n// securityContext:\n//   fsGroup: 65534","handlingStrategy":"validation","validationCode":"thanosDir := filepath.Join(dataDir, \"thanos\")\nif err := os.MkdirAll(thanosDir, 0750); err != nil {\n    return fmt.Errorf(\"data dir not writable for shipper: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"create upload dir\") {\n    switch {\n    case strings.Contains(err.Error(), \"ENOSPC\"):\n        log.Printf(\"disk full — free space before next sync\")\n    case strings.Contains(err.Error(), \"permission denied\", \"read-only\"):\n        log.Printf(\"fix data-dir permissions/mount: %v\", err)\n    }\n}","preventionTips":["Preflight-check write access to <data-dir>/thanos in the sidecar's readiness probe","Monitor disk space and inode usage on the data volume","Set K8s securityContext fsGroup so the volume is group-writable","Never mount the data volume read-only when shipping is enabled"],"tags":["filesystem","mkdir","permissions","upload"],"backgroundTag":"mkdir-permission-denied","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}