{"record":{"id":"cd2d08f84987bf86","repo":"thanos-io/thanos","slug":"write-meta-file","errorCode":null,"errorMessage":"write meta file","messagePattern":"write meta file","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":512,"sourceCode":"\t\t\tlevel.Error(s.logger).Log(\"msg\", \"failed to clean upload directory\", \"err\", err)\n\t\t}\n\t}()\n\n\tabsUpdir := filepath.Join(s.dir.Name(), updir)\n\tdir := filepath.Join(s.dir.Name(), meta.ULID.String())\n\tif err := hardlinkBlock(dir, absUpdir); err != nil {\n\t\treturn errors.Wrap(err, \"hard link block\")\n\t}\n\t// Attach current labels and write a new meta file with Thanos extensions.\n\tif lset := s.labels(); !lset.IsEmpty() {\n\t\tlset.Range(func(l labels.Label) {\n\t\t\tmeta.Thanos.Labels[l.Name] = l.Value\n\t\t})\n\t}\n\tmeta.Thanos.Source = s.source\n\tmeta.Thanos.SegmentFiles = block.GetSegmentFiles(absUpdir)\n\tif err := meta.WriteToDir(s.logger, absUpdir); err != nil {\n\t\treturn errors.Wrap(err, \"write meta file\")\n\t}\n\tvar uploadOptions []objstore.UploadOption\n\tif s.uploadConcurrency > 0 {\n\t\tuploadOptions = append(uploadOptions, objstore.WithUploadConcurrency(s.uploadConcurrency))\n\t}\n\treturn block.Upload(ctx, s.logger, s.bucket, absUpdir, s.hashFunc, uploadOptions...)\n}\n\n// blockMetasFromOldest returns the block meta of each block found in dir\n// sorted by minTime asc.\nfunc (s *Shipper) blockMetasFromOldest() (metas []*metadata.Meta, failedBlocks []string, _ error) {\n\n\tdir, err := s.dir.Open(\".\")\n\tif err != nil {\n\t\treturn nil, nil, errors.Wrap(err, \"open dir\")\n\t}\n\tdefer runutil.CloseWithLogOnErr(s.logger, dir, \"shipper dir\")\n","sourceCodeStart":494,"sourceCodeEnd":530,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L494-L530","documentation":"This error wraps a failure from metadata.Meta.WriteToDir, which serializes the block meta (with Thanos extensions: labels, source, segment files) as meta.json into the block directory before upload. It is thrown when writing/flushing meta.json to the block dir fails, typically due to filesystem permission, disk-full, or directory-state issues.","triggerScenarios":"Calling Shipper.Sync when the block directory (hardlinked ULID dir) is not writable by the process, the filesystem is out of space or read-only, or the directory disappeared between hardlink and meta write (concurrent deletion).","commonSituations":"Read-only volume after pod rescheduling; disk-full on the Prometheus data volume; running the process as non-root against a data dir owned by another user; antivirus/backup tooling locking files on the volume.","solutions":["Check free disk space on the data volume (df -h) and free space or grow the volume.","Fix permissions/ownership of the blocks directory so the shipping process can write meta.json.","Ensure the volume is mounted read-write (check /proc/mounts for 'ro').","Verify the block directory still exists (no concurrent cleanup); re-run Sync after the block is restored."],"exampleFix":"// before: container runs as uid 65534 against root-owned data\n$ kubectl exec sidecar -- ls -ld /prometheus\n// after: set matching securityContext\nsecurityContext:\n  runAsUser: 0        # or chown data dir to the sidecar uid\n  fsGroup: 2000","handlingStrategy":"validation","validationCode":"info, err := os.Stat(absUpdir)\nif err != nil {\n    return fmt.Errorf(\"block dir missing: %w\", err)\n}\nif !info.IsDir() {\n    return errors.New(\"block path is not a directory\")\n}\n// probe writability\nprobe := filepath.Join(absUpdir, \".write-test\")\nif err := os.WriteFile(probe, nil, 0o600); err != nil {\n    return fmt.Errorf(\"block dir not writable: %w\", err)\n}\nos.Remove(probe)","typeGuard":"func dirWritable(path string) bool {\n    probe := filepath.Join(path, \".writetest\")\n    if err := os.WriteFile(probe, nil, 0o600); err != nil {\n        return false\n    }\n    os.Remove(probe)\n    return true\n}","tryCatchPattern":"if err := meta.WriteToDir(s.logger, absUpdir); err != nil {\n    if errors.Is(err, syscall.ENOSPC) {\n        return errors.Wrap(err, \"write meta file: disk full, free space and retry\")\n    }\n    if os.IsPermission(err) {\n        return errors.Wrap(err, \"write meta file: check volume permissions/ownership\")\n    }\n    return errors.Wrap(err, \"write meta file\")\n}","preventionTips":["Alert on disk usage of the Prometheus data volume before it fills.","Run the shipping process with the same uid/gid that owns the TSDB dir (or set fsGroup in Kubernetes).","Ensure volumes are mounted read-write; ro mounts cause immediate write failures.","Avoid concurrent cleanup jobs that could delete block dirs mid-write."],"tags":["go","filesystem","meta-json","shipper","disk"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}