{"record":{"id":"8d86223916a0bb48","repo":"thanos-io/thanos","slug":"read-meta-from-s","errorCode":null,"errorMessage":"read meta from %s","messagePattern":"read meta from (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/compact.go","lineNumber":1136,"sourceCode":"\ttmpdir, err := os.MkdirTemp(\"\", fmt.Sprintf(\"repair-issue-347-id-%s-\", ie.id))\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tdefer func() {\n\t\tif err := os.RemoveAll(tmpdir); err != nil {\n\t\t\tlevel.Warn(logger).Log(\"msg\", \"failed to remote tmpdir\", \"err\", err, \"tmpdir\", tmpdir)\n\t\t}\n\t}()\n\n\tbdir := filepath.Join(tmpdir, ie.id.String())\n\tif err := block.Download(ctx, logger, bkt, ie.id, bdir); err != nil {\n\t\treturn retry(errors.Wrapf(err, \"download block %s\", ie.id))\n\t}\n\n\tmeta, err := metadata.ReadFromDir(bdir)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"read meta from %s\", bdir)\n\t}\n\n\tresid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"repair failed for block %s\", ie.id)\n\t}\n\n\t// Verify repaired id before uploading it.\n\tif err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {\n\t\treturn errors.Wrapf(err, \"repaired block is invalid %s\", resid)\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"uploading repaired block\", \"newID\", resid)\n\tif err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {\n\t\treturn retry(errors.Wrapf(err, \"upload of %s failed\", resid))\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"deleting broken block\", \"id\", ie.id)","sourceCodeStart":1118,"sourceCodeEnd":1154,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1118-L1154","documentation":"This error is wrapped around a failure to read a TSDB block's meta.json from the directory the compactor downloaded it into (bdir), during repair of an overlapping/issue-347 block inside Thanos' compact.Group repair flow (compact.go repairBucketBlock). The underlying error comes from metadata.ReadFromDir, which parses and validates meta.json (ULID, time range, compaction level, Thanos version, labels). The wrap preserves the block directory path so operators can locate the corrupted download.","triggerScenarios":"metadata.ReadFromDir(bdir) returns an error during repairBucketBlock after block.Download succeeds: meta.json missing from the downloaded block directory, meta.json unparseable JSON, meta.json failing validation (invalid ULID, minTime>maxTime, missing thanos.version field for Thanos-written blocks, mismatched block id between directory name and meta), or the download produced an empty/partial directory that still passed block.Download's checks.","commonSituations":"Object storage corruption or truncated uploads of meta.json; blocks uploaded by very old Prometheus/Thanos versions lacking required meta fields; partial block downloads from flaky object storage backends (S3 throttling, network interruption) where the index downloaded fine but meta.json did not; a bucket containing a directory that is not a real TSDB block (e.g. a marker file or partial upload from a crashed uploader).","solutions":["Inspect the block directory in the bucket (thanos tools bucket verify or ls) and check meta.json contents for corruption or missing fields","Delete or move the broken block out of the bucket so the compactor no longer tries to repair it (only if it is not referenced/needed), then let the compactor resync","Re-download the block manually (thanos tools bucket download) to rule out transient download corruption and confirm the object in storage itself is intact","Check the producing side: ensure Prometheus sidecar/uploaded blocks have valid, complete meta.json (upgrade Thanos/Prometheus if very old format)","If object storage returned partial data consistently, verify bucket permissions and that the bucket is not being concurrently modified by another compactor"],"exampleFix":"// before (manual diagnosis of the failing dir)\n// compactor logs: read meta from /var/thanos/compact/tmp/repair/01ARZ3N...\n\n// after: validate the block in the bucket before compaction runs\n$ thanos tools bucket verify --objstore.config-file=bucket.yaml\n$ thanos tools bucket inspect --objstore.config-file=bucket.yaml\n// if meta.json is truly corrupt, remove the broken block:\n$ thanos tools bucket webhooks... # or manually delete the block ULID dir in the bucket","handlingStrategy":"validation","validationCode":"// before running the compactor, verify every block in the bucket parses\nerr := bucket.Iterate(ctx, \"\", func(id string) error {\n    return nil // enumerate block ULIDs first\n})\nmeta, err := metadata.ReadFromDir(blockDir)\nif err != nil {\n    logger.Warn(\"skipping unreadable block\", \"dir\", blockDir, \"err\", err)\n}\n// preflight: thanos tools bucket verify --objstore.config-file=bucket.yaml","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run 'thanos tools bucket verify' periodically to catch corrupt meta.json before compaction","Keep Thanos/Prometheus versions current so uploaded meta.json includes all required fields","Avoid killing uploaders mid-write; use sidecar which uploads blocks atomically enough","Monitor compactor logs for download errors — a flaky storage backend corrupts downloads repeatedly"],"tags":["thanos","compactor","object-storage","metadata","corruption"],"backgroundTag":"file-read-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}