{"record":{"id":"1528b04ef6dd1deb","repo":"thanos-io/thanos","slug":"repaired-block-is-invalid-s","errorCode":null,"errorMessage":"repaired block is invalid %s","messagePattern":"repaired block is invalid (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/compact.go","lineNumber":1146,"sourceCode":"\n\tbdir := filepath.Join(tmpdir, ie.id.String())\n\tif err := block.Download(ctx, logger, bkt, ie.id, bdir); err != nil {\n\t\treturn retry(errors.Wrapf(err, \"download block %s\", ie.id))\n\t}\n\n\tmeta, err := metadata.ReadFromDir(bdir)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"read meta from %s\", bdir)\n\t}\n\n\tresid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"repair failed for block %s\", ie.id)\n\t}\n\n\t// Verify repaired id before uploading it.\n\tif err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {\n\t\treturn errors.Wrapf(err, \"repaired block is invalid %s\", resid)\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"uploading repaired block\", \"newID\", resid)\n\tif err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {\n\t\treturn retry(errors.Wrapf(err, \"upload of %s failed\", resid))\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"deleting broken block\", \"id\", ie.id)\n\n\t// Spawn a new context so we always mark a block for deletion in full on shutdown.\n\tdelCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)\n\tdefer cancel()\n\n\t// TODO(bplotka): Issue with this will introduce overlap that will halt compactor. Automate that (fix duplicate overlaps caused by this).\n\tif err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, \"source of repaired block\", blocksMarkedForDeletion); err != nil {\n\t\treturn errors.Wrapf(err, \"marking old block %s for deletion has failed\", ie.id)\n\t}\n\treturn nil","sourceCodeStart":1128,"sourceCodeEnd":1164,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1128-L1164","documentation":"This error wraps a failure from block.VerifyIndex after the repair step produced a new block (resid) in tmpdir (compact.go repairBucketBlock). VerifyIndex opens the repaired block's index and checks postings/series against the expected [meta.MinTime, meta.MaxTime] window. If the freshly repaired block does not pass index validation, the compactor refuses to upload it, preventing corrupt data from entering the bucket.","triggerScenarios":"block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime) returns error: the repaired index has postings outside or inconsistent with the original block's minTime/maxTime, the index file is missing at tmpdir/<resid>/index, index binary format is unreadable, or the repair rewrote series incorrectly producing lookup failures.","commonSituations":"Repair produced a block whose time bounds shifted relative to the source meta.json; disk corruption or truncation in tmpdir; a Thanos bug in the issue-347 repair rewriting chunks producing an inconsistent index; running an older Thanos compactor repairing blocks written by newer Prometheus with index format changes.","solutions":["Read the inner VerifyIndex error: it names the failing index component (postings, symbols, series) and ULID range mismatch","Re-run the repair with free disk space and a clean tmpdir (stale tmp files can produce partial blocks)","Upgrade Thanos to the latest patch version; several repair/VerifyIndex bugs were fixed over releases","If repair repeatedly fails verification, delete the broken source block from the bucket (it is unusable anyway) rather than retrying indefinitely","Verify the source block with 'thanos tools bucket verify' to confirm whether the input was already unrecoverable"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// sanity-check the repaired index before verification-time surprises\nif _, err := os.Stat(filepath.Join(tmpdir, resid.String(), block.IndexFilename)); err != nil {\n    return fmt.Errorf(\"repaired index missing: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := block.VerifyIndex(ctx, logger, idxPath, meta.MinTime, meta.MaxTime); err != nil {\n    logger.Error(\"repaired block failed verification\", \"newID\", resid, \"err\", err)\n    os.RemoveAll(filepath.Join(tmpdir, resid.String())) // clean partial repair output\n    return errors.Wrapf(err, \"repaired block is invalid %s\", resid)\n}","preventionTips":["Keep tmpdir clean between compactor restarts (stale partial repairs fail verification)","Run the latest Thanos patch release; index verification failures in repair were fixed over time","Verify repaired blocks also on compactor restart — never upload unverified data","Check Thanos version compatibility with the Prometheus index format producing the blocks"],"tags":["thanos","compactor","verification","index","corruption"],"backgroundTag":"schema-validation-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}