thanos-io/thanos · error

remove tombstones

Error message

remove tombstones

What it means

Wraps os.Remove of the tombstone file inside a freshly compacted result block. The compactor deletes TSDB-internal tombstones before validating the output block; if removal fails (permissions, unexpected file state) the result block cannot be verified and the compaction is retried.

Solutions

1. Check filesystem permissions on the block dir
2. Free disk/repair filesystem errors
3. Remove tombstones manually if the block is otherwise valid

When it happens

Trigger: Thrown at pkg/compact/compact.go:1306 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/557c40c424343dac. Report an issue: GitHub.

Appendix: source

Thrown at pkg/compact/compact.go:1306

	}
	cg.compactions.Inc()
	if overlappingBlocks {
		cg.verticalCompactions.Inc()
	}
	compIDStrings := make([]string, 0, len(compIDs))
	for _, compID := range compIDs {
		compIDStrings = append(compIDStrings, compID.String())
	}
	compIDStrs := fmt.Sprintf("%v", compIDStrings)
	level.Info(cg.logger).Log("msg", "compacted blocks", "new", compIDStrs,
		"duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds(), "overlapping_blocks", overlappingBlocks, "blocks", sourceBlockStr)

	for _, compID := range compIDs {
		bdir := filepath.Join(dir, compID.String())
		index := filepath.Join(bdir, block.IndexFilename)

		if err := os.Remove(filepath.Join(bdir, "tombstones")); err != nil {
			return false, nil, errors.Wrap(err, "remove tombstones")
		}

		newMeta, err := metadata.ReadFromDir(bdir)
		if err != nil {
			return false, nil, errors.Wrap(err, "read new meta")
		}

		var stats block.HealthStats
		// Ensure the output block is valid.
		err = tracing.DoInSpanWithErr(ctx, "compaction_verify_index", func(ctx context.Context) error {
			stats, err = block.GatherIndexHealthStats(ctx, cg.logger, index, newMeta.MinTime, newMeta.MaxTime)
			if err != nil {
				return err
			}
			return stats.AnyErr()
		})
		if !cg.acceptMalformedIndex && err != nil {
			return false, nil, halt(errors.Wrapf(err, "invalid result block %s", bdir))

View on GitHub (pinned to 35b8b99117)