thanos-io/thanos · error

existing tsdb block is invalid

Error message

existing tsdb block is invalid

What it means

backupDownloaded validates the on-disk TSDB block by stat-ing its meta.json before uploading it to the backup bucket. If os.Stat fails for any reason, the block is declared invalid and the upload is refused, wrapping the stat error with this message.

Solutions

1. Ensure the block directory contains a readable meta.json (ls the dir, check permissions)
2. Re-download/refresh the block to local disk before repairing
3. Verify the bdir path passed to the repair command is correct
4. Inspect the wrapped os.Stat error for the exact cause

Example fix

// before
// proceeding to upload without checking meta.json
// after
if _, err := os.Stat(filepath.Join(bdir, "meta.json")); err != nil {
	return errors.Wrap(err, "existing tsdb block is invalid")
}

if _, err := os.Stat(filepath.Join(bdir, "meta.json")); err != nil {
	// re-download block or fix bdir before calling backupDownloaded
}

When it happens

Trigger: BackupAndDelete/BackupAndDeleteDownloaded call backupDownloaded with a block directory (bdir) where meta.json is missing, unreadable, or the path is wrong/inaccessible.

Common situations: Incomplete or corrupted block download on local disk; wrong --data-dir path; permissions issues on the local filesystem; partially cleaned-up block directory.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• write meta file
• create meta fetcher
• create working compact directory
• create working downsample directory
• create dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/4e68f979d4acd8f5. Report an issue: GitHub.

Appendix: source

Thrown at pkg/verifier/safe_delete.go:136

	}

	level.Info(ctx.Logger).Log("msg", "Marking block as deleted", "id", id.String())
	if err := block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, "manual verify-repair", ctx.metrics.blocksMarkedForDeletion); err != nil {
		return errors.Wrap(err, "marking delete from source")
	}
	return nil
}

// backupDownloaded is a helper function that uploads a TSDB block
// found on disk to the given bucket. An error is returned if any operation
// fails.
func backupDownloaded(ctx context.Context, logger log.Logger, bdir string, backupBkt objstore.Bucket, id ulid.ULID) error {
	// Safety checks.
	if _, err := os.Stat(filepath.Join(bdir, "meta.json")); err != nil {
		// If there is any error stat'ing meta.json inside the TSDB block
		// then declare the existing block as bad and refuse to upload it.
		// TODO: Make this check more robust.
		return errors.Wrap(err, "existing tsdb block is invalid")
	}

	// Upload the on disk TSDB block.
	level.Info(logger).Log("msg", "Uploading block to backup bucket", "id", id.String())
	if err := block.Upload(ctx, logger, backupBkt, bdir, metadata.NoneFunc); err != nil {
		return errors.Wrap(err, "upload to backup")
	}

	return nil
}

View on GitHub (pinned to 35b8b99117)