thanos-io/thanos · error

new bucket block

Error message

new bucket block

What it means

Thrown in pkg/store/bucket.go when BucketStore.loadBlock fails to construct a new bucket block (Block.NewBucketBlock) from the object-store bucket. It wraps any failure while opening the block's index, chunks, tombstones, or index-header during block sync/open. The deferred handler closes the partially-opened block so resources are not leaked.

Solutions

1. Verify the block in the bucket is complete: check for meta.json, index, and chunks files; re-upload or remove the corrupted block.
2. Rebuild or delete the stale index-header file (thanos tools bucket verify / remove the .index-header and let the store rebuild it).
3. Check object-store credentials, permissions, and rate limits in the store gateway logs for the wrapped root error.
4. Restart the store gateway to re-run the block sync after fixing the bucket contents.

Example fix

// before: block partially uploaded, index missing
// after: ensure block is uploaded atomically
func uploadBlock(bkt objstore.Bucket, dir string) error {
	return objstore.UploadDir(context.Background(), bkt, dir, dir) // uploads meta.json last
}

// Before opening: confirm block completeness
for _, f := range []string{"meta.json", "index"} {
	if ok, _ := bkt.Exists(ctx, path.Join(ulid.String(), f)); !ok {
		return fmt.Errorf("block %s incomplete: missing %s", ulid, f)
	}
}

if err := store.SyncBlocks(ctx); err != nil {
	var blockErr error
	if errors.As(err, &blockErr) && strings.Contains(err.Error(), "new bucket block") {
		// mark block ID as suspect and skip it this sync cycle
		logger.Warn("skipping unloadable block", "err", err)
	} else {
		return err
	}
}

When it happens

Trigger: Block discovery finds a meta.json ULID in the bucket, and NewBucketBlock returns an error: the block directory cannot be downloaded/opened, index-header cannot be built (corrupt or missing index-header file when lazy index-header is enabled), or the underlying object storage call fails.

Common situations: Corrupted or truncated block uploads (e.g. compactor died mid-upload), missing index-header with index-header lazy downloading enabled, object-store permission or throttling errors during sync, blocks removed from the bucket while the store is syncing them.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• meta.json not found
• add block to set
• close block
• load chunks
• create meta fetcher

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/445b83c63cc757fb. Report an issue: GitHub.

Appendix: source

Thrown at pkg/store/bucket.go:896

			runutil.CloseWithErrCapture(&err, indexHeaderReader, "index-header")
		}
	}()

	b, err := newBucketBlock(
		ctx,
		s.metrics,
		meta,
		s.bkt,
		dir,
		s.indexCache,
		s.chunkPool,
		indexHeaderReader,
		s.partitioner,
		s.blockEstimatedMaxSeriesFunc,
		s.blockEstimatedMaxChunkFunc,
	)
	if err != nil {
		return errors.Wrap(err, "new bucket block")
	}
	defer func() {
		if err != nil {
			runutil.CloseWithErrCapture(&err, b, "index-header")
		}
	}()

	s.mtx.Lock()
	defer s.mtx.Unlock()

	set, ok := s.blockSets[h]
	if !ok {
		set = newBucketBlockSet(lset)
		s.blockSets[h] = set
	}

	if err = set.add(b); err != nil {
		return errors.Wrap(err, "add block to set")

View on GitHub (pinned to 35b8b99117)