thanos-io/thanos · error

iterate series

Error message

iterate series

What it means

In Thanos, pkg/block/index.go rewrite() builds a new block index by iterating over all series of the source block via a series set. This error wraps any failure from iterating the series set (decoding index entries, symbol lookups, chunk metadata reads) so the caller knows the new index could not be produced because the source series could not be read.

Solutions

1. Check the wrapped cause (errors.Cause) for the real I/O or corruption error and confirm the source block's index.bin integrity
2. Re-download or re-upload the block from a good replica, or delete the corrupted block from the bucket (Thanos compactor will then repair/remove it)
3. Run `thanos tools bucket verify` on the block to confirm corruption before rewriting
4. Retry the rewrite after fixing disk/filesystem-level I/O errors

Example fix

// before
ir, err := rewrite(ctx, bkt, id, logger)
if err != nil { return err }
// after
if err := block.VerifyIndex(ctx, bkt, id, meta.MinTime, meta.MaxTime); err != nil {
	logger.Warn("index corrupt, re-downloading block", "err", err)
	if err := redownloadBlock(ctx, bkt, id); err != nil { return err }
}
if _, err := rewrite(ctx, bkt, id, logger); err != nil {
	return errors.Wrapf(err, "rewrite block %s", id)
}

meta, err := block.DownloadMeta(ctx, bkt, id)
if err != nil { return err }
if err := block.VerifyIndex(ctx, bkt, id, meta.MinTime, meta.MaxTime); err != nil {
	return errors.Wrapf(err, "block %s index invalid, skipping rewrite", id)
}

func blockReadable(ctx context.Context, bkt objstore.Bucket, id ulid.ULID) error {
	rc, err := bkt.Get(ctx, path.Join(id.String(), block.IndexFilename))
	if err != nil { return err }
	defer rc.Close()
	_, err = io.Copy(io.Discard, io.LimitReader(rc, 1))
	return err
}

ir, err := rewrite(ctx, bkt, id)
if err != nil {
	cause := errors.Cause(err)
	if strings.Contains(cause.Error(), "corrupted index") || objstore.IsNotFoundError(err) {
		return handleCorruptBlock(ctx, bkt, id) // mark/delete and re-sync
	}
	return errors.Wrapf(err, "rewrite block %s", id)
}

When it happens

Trigger: rewrite() is called (directly or via Repair after verification failures); the underlying seriesSet.Err() is non-nil after full iteration — e.g. corrupted index file in the source block directory, truncated or partially downloaded block, or an I/O error while reading the index while dumping all series.

Common situations: Running `thanos tools bucket verify --repair` on a block whose index was corrupted by a crashed Prometheus or an interrupted upload; snapshotting/copying block directories mid-compaction so index.bin is truncated; disk read errors on object storage download.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

• add series
• open block
• open block
• open chunks
• next symbol

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/14b2b2f195321abd. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:644

		}

		chks, err := sanitizeChunkSequence(chks, meta.MinTime, meta.MaxTime, ignoreChkFns)
		if err != nil {
			return err
		}

		if len(chks) == 0 {
			continue
		}

		series = append(series, seriesRepair{
			lset: builder.Labels(),
			chks: chks,
		})
	}

	if all.Err() != nil {
		return errors.Wrap(all.Err(), "iterate series")
	}

	// Sort the series, if labels are re-ordered then the ordering of series
	// will be different.
	sort.Slice(series, func(i, j int) bool {
		return labels.Compare(series[i].lset, series[j].lset) < 0
	})

	lastSet := labels.Labels{}
	// Build a new TSDB block.
	for _, s := range series {
		// The TSDB library will throw an error if we add a series with
		// identical labels as the last series. This means that we have
		// discovered a duplicate time series in the old block. We drop
		// all duplicate series preserving the first one.
		// TODO: Add metric to count dropped series if repair becomes a daemon
		// rather than a batch job.
		if labels.Compare(lastSet, s.lset) == 0 {

View on GitHub (pinned to 35b8b99117)