thanos-io/thanos · error

write chunks

Error message

write chunks

What it means

During rewrite(), after collecting and sorting series, each series' chunks are written to the new index file with indexw.WriteChunks. Failure here is wrapped with "write chunks". This means writing chunk metadata (chunk metas, offsets, CRCs) into the new TSDB index failed.

Solutions

1. Check the wrapped cause: if it is a disk/space or permission error, free space or fix permissions on the tmp repair directory and rerun the repair
2. Verify the source block for corruption (`thanos tools bucket verify`); if chunks themselves are corrupt, delete the block so compactor repairs or re-syncs it
3. Re-download the block from object storage to get an intact copy before rewriting
4. Retry the repair operation after transient I/O failures

func ensureDiskSpace(dir string, minBytes uint64) error {
	var st syscall.Statfs_t
	if err := syscall.Statfs(dir, &st); err != nil { return err }
	if uint64(st.Bavail)*uint64(st.Bsize) < minBytes {
		return errors.Errorf("insufficient space in %s", dir)
	}
	return nil
}

if err := rewrite(...); err != nil {
	if errors.Is(errors.Cause(err), syscall.ENOSPC) {
		// free space / use another volume, then retry
	} else if isPermErr(errors.Cause(err)) {
		// fix tmp dir permissions
	}
	return err
}

When it happens

Trigger: chunkw.WriteChunks(s.chks...) returns an error during rewrite/Repair — typically because the underlying index writer hit an I/O error (disk full, permission denied on tmp dir), or a chunk is malformed/overlapping so the TSDB index writer rejects it.

Common situations: Disk full on the volume holding the repair tmp directory when running compactor repair; corrupted chunk metadata carried over from a damaged source index causing writer validation failures; filesystem permission problems in the block directory.

Related errors

• add series
• iterate series
• start remote write agent db
• open TSDB
• rewrite configuration should be provided

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2831b681225d6f44. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/index.go:670

	lastSet := labels.Labels{}
	// Build a new TSDB block.
	for _, s := range series {
		// The TSDB library will throw an error if we add a series with
		// identical labels as the last series. This means that we have
		// discovered a duplicate time series in the old block. We drop
		// all duplicate series preserving the first one.
		// TODO: Add metric to count dropped series if repair becomes a daemon
		// rather than a batch job.
		if labels.Compare(lastSet, s.lset) == 0 {
			level.Warn(logger).Log("msg",
				"dropping duplicate series in tsdb block found",
				"labelset", s.lset.String(),
			)
			continue
		}
		if err := chunkw.WriteChunks(s.chks...); err != nil {
			return errors.Wrap(err, "write chunks")
		}
		if err := indexw.AddSeries(i, s.lset, s.chks...); err != nil {
			return errors.Wrap(err, "add series")
		}

		meta.Stats.NumChunks += uint64(len(s.chks))
		meta.Stats.NumSeries++

		for _, chk := range s.chks {
			meta.Stats.NumSamples += uint64(chk.Chunk.NumSamples())
		}

		s.lset.Range(func(l labels.Label) {
			valset, ok := values[l.Name]
			if !ok {
				valset = stringset{}
				values[l.Name] = valset
			}

View on GitHub (pinned to 35b8b99117)