thanos-io/thanos · error

flush

Error message

flush

What it means

Wraps the destination writer d.Flush() failing after series were written during rewrite. Flush finalizes the new block's index and chunk files and computes stats; failures indicate local disk problems (no space, I/O error) or an internal error building the index for the newly written data.

Solutions

1. Point --tmp.dir to a volume with enough free space for the rewritten block (approx. source block size).
2. Check wrapped cause for ENOSPC/EIO and fix the disk condition, then clean up partial new-ID dirs in tmpDir and retry.
3. Rewrite blocks in smaller batches to reduce peak disk usage.

Example fix

// before
thanos tools bucket rewrite --tmp.dir=/tmp ID
// after
thanos tools bucket rewrite --tmp.dir=/mnt/big-volume/thanos-rewrite ID

if free, _ := diskFree(tmpDir); free < requiredBytes {
    return fmt.Errorf("insufficient disk in %s", tmpDir)
}

meta.Stats, err = d.Flush()
if err != nil {
    return errors.Wrap(err, "flush: check tmp dir disk space and I/O errors")
}

When it happens

Trigger: tmpDir disk full while flushing the new block; underlying file I/O error; writing a large number of rewritten series exhausting resources; flush failing to build index segments from written chunks.

Common situations: Small ephemeral /tmp volume on Kubernetes nodes during a rewrite of large blocks; concurrent processes filling the same volume; memory pressure on large flush operations.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• create bucket compactor
• compaction
• first pass of downsampling failed
• second pass of downsampling failed
• critical error detected

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/e3d35802e4bb0107. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/tools_bucket.go:1313

					comp = compactv2.NewDryRun(tbc.tmpDir, logger, changeLog, chunkPool)
				} else {
					comp = compactv2.New(tbc.tmpDir, logger, changeLog, chunkPool)
				}

				level.Info(logger).Log("msg", "starting rewrite for block", "source", id, "new", newID, "toDelete", string(deletionsYaml), "toRelabel", string(relabelYaml))
				if err := comp.WriteSeries(ctx, []block.Reader{b}, d, p, modifiers...); err != nil {
					return errors.Wrapf(err, "writing series from %v to %v", id, newID)
				}

				if tbc.dryRun {
					level.Info(logger).Log("msg", "dry run finished. Changes should be printed to stderr", "Block ID", id)
					continue
				}

				level.Info(logger).Log("msg", "wrote new block after modifications; flushing", "source", id, "new", newID)
				meta.Stats, err = d.Flush()
				if err != nil {
					return errors.Wrap(err, "flush")
				}
				if err := meta.WriteToDir(logger, filepath.Join(tbc.tmpDir, newID.String())); err != nil {
					return err
				}

				level.Info(logger).Log("msg", "uploading new block", "source", id, "new", newID)
				if tbc.promBlocks {
					if err := block.UploadPromBlock(ctx, logger, insBkt, filepath.Join(tbc.tmpDir, newID.String()), metadata.HashFunc(*hashFunc)); err != nil {
						return errors.Wrap(err, "upload")
					}
				} else {
					if err := block.Upload(ctx, logger, insBkt, filepath.Join(tbc.tmpDir, newID.String()), metadata.HashFunc(*hashFunc)); err != nil {
						return errors.Wrap(err, "upload")
					}
				}
				level.Info(logger).Log("msg", "uploaded", "source", id, "new", newID)

				if !tbc.dryRun && tbc.deleteBlocks {

View on GitHub (pinned to 35b8b99117)