thanos-io/thanos · error

sync

Error message

sync

What it means

After flushing, WriteBinary calls bw.writer.Sync() (fsync) to durably persist the index-header bytes before the atomic rename into place. This error wraps an fsync failure: the OS could not flush dirty pages of the file to stable storage.

Solutions

1. Check dmesg for fsync/EIO errors on the device and run a filesystem check.
2. Verify free disk space (fsync can fail with ENOSPC when metadata needs flushing).
3. Confirm the volume is still mounted read-write.
4. Retry after storage recovers; the temp-file + rename design means no partial index-header was published, so retrying is safe.

Example fix

// before
if err := bw.writer.Sync(); err != nil {
    return nil, errors.Wrap(err, "sync")
}
// after
if err := bw.writer.Sync(); err != nil {
    os.Remove(tmpFilename)
    return nil, errors.Wrapf(err, "fsync index-header %s (check disk health/space)", tmpFilename)
}

// Go: check device is writable and healthy before fsync-heavy batch work
func deviceHealthy(dir string) error {
    var st syscall.Statfs_t
    if err := syscall.Statfs(dir, &st); err != nil { return err }
    if st.Flags&syscall.ST_RDONLY != 0 { return errors.New("filesystem mounted read-only") }
    return nil
}

// Go
if err := buildIndexHeader(); err != nil {
    var se syscall.Errno
    if errors.As(err, &se) && (se == syscall.EIO || se == syscall.ENOSPC || se == syscall.EROFS) {
        log.Error("fsync failed; storage problem", "errno", se)
        // page node / alert, do not hot-retry
        return err
    }
    return retryWriteBinary()
}

When it happens

Trigger: WriteBinary -> bw.writer.Sync() returning an error such as EIO, ENOSPC, or EROFS from fsync on the temp index-header file, right before os.Rename(tmpFilename, filename).

Common situations: Underlying disk failure or full device surfaced only at fsync; Kubernetes nodes with failing disks; virtualized/cloud storage with intermittent EIO; read-only remount after filesystem error.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• sync dir
• read meta file
• open index file
• add index meta
• flush

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/2cb7d3f7a0d44914. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/indexheader/binary_reader.go:162

	if err := ir.CopyPostingsOffsets(bw.PostingOffsetsWriter(), buf); err != nil {
		return nil, err
	}

	if err := bw.writer.Flush(); err != nil {
		return nil, errors.Wrap(err, "flush")
	}

	if err := bw.WriteTOC(); err != nil {
		return nil, errors.Wrap(err, "write index header TOC")
	}

	if err := bw.writer.Flush(); err != nil {
		return nil, errors.Wrap(err, "flush")
	}

	if err := bw.writer.Sync(); err != nil {
		return nil, errors.Wrap(err, "sync")
	}

	if tmpFilename != "" {
		// Create index-header in atomic way, to avoid partial writes (e.g during restart or crash of store GW).
		return nil, os.Rename(tmpFilename, filename)
	}

	return bw.Buffer(), nil
}

type chunkedIndexReader struct {
	ctx  context.Context
	path string
	size uint64
	bkt  objstore.BucketReader
	toc  *index.TOC
}

View on GitHub (pinned to 35b8b99117)