What causes "invalid magic number %x" in thanos-io/thanos?

Delete the cached file at that path and let the library regenerate it from the bucket.

invalid magic number

Error message

invalid magic number %x

What it means

While validating the binary index-header header, the first 4 bytes must equal MagicIndex. This error is raised when the magic number read from the buffer does not match, meaning the data at the cache path is not a Thanos binary index-header of the expected format.

Solutions

Delete the cached file at that path and let the library regenerate it from the bucket.
Verify you are reading the intended .index-header file, not the full index or another file.
Check Thanos version compatibility between writer and reader of the cache file.
If corruption recurs on the same volume, check disk health.

Example fix

null

Defensive patterns

Strategy: validation

Validate before calling

b, err := os.ReadFile(hdrPath)
if err == nil && len(b) >= 4 && binary.BigEndian.Uint32(b[:4]) != MagicIndex {
    os.Remove(hdrPath) // wrong file or corruption; regenerate
}

Type guard

func hasIndexMagic(b []byte) bool {
    return len(b) >= 4 && binary.BigEndian.Uint32(b[:4]) == MagicIndex
}

Try / catch

r, err := NewBinaryReader(ctx, logger, bkt, dir, id, sampling, metrics)
if err != nil && strings.Contains(err.Error(), "invalid magic number") {
    os.Remove(filepath.Join(dir, id.String()+".index-header"))
    return NewBinaryReader(ctx, logger, bkt, dir, id, sampling, metrics)
}

Prevention

Never write other files to the cache path of an index-header.
Keep consistent Thanos versions across writers and readers of the cache.
Delete unrecognizable cache files instead of failing queries.
Check memory/disk corruption if magic bytes flip randomly.

When it happens

Trigger: newBinaryReader/validate on a buffer whose first 4 bytes differ from MagicIndex: the cache file is some other file (e.g., the raw index), the file was written by a different/older format, or the bytes are scrambled garbage.

Common situations: Cache path pointing to the wrong file; manually copied files; files produced by an incompatible Thanos/Thanos-fork version; bit-rot corruption of the first bytes.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/7cd3841798b8df6b. Report an issue: GitHub.

Appendix: source

Thrown at pkg/block/indexheader/binary_reader.go:674

	return &BinaryTOC{
		Symbols:             d.Be64(),
		PostingsOffsetTable: d.Be64(),
	}, nil
}

func (r *BinaryReader) init() (err error) {
	start := time.Now()

	defer func() {
		r.metrics.loadDuration.Observe(time.Since(start).Seconds())
	}()
	// Verify header.
	if r.b.Len() < headerLen {
		return errors.Wrap(encoding.ErrInvalidSize, "index header's header")
	}
	if m := binary.BigEndian.Uint32(r.b.Range(0, 4)); m != MagicIndex {
		return errors.Errorf("invalid magic number %x", m)
	}
	r.version = int(r.b.Range(4, 5)[0])
	r.indexVersion = int(r.b.Range(5, 6)[0])

	r.indexLastPostingEnd = int64(binary.BigEndian.Uint64(r.b.Range(6, headerLen)))

	if r.version != BinaryFormatV1 {
		return errors.Errorf("unknown index header file version %d", r.version)
	}

	r.toc, err = newBinaryTOCFromByteSlice(r.b)
	if err != nil {
		return errors.Wrap(err, "read index header TOC")
	}

	// TODO(bwplotka): Consider contributing to Prometheus to allow specifying custom number for symbolsFactor.
	r.symbols, err = index.NewSymbols(r.b, r.indexVersion, int(r.toc.Symbols))
	if err != nil {

View on GitHub (pinned to 35b8b99117)