weaviate/weaviate · error

parse header: %w

Error message

parse header: %w

What it means

Thrown in newSegment (segment.go:316) when segmentindex.ParseHeader fails on the first HeaderSize bytes of the segment contents. ParseHeader validates the magic/version framing, so failure means the bytes do not form a valid LSM segment header.

Source

Thrown at adapters/repos/db/lsmkv/segment.go:316

		if err != nil {
			return nil, fmt.Errorf("mmap file: %w", err)
		}
		contents = contents2
		unMapContents = true
	} else { // read the file into memory if it's small enough and we have enough memory
		meteredF := diskio.NewMeteredReader(file, diskio.MeteredReaderCallback(metrics.ReadObserver("readSegmentFile")))
		bufio.NewReader(meteredF)
		contents, err = io.ReadAll(meteredF)
		if err != nil {
			return nil, fmt.Errorf("read file: %w", err)
		}
		unMapContents = false
		readFromMemory = true
		useBloomFilter = false
	}
	header, err := segmentindex.ParseHeader(contents[:segmentindex.HeaderSize])
	if err != nil {
		return nil, fmt.Errorf("parse header: %w", err)
	}

	if err := segmentindex.CheckExpectedStrategy(header.Strategy); err != nil {
		return nil, fmt.Errorf("unsupported strategy in segment: %w", err)
	}

	if header.Version >= segmentindex.SegmentV1 && cfg.enableChecksumValidation {
		file.Seek(0, io.SeekStart)
		headerSize := int64(segmentindex.HeaderSize)
		if header.Strategy == segmentindex.StrategyInverted {
			headerSize += int64(segmentindex.HeaderInvertedSize)
		}
		segmentFile := segmentindex.NewSegmentFile(segmentindex.WithReader(file))
		if err := segmentFile.ValidateChecksum(size, headerSize); err != nil {
			return nil, fmt.Errorf("validate segment %q: %w", path, err)
		}
	}

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check the file size and first bytes (hexdump) — if empty or garbage, the segment is corrupt; remove it and restore the shard from backup.
2. Verify the segment was produced by a compatible Weaviate version; don't hand-copy segment files across versions or clusters.
3. Check disk health (SMART, filesystem fsck) — bitrot is the usual root cause of magic-byte mismatch.
4. If only one shard/segment is affected, drop the shard and rebuild from replicas or re-ingest the data.
5. Enable checksum validation (enableChecksumValidation) so corrupt segments are caught with clearer errors at load time.

Example fix

// before: hand-copied segment from an old version into the shard dir
cp ~/old-node/segment-7 /var/lib/weaviate/.../shard/segment-7
// after: restore the shard properly from a backup
weaviate backup restore --backend filesystem --id backup-1 && restart weaviate

func looksLikeSegmentFile(path string) error {
    f, err := os.Open(path)
    if err != nil {
        return err
    }
    defer f.Close()
    buf := make([]byte, segmentindex.HeaderSize)
    n, err := io.ReadFull(f, buf)
    if err != nil {
        return fmt.Errorf("segment %q too small for header (%d bytes): %w", path, n, err)
    }
    return segmentindex.ParseHeader(buf)
}

func isValidSegmentHeader(b []byte) bool {
    if len(b) < segmentindex.HeaderSize {
        return false
    }
    return segmentindex.ParseHeader(b[:segmentindex.HeaderSize]) == nil
}

header, err := segmentindex.ParseHeader(contents[:segmentindex.HeaderSize])
if err != nil {
    return nil, fmt.Errorf("segment %q has corrupt/foreign header — restore from backup: %w", path, err)
}

When it happens

Trigger: Opening a segment whose first bytes are not a valid header: truncated/empty file (contents shorter than segmentindex.HeaderSize), file written by an incompatible or corrupted write path, bitrot flipping magic/version bytes, or pointing Weaviate at a file that isn't a segment.

Common situations: Disk corruption or partial write after a crash leaving a zero-length or garbage segment; manually copying segment files between nodes/versions with mismatched segment formats; restoring a backup into the wrong location; storage layer silently returning zeros (misconfigured volume).

Related errors

• precompute metadata: %w
• unsupported strategy in segment: %w
• %q:%w
• not compatible with strategies %v
• recover: %v

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/9fec425625c0aebd. Report an issue: GitHub.