weaviate/weaviate · error

initialize new segment: %w

Error message

initialize new segment: %w

What it means

preinitializeNewSegment calls newSegment() to open the freshly written .tmp file as a segment (mmap contents, bloom filter, count-net-additions, checksum validation, etc., per the SegmentGroup's config). This error wraps any failure from that segment initialization — file open/read errors, mmap failures, checksummed metadata problems, or memory-allocation (allocChecker) refusals.

Source

Thrown at adapters/repos/db/lsmkv/segment_group_compaction.go:693

	seg, err := newSegment(newPathTmp, sg.logger, sg.metrics, nil,
		segmentConfig{
			mmapContents:                 sg.mmapContents,
			useBloomFilter:               sg.useBloomFilter,
			calcCountNetAdditions:        sg.calcCountNetAdditions,
			overwriteDerived:             true,
			enableChecksumValidation:     sg.enableChecksumValidation,
			sequentialAccess:             sg.sequentialAccess,
			MinMMapSize:                  sg.MinMMapSize,
			allocChecker:                 sg.allocChecker,
			precomputedCountNetAdditions: &updatedCountNetAdditions,
			fileList:                     make(map[string]int64), // empty to not check if bloom/cna files already exist
			writeMetadata:                sg.writeMetadata,
			deleteMarkerCounter:          sg.deleteMarkerCounter.Add(1),
			lazyPropertyLengths:          sg.lazyPropertyLengths,
		})
	if err != nil {
		return nil, fmt.Errorf("initialize new segment: %w", err)
	}

	return seg, nil
}

func (sg *SegmentGroup) waitForReferenceCountToReachZero(segments ...Segment) {
	if len(segments) == 0 {
		return
	}

	const (
		tickerInterval = 100 * time.Millisecond
		warnThreshold  = 10 * time.Second
		warnInterval   = 10 * time.Second
	)

	start := time.Now()
	var lastWarn time.Time

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check node memory and free disk space (compaction temporarily doubles footprint); reduce memory pressure or raise available RAM before retrying compaction.
2. Check open-file limits (ulimit -n) and vm.max_map_count if mmap-related errors appear in the wrapped cause.
3. Inspect the wrapped error in the log ('initialize new segment: <cause>') for the specific cause; for corruption, delete the orphaned .tmp file and let compaction re-run, or restore the shard from backup/replica.
4. Increase the allocChecker budget or configure a larger memory allocation for Weaviate if allocation-checker refusals are the cause.

// precheck resources before triggering compaction-heavy workloads
if freeMemoryBytes < requiredMmapBudget || fdAvailable < 1024 {
    return errors.New("insufficient memory or file descriptors for segment initialization")
}

seg, err := sg.preinitializeNewSegment(path, pair...)
if err != nil {
    return fmt.Errorf("preinitialize new segment: %w", err) // inspect inner cause
}

When it happens

Trigger: Called from compactOnceAbortable/replaceSegment right after the compacted file is fsynced and closed. Fails when: the .tmp file is unreadable/corrupt (disk error, partial write despite fsync), mmap is requested but fails (out of address space, mmap disabled by fs/mount options), the allocation checker rejects the allocation due to low free memory, or derived-file (bloom/cna/metadata) writing fails due to disk pressure.

Common situations: Low free memory on the node tripping allocChecker; running inside containers with restricted vm.max_map_count or low file-descriptor limits; full disk during compaction; corrupted output after an earlier I/O error; mmap of very large compacted segments on 32-bit or memory-limited environments.

Related errors

• could not copy all data into buffer
• pause compaction for rangeable in-memory rebuild: %w
• pause compaction: %w
• flush buffered: %w
• write compactorMap segment checksum: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/977d8486aac5e44b. Report an issue: GitHub.