{"record":{"id":"977d8486aac5e44b","repo":"weaviate/weaviate","slug":"initialize-new-segment-w","errorCode":null,"errorMessage":"initialize new segment: %w","messagePattern":"initialize new segment: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/lsmkv/segment_group_compaction.go","lineNumber":693,"sourceCode":"\n\tseg, err := newSegment(newPathTmp, sg.logger, sg.metrics, nil,\n\t\tsegmentConfig{\n\t\t\tmmapContents:                 sg.mmapContents,\n\t\t\tuseBloomFilter:               sg.useBloomFilter,\n\t\t\tcalcCountNetAdditions:        sg.calcCountNetAdditions,\n\t\t\toverwriteDerived:             true,\n\t\t\tenableChecksumValidation:     sg.enableChecksumValidation,\n\t\t\tsequentialAccess:             sg.sequentialAccess,\n\t\t\tMinMMapSize:                  sg.MinMMapSize,\n\t\t\tallocChecker:                 sg.allocChecker,\n\t\t\tprecomputedCountNetAdditions: &updatedCountNetAdditions,\n\t\t\tfileList:                     make(map[string]int64), // empty to not check if bloom/cna files already exist\n\t\t\twriteMetadata:                sg.writeMetadata,\n\t\t\tdeleteMarkerCounter:          sg.deleteMarkerCounter.Add(1),\n\t\t\tlazyPropertyLengths:          sg.lazyPropertyLengths,\n\t\t})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"initialize new segment: %w\", err)\n\t}\n\n\treturn seg, nil\n}\n\nfunc (sg *SegmentGroup) waitForReferenceCountToReachZero(segments ...Segment) {\n\tif len(segments) == 0 {\n\t\treturn\n\t}\n\n\tconst (\n\t\ttickerInterval = 100 * time.Millisecond\n\t\twarnThreshold  = 10 * time.Second\n\t\twarnInterval   = 10 * time.Second\n\t)\n\n\tstart := time.Now()\n\tvar lastWarn time.Time","sourceCodeStart":675,"sourceCodeEnd":711,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/lsmkv/segment_group_compaction.go#L675-L711","documentation":"preinitializeNewSegment calls newSegment() to open the freshly written .tmp file as a segment (mmap contents, bloom filter, count-net-additions, checksum validation, etc., per the SegmentGroup's config). This error wraps any failure from that segment initialization — file open/read errors, mmap failures, checksummed metadata problems, or memory-allocation (allocChecker) refusals.","triggerScenarios":"Called from compactOnceAbortable/replaceSegment right after the compacted file is fsynced and closed. Fails when: the .tmp file is unreadable/corrupt (disk error, partial write despite fsync), mmap is requested but fails (out of address space, mmap disabled by fs/mount options), the allocation checker rejects the allocation due to low free memory, or derived-file (bloom/cna/metadata) writing fails due to disk pressure.","commonSituations":"Low free memory on the node tripping allocChecker; running inside containers with restricted vm.max_map_count or low file-descriptor limits; full disk during compaction; corrupted output after an earlier I/O error; mmap of very large compacted segments on 32-bit or memory-limited environments.","solutions":["Check node memory and free disk space (compaction temporarily doubles footprint); reduce memory pressure or raise available RAM before retrying compaction.","Check open-file limits (ulimit -n) and vm.max_map_count if mmap-related errors appear in the wrapped cause.","Inspect the wrapped error in the log ('initialize new segment: <cause>') for the specific cause; for corruption, delete the orphaned .tmp file and let compaction re-run, or restore the shard from backup/replica.","Increase the allocChecker budget or configure a larger memory allocation for Weaviate if allocation-checker refusals are the cause."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// precheck resources before triggering compaction-heavy workloads\nif freeMemoryBytes < requiredMmapBudget || fdAvailable < 1024 {\n    return errors.New(\"insufficient memory or file descriptors for segment initialization\")\n}","typeGuard":null,"tryCatchPattern":"seg, err := sg.preinitializeNewSegment(path, pair...)\nif err != nil {\n    return fmt.Errorf(\"preinitialize new segment: %w\", err) // inspect inner cause\n}","preventionTips":["Provision headroom: compaction briefly needs memory and disk for both old and new segments","Raise ulimit -n and vm.max_map_count in containerized deployments","Keep the allocChecker budget aligned with actual available memory","Inspect the wrapped cause in logs to distinguish mmap vs disk vs allocation failures"],"tags":["lsmkv","compaction","mmap","segment-initialization"],"backgroundTag":"segment-init-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}