{"record":{"id":"d69812173cc4cd23","repo":"kopia/kopia","slug":"error-writing-extra-random-suffix-to-ensure-indexe-d69812","errorCode":null,"errorMessage":"error writing extra random suffix to ensure indexes are always globally unique","messagePattern":"error writing extra random suffix to ensure indexes are always globally unique","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"repo/content/index/one_use_index_builder.go","lineNumber":130,"sourceCode":"\t\tdataShardsBuf = append(dataShardsBuf, buf)\n\n\t\tif err := buildSortedContents(s, buf, indexVersion); err != nil {\n\t\t\tcloseShards()\n\n\t\t\treturn nil, nil, errors.Wrap(err, \"error building index shard\")\n\t\t}\n\n\t\tif !stable {\n\t\t\tif _, err := rand.Read(randomSuffix[:]); err != nil {\n\t\t\t\tcloseShards()\n\n\t\t\t\treturn nil, nil, errors.Wrap(err, \"error getting random bytes for suffix\")\n\t\t\t}\n\n\t\t\tif _, err := buf.Write(randomSuffix[:]); err != nil {\n\t\t\t\tcloseShards()\n\n\t\t\t\treturn nil, nil, errors.Wrap(err, \"error writing extra random suffix to ensure indexes are always globally unique\")\n\t\t\t}\n\t\t}\n\n\t\tdataShards = append(dataShards, buf.Bytes())\n\t}\n\n\treturn dataShards, closeShards, nil\n}\n","sourceCodeStart":112,"sourceCodeEnd":139,"githubUrl":"https://github.com/kopia/kopia/blob/82495e54b584c1ef6073c9e1be048f57f8aef078/repo/content/index/one_use_index_builder.go#L112-L139","documentation":"After generating the random suffix, BuildShards (when stable=false) writes it into the shard's write buffer. If buf.Write fails, partial buffers are closed via closeShards() and the error is wrapped as \"error writing extra random suffix to ensure indexes are always globally unique\". This signals an in-memory write-buffer failure while appending the uniqueness suffix.","triggerScenarios":"Calling BuildShards with stable=false when the gather.WriteBuffer's Write call fails — typically memory allocation failure or a buffer implementation error while appending len(randomSuffix) bytes.","commonSituations":"Out-of-memory / resource exhaustion while building very large shard sets; a defective or prematurely closed write buffer.","solutions":["Check the wrapped cause for memory/resource exhaustion and reduce concurrent shard builds or batch size.","Ensure closeShards() is not being invoked before this write (buffers must still be open).","Retry the operation once memory pressure is relieved; if uniqueness is unnecessary, use stable=true to skip the suffix write."],"exampleFix":"// before\nshards, cleanup, err := builder.BuildShards(v, false, hugeSize) // OOM during suffix write\n// after\nbuilder2 := builder.Split() // smaller batches\nshards, cleanup, err := builder2.BuildShards(v, false, hugeSize)","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"shards, cleanup, err := builder.BuildShards(v, false, size)\nif err != nil {\n\truntime.GC()\n\tshards, cleanup, err = builder.BuildShards(v, false, size)\n}","preventionTips":["Bound concurrent shard builds to limit memory pressure in write buffers.","Use stable=true to skip the random-suffix write when uniqueness is handled elsewhere.","Monitor memory before large index compactions."],"tags":["buffer","memory","index","write"],"backgroundTag":"file-write-failed","analyzedSha":"82495e54b584c1ef6073c9e1be048f57f8aef078","analyzedAt":"2026-09-07T20:35:21.689Z","contentChangedAt":"2026-09-07T20:35:21.689Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}