{"record":{"id":"ff35d4e25433532c","repo":"weaviate/weaviate","slug":"centered-4-bit-codes-cap-the-output-dimension-at","errorCode":null,"errorMessage":"centered 4-bit codes cap the output dimension at %d, got %d","messagePattern":"centered 4-bit codes cap the output dimension at (.+?), got (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/vector/compressionhelpers/rq4.go","lineNumber":163,"sourceCode":"\t}\n\tif len(mean) != inputDim {\n\t\treturn nil, errors.Errorf(\"centering mean length %d does not match input dimension %d\", len(mean), inputDim)\n\t}\n\trq := NewFourBitRotationalQuantizer(inputDim, seed, distancer)\n\tif err := rq4CheckCenteredDim(rq.OutputDimension()); err != nil {\n\t\treturn nil, err\n\t}\n\trq.mean = mean\n\trq.meanNorm2 = dotProduct(mean, mean)\n\trq.layout = rq4LayoutFor(rq.OutputDimension())\n\trq.metaSize = rq.layout.size\n\trq.scratch.New = func() any { return newRQ4Scratch(rq.OutputDimension(), len(mean)) }\n\treturn rq, nil\n}\n\nfunc rq4CheckCenteredDim(outputDim int) error {\n\tif outputDim > rq4CenteredMaxOutputDim {\n\t\treturn errors.Errorf(\"centered 4-bit codes cap the output dimension at %d, got %d\",\n\t\t\trq4CenteredMaxOutputDim, outputDim)\n\t}\n\treturn nil\n}\n\nfunc putUint32(b []byte, pos int, x uint32) {\n\tbinary.BigEndian.PutUint32(b[pos:], x)\n}\n\nfunc getUint32(b []byte, pos int) uint32 {\n\treturn binary.BigEndian.Uint32(b[pos:])\n}\n\nfunc RestoreFourBitRotationalQuantizer(inputDim int, outputDim int, rounds int, swaps [][]compression.Swap, signs [][]float32, mean []float32, distancer distancer.Provider) (*FourBitRotationalQuantizer, error) {\n\t// Normalize empty to nil: an uncentered quantizer must never reach the\n\t// SIMD dot below — the amd64 kernel faults on a nil/empty slice.\n\tif len(mean) == 0 {\n\t\tmean = nil","sourceCodeStart":145,"sourceCodeEnd":181,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/vector/compressionhelpers/rq4.go#L145-L181","documentation":"Centered 4-bit rotational quantization stores codes with an expanded output dimension (due to padding for SIMD), and rq4CenteredMaxOutputDim caps how large that output dimension may be. rq4CheckCenteredDim rejects configurations whose outputDim exceeds this cap, since centered-RQ4 kernels and layouts are only defined up to that limit.","triggerScenarios":"Creating a centered RQ4 quantizer (NewCenteredFourBitRotationalQuantizer) or restoring one (RestoreFourBitRotationalQuantizer) for a class whose vector dimension is so large that the padded output dimension exceeds rq4CenteredMaxOutputDim — i.e. very high-dimensional vectors combined with centered RQ4 compression.","commonSituations":"Using centered RQ4 on models with very large embedding dimensions (e.g. long-context or multi-vector embeddings); upgrading where a new cap was introduced and previously-valid persisted quantizer state no longer restores.","solutions":["Use plain (non-centered) FourBitRotationalQuantizer or standard RQ/PQ/BQ compression for very high-dimensional vectors","Reduce the embedding dimension of the model or truncate vectors before import if the application allows","If restoring previously-persisted state, re-create the compression config so the quantizer is rebuilt under the current cap"],"exampleFix":"// before: centered RQ4 on an over-dimensioned class\nq, err := NewCenteredFourBitRotationalQuantizer(8192, seed, distancer, mean)\n// after: fall back to uncentered RQ4 when the centered output dim exceeds the cap\nrq := NewFourBitRotationalQuantizer(8192, seed, distancer)\nif err := rq4CheckCenteredDim(rq.OutputDimension()); err != nil {\n\tlogger.Warnf(\"falling back to uncentered RQ4: %v\", err)\n}\nq, err := NewFourBitRotationalQuantizer(8192, seed, distancer)","handlingStrategy":"validation","validationCode":"// check the padded output dim before requesting centered RQ4\nrq := NewFourBitRotationalQuantizer(inputDim, seed, distancer)\nif err := rq4CheckCenteredDim(rq.OutputDimension()); err != nil {\n\treturn fmt.Errorf(\"use uncentered RQ4/PQ for dim %d: %w\", inputDim, err)\n}\n","typeGuard":null,"tryCatchPattern":"q, err := NewCenteredFourBitRotationalQuantizer(inputDim, seed, distancer, mean)\nif err != nil {\n\t// fall back to uncentered or a different compression method\n\tq, err = NewFourBitRotationalQuantizer(inputDim, seed, distancer)\n}","preventionTips":["Check rq4CenteredMaxOutputDim against your vector dimension before configuring centered RQ4","Use plain RQ4, PQ, or BQ for very high-dimensional vectors","Re-verify persisted compression configs after upgrades that may change the cap"],"tags":["quantization","rq4","limits","configuration"],"backgroundTag":"dimension-limit-exceeded","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}