{"record":{"id":"5e2ac10ad8e820c0","repo":"dotnet/machinelearning","slug":"failed-to-modify-unit-too-large-offset","errorCode":null,"errorMessage":"failed to modify unit: too large offset","messagePattern":"failed to modify unit: too large offset","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs","lineNumber":656,"sourceCode":"        // For this feature, leaf unit's label returns an id that has the MSB of 1.\n        public uint Label\n        {\n            get => _unit & ((1U << 31) | 0xFF);\n            set\n            {\n                _unit = (_unit & ~0xFFU) | value;\n            }\n        }\n\n        // offset() returns the offset from the unit to its derived units.\n        public uint Offset\n        {\n            get => (_unit >> 10) << (int)((_unit & (1U << 9)) >> 6);\n            set\n            {\n                if (value >= 1U << 29)\n                {\n                    throw new InvalidOperationException(\"failed to modify unit: too large offset\");\n                }\n\n                _unit &= (1U << 31) | (1U << 8) | 0xFF;\n\n                if (value < 1U << 21)\n                {\n                    _unit |= value << 10;\n                }\n                else\n                {\n                    _unit |= (value << 2) | (1U << 9);\n                }\n            }\n        }\n\n    }\n\n    //","sourceCodeStart":638,"sourceCodeEnd":674,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs#L638-L674","documentation":"The Unit.Offset property packs the trie node offset into bit fields that can only represent offsets up to 1<<29. When a computed offset exceeds that capacity, it throws this InvalidOperationException rather than silently truncating bits. This means the trie being built is too large/sparse for the encoding, not that the input data is wrong.","triggerScenarios":"Building a DoubleArrayTrie over a very large or highly fragmented key set where the node placement algorithm computes an offset >= 1U<<29 during unit modification.","commonSituations":"Loading an extremely large vocabulary (tens of millions of keys), an unlucky/sparse key distribution inflating the double-array size, or memory-constrained builds that reuse an undersized node array.","solutions":["Reduce the key set size or split the vocabulary across multiple tries.","Check for duplicated or near-duplicate keys inflating the structure; deduplicate before building.","Upgrade/patch to a library version with a wider offset encoding if you genuinely need a trie this large.","Profile the build to confirm the vocabulary size is intended (e.g. not loading a full corpus instead of a vocab)."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (vocabCount > 10_000_000) throw new NotSupportedException(\"Vocabulary too large for single DoubleArrayTrie build\");","typeGuard":null,"tryCatchPattern":"try { trie.Build(...); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"too large offset\"))\n{ /* split vocabulary into multiple tries or use a larger-capacity structure */ }","preventionTips":["Deduplicate keys before building to reduce trie size","Split very large vocabularies into shards","Monitor build size against the 1<<29 offset limit"],"tags":["trie","internal-limit","capacity","tokenizer"],"backgroundTag":"value-out-of-range","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}