dotnet/machinelearning · error · InvalidOperationException

failed to modify unit: too large offset

Error message

failed to modify unit: too large offset

What it means

The Unit.Offset property packs the trie node offset into bit fields that can only represent offsets up to 1<<29. When a computed offset exceeds that capacity, it throws this InvalidOperationException rather than silently truncating bits. This means the trie being built is too large/sparse for the encoding, not that the input data is wrong.

Source

Thrown at src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs:656

        // For this feature, leaf unit's label returns an id that has the MSB of 1.
        public uint Label
        {
            get => _unit & ((1U << 31) | 0xFF);
            set
            {
                _unit = (_unit & ~0xFFU) | value;
            }
        }

        // offset() returns the offset from the unit to its derived units.
        public uint Offset
        {
            get => (_unit >> 10) << (int)((_unit & (1U << 9)) >> 6);
            set
            {
                if (value >= 1U << 29)
                {
                    throw new InvalidOperationException("failed to modify unit: too large offset");
                }

                _unit &= (1U << 31) | (1U << 8) | 0xFF;

                if (value < 1U << 21)
                {
                    _unit |= value << 10;
                }
                else
                {
                    _unit |= (value << 2) | (1U << 9);
                }
            }
        }

    }

    //

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Reduce the key set size or split the vocabulary across multiple tries.
2. Check for duplicated or near-duplicate keys inflating the structure; deduplicate before building.
3. Upgrade/patch to a library version with a wider offset encoding if you genuinely need a trie this large.
4. Profile the build to confirm the vocabulary size is intended (e.g. not loading a full corpus instead of a vocab).

if (vocabCount > 10_000_000) throw new NotSupportedException("Vocabulary too large for single DoubleArrayTrie build");

try { trie.Build(...); }
catch (InvalidOperationException ex) when (ex.Message.Contains("too large offset"))
{ /* split vocabulary into multiple tries or use a larger-capacity structure */ }

When it happens

Trigger: Building a DoubleArrayTrie over a very large or highly fragmented key set where the node placement algorithm computes an offset >= 1U<<29 during unit modification.

Common situations: Loading an extremely large vocabulary (tens of millions of keys), an unlucky/sparse key distribution inflating the double-array size, or memory-constrained builds that reuse an undersized node array.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• failed to insert key: negative value
• failed to insert key: zero-length key
• failed to insert key: invalid null character
• failed to insert key: wrong key order
• {capacity} exceeds buffer capacity

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5e2ac10ad8e820c0. Report an issue: GitHub.