dotnet/machinelearning · error · InvalidOperationException

failed to modify unit: too large offset

Error message

failed to modify unit: too large offset

What it means

The Unit.Offset property packs the trie node offset into bit fields that can only represent offsets up to 1<<29. When a computed offset exceeds that capacity, it throws this InvalidOperationException rather than silently truncating bits. This means the trie being built is too large/sparse for the encoding, not that the input data is wrong.

Solutions

1. Reduce the key set size or split the vocabulary across multiple tries.
2. Check for duplicated or near-duplicate keys inflating the structure; deduplicate before building.
3. Upgrade/patch to a library version with a wider offset encoding if you genuinely need a trie this large.
4. Profile the build to confirm the vocabulary size is intended (e.g. not loading a full corpus instead of a vocab).

if (vocabCount > 10_000_000) throw new NotSupportedException("Vocabulary too large for single DoubleArrayTrie build");

try { trie.Build(...); }
catch (InvalidOperationException ex) when (ex.Message.Contains("too large offset"))
{ /* split vocabulary into multiple tries or use a larger-capacity structure */ }

When it happens

Trigger: Building a DoubleArrayTrie over a very large or highly fragmented key set where the node placement algorithm computes an offset >= 1U<<29 during unit modification.

Common situations: Loading an extremely large vocabulary (tens of millions of keys), an unlucky/sparse key distribution inflating the double-array size, or memory-constrained builds that reuse an undersized node array.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• failed to insert key: invalid null character
• failed to insert key: negative value
• failed to insert key: wrong key order
• failed to insert key: zero-length key
• A piece string in 'model.vocab' is null.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5e2ac10ad8e820c0. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs:656

        // For this feature, leaf unit's label returns an id that has the MSB of 1.
        public uint Label
        {
            get => _unit & ((1U << 31) | 0xFF);
            set
            {
                _unit = (_unit & ~0xFFU) | value;
            }
        }

        // offset() returns the offset from the unit to its derived units.
        public uint Offset
        {
            get => (_unit >> 10) << (int)((_unit & (1U << 9)) >> 6);
            set
            {
                if (value >= 1U << 29)
                {
                    throw new InvalidOperationException("failed to modify unit: too large offset");
                }

                _unit &= (1U << 31) | (1U << 8) | 0xFF;

                if (value < 1U << 21)
                {
                    _unit |= value << 10;
                }
                else
                {
                    _unit |= (value << 2) | (1U << 9);
                }
            }
        }

    }

    //

View on GitHub (pinned to 7b76e69cf9)