dotnet/machinelearning · error · ArgumentException

failed to insert key: zero-length key

Error message

failed to insert key: zero-length key

What it means

DoubleArrayTrie.Insert throws this ArgumentException when the key length passed is zero. An empty key cannot be represented in the double-array structure, so the library refuses it up front. Like the negative-value check, this fires while building the trie from a vocabulary.

Source

Thrown at src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs:346

        public void Finish()
        {
            Flush(0);

            _units[0] = _nodes[0].Unit;
            _labels[0] = _nodes[0].Label;
            _isIntersections.Build();
        }

        public void Insert(ReadOnlySpan<byte> key, int length, int value)
        {
            if (value < 0)
            {
                throw new ArgumentException("failed to insert key: negative value");
            }
            else if (length == 0)
            {
                throw new ArgumentException("failed to insert key: zero-length key");
            }

            uint id = 0;
            int keyPos = 0;

            for (; keyPos <= length; ++keyPos)
            {
                uint childId = _nodes[(int)id].Child;
                if (childId == 0)
                {
                    break;
                }

                byte keyLabel = key[keyPos];
                if (keyPos < length && keyLabel == 0)
                {
                    throw new InvalidOperationException("failed to insert key: invalid null character");
                }

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Filter out empty keys before inserting: skip entries where the piece is the empty string.
2. Fix the splitting/parsing code that produces empty tokens.
3. Check the vocab file for blank lines or empty pieces and clean it.
4. Add a guard in the builder loop: if (piece.Length == 0) continue;

Example fix

// before
foreach (var piece in vocab.Keys)
    trie.Insert(Encoding.UTF8.GetBytes(piece), Encoding.UTF8.GetByteCount(piece), id++);
// after
foreach (var piece in vocab.Keys.Where(p => p.Length > 0))
    trie.Insert(Encoding.UTF8.GetBytes(piece), Encoding.UTF8.GetByteCount(piece), id++);

if (piece.Length == 0) return; // or throw
trie.Insert(Encoding.UTF8.GetBytes(piece), Encoding.UTF8.GetByteCount(piece), id);

static bool IsInsertableKey(ReadOnlySpan<byte> key) => key.Length > 0 && !key.Contains((byte)0);

try { trie.Insert(key, key.Length, id); }
catch (ArgumentException ex) when (ex.Message.Contains("zero-length"))
{ /* skip or log the empty key */ }

When it happens

Trigger: Calling Insert with length == 0, e.g. BuildDawg encountering an empty string entry in the vocabulary (empty token in the sentencepiece vocab or a splitting bug producing empty pieces).

Common situations: A malformed vocab file with an empty token line, a text-splitting routine that emits empty substrings, or a loop bug passing zero length.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• failed to insert key: negative value
• failed to insert key: invalid null character
• failed to insert key: wrong key order
• failed to modify unit: too large offset
• Expected either {0} or {1} to be provided

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/448ab97cc26ad05a. Report an issue: GitHub.