dotnet/machinelearning · error · InvalidOperationException

failed to insert key: invalid null character

Error message

failed to insert key: invalid null character

What it means

During trie construction, Insert detects that a key contains a null (0x00) byte at a position before the key's end and throws this InvalidOperationException. 0x00 is reserved as the internal end-of-key marker in the double-array structure, so embedded nulls would be ambiguous. This indicates the key bytes are invalid for this data structure.

Source

Thrown at src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs:363

            {
                throw new ArgumentException("failed to insert key: zero-length key");
            }

            uint id = 0;
            int keyPos = 0;

            for (; keyPos <= length; ++keyPos)
            {
                uint childId = _nodes[(int)id].Child;
                if (childId == 0)
                {
                    break;
                }

                byte keyLabel = key[keyPos];
                if (keyPos < length && keyLabel == 0)
                {
                    throw new InvalidOperationException("failed to insert key: invalid null character");
                }

                byte unitLabel = _nodes[(int)childId].Label;
                if (keyLabel < unitLabel)
                {
                    throw new InvalidOperationException("failed to insert key: wrong key order");
                }
                else if (keyLabel > unitLabel)
                {
                    _nodes[(int)childId].HasSibling = true;
                    Flush(childId);
                    break;
                }

                id = childId;
            }

            if (keyPos > length)

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Encode keys as UTF-8 (or another null-free encoding) before insertion.
2. Pass the exact key length so trailing padding bytes are not read as key content.
3. If binary keys must be supported, pre-map them to a null-free representation (e.g. hex/base64 string).
4. Sanitize or reject keys containing '\0' before calling Insert.

Example fix

// before: UTF-16 encoding introduces 0x00 bytes
byte[] key = Encoding.Unicode.GetBytes(piece);
trie.Insert(key, key.Length, id);
// after
byte[] key = Encoding.UTF8.GetBytes(piece);
trie.Insert(key, key.Length, id);

byte[] keyBytes = Encoding.UTF8.GetBytes(piece);
if (keyBytes.Contains((byte)0)) throw new ArgumentException("Key contains embedded null byte");

static bool IsNullFreeKey(ReadOnlySpan<byte> key) => !key.Slice(0, Math.Max(0, key.Length - 1)).Contains((byte)0);

try { trie.Insert(keyBytes, keyBytes.Length, id); }
catch (InvalidOperationException ex) when (ex.Message.Contains("invalid null character"))
{ /* re-encode key or reject entry */ }

When it happens

Trigger: Inserting a key whose byte representation contains an interior 0x00 byte — e.g. UTF-16 bytes inserted directly without conversion, binary data used as a key, or a buffer copied with trailing/padding zeros misread as part of the key.

Common situations: Encoding keys with a fixed-width charset (UTF-16/UTF-32) instead of UTF-8, passing a buffer larger than the actual key length so padding zeros are treated as key content, or inserting raw binary hashes as trie keys.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• The maximum number of tokens must be greater than zero.
• The maximum number of tokens must be greater than zero.
• failed to insert key: negative value
• failed to insert key: zero-length key
• failed to insert key: wrong key order

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/1940b1796db0c741. Report an issue: GitHub.