{"record":{"id":"1940b1796db0c741","repo":"dotnet/machinelearning","slug":"failed-to-insert-key-invalid-null-character","errorCode":null,"errorMessage":"failed to insert key: invalid null character","messagePattern":"failed to insert key: invalid null character","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs","lineNumber":363,"sourceCode":"            {\n                throw new ArgumentException(\"failed to insert key: zero-length key\");\n            }\n\n            uint id = 0;\n            int keyPos = 0;\n\n            for (; keyPos <= length; ++keyPos)\n            {\n                uint childId = _nodes[(int)id].Child;\n                if (childId == 0)\n                {\n                    break;\n                }\n\n                byte keyLabel = key[keyPos];\n                if (keyPos < length && keyLabel == 0)\n                {\n                    throw new InvalidOperationException(\"failed to insert key: invalid null character\");\n                }\n\n                byte unitLabel = _nodes[(int)childId].Label;\n                if (keyLabel < unitLabel)\n                {\n                    throw new InvalidOperationException(\"failed to insert key: wrong key order\");\n                }\n                else if (keyLabel > unitLabel)\n                {\n                    _nodes[(int)childId].HasSibling = true;\n                    Flush(childId);\n                    break;\n                }\n\n                id = childId;\n            }\n\n            if (keyPos > length)","sourceCodeStart":345,"sourceCodeEnd":381,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Utils/DoubleArrayTrie.cs#L345-L381","documentation":"During trie construction, Insert detects that a key contains a null (0x00) byte at a position before the key's end and throws this InvalidOperationException. 0x00 is reserved as the internal end-of-key marker in the double-array structure, so embedded nulls would be ambiguous. This indicates the key bytes are invalid for this data structure.","triggerScenarios":"Inserting a key whose byte representation contains an interior 0x00 byte — e.g. UTF-16 bytes inserted directly without conversion, binary data used as a key, or a buffer copied with trailing/padding zeros misread as part of the key.","commonSituations":"Encoding keys with a fixed-width charset (UTF-16/UTF-32) instead of UTF-8, passing a buffer larger than the actual key length so padding zeros are treated as key content, or inserting raw binary hashes as trie keys.","solutions":["Encode keys as UTF-8 (or another null-free encoding) before insertion.","Pass the exact key length so trailing padding bytes are not read as key content.","If binary keys must be supported, pre-map them to a null-free representation (e.g. hex/base64 string).","Sanitize or reject keys containing '\\0' before calling Insert."],"exampleFix":"// before: UTF-16 encoding introduces 0x00 bytes\nbyte[] key = Encoding.Unicode.GetBytes(piece);\ntrie.Insert(key, key.Length, id);\n// after\nbyte[] key = Encoding.UTF8.GetBytes(piece);\ntrie.Insert(key, key.Length, id);","handlingStrategy":"validation","validationCode":"byte[] keyBytes = Encoding.UTF8.GetBytes(piece);\nif (keyBytes.Contains((byte)0)) throw new ArgumentException(\"Key contains embedded null byte\");","typeGuard":"static bool IsNullFreeKey(ReadOnlySpan<byte> key) => !key.Slice(0, Math.Max(0, key.Length - 1)).Contains((byte)0);","tryCatchPattern":"try { trie.Insert(keyBytes, keyBytes.Length, id); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"invalid null character\"))\n{ /* re-encode key or reject entry */ }","preventionTips":["Always encode keys as UTF-8, never UTF-16/UTF-32","Pass exact key lengths, not buffer capacities","Reject binary keys containing '\\0' at the API boundary"],"tags":["trie","invalid-key","encoding","tokenizer"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}