{"record":{"id":"e93c6b2f373829c3","repo":"dotnet/machinelearning","slug":"invalid-base64-string-base64string-substring-off","errorCode":null,"errorMessage":"Invalid base64 string '{base64String.Substring(offset, length)}'","messagePattern":"Invalid base64 string '(.+?)'","errorType":"validation","errorClass":"FormatException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Utils/Helpers.netcoreapp.cs","lineNumber":40,"sourceCode":"    {\n        public static ValueTask<string?> ReadLineAsync(StreamReader reader, CancellationToken cancellationToken) =>\n            reader.ReadLineAsync(cancellationToken);\n\n        public static Task<Stream> GetStreamAsync(HttpClient client, string url, CancellationToken cancellationToken = default) =>\n            client.GetStreamAsync(url, cancellationToken);\n\n        public static Stream GetStream(HttpClient client, string url)\n        {\n            HttpResponseMessage response = client.Send(new HttpRequestMessage(HttpMethod.Get, url), HttpCompletionOption.ResponseHeadersRead);\n            response.EnsureSuccessStatusCode();\n            return response.Content.ReadAsStream();\n        }\n\n        public static byte[] FromBase64String(string base64String, int offset, int length)\n        {\n            if (!Base64.IsValid(base64String.AsSpan(offset, length), out int decodedLength))\n            {\n                throw new FormatException($\"Invalid base64 string '{base64String.Substring(offset, length)}'\");\n            }\n\n            byte[] bytes = new byte[decodedLength];\n            bool success = Convert.TryFromBase64Chars(base64String.AsSpan(offset, length), bytes, out int bytesWritten);\n            Debug.Assert(success);\n            Debug.Assert(bytes.Length == bytesWritten);\n            return bytes;\n        }\n\n        internal static bool TryParseInt32(string s, int offset, out int result)\n            => int.TryParse(s.AsSpan().Slice(offset), NumberStyles.None, CultureInfo.InvariantCulture, out result);\n\n        internal static int GetHashCode(ReadOnlySpan<char> span) => string.GetHashCode(span);\n\n        internal static unsafe int GetUtf8Bytes(ReadOnlySpan<char> source, Span<byte> destination)\n            => Encoding.UTF8.GetBytes(source, destination);\n\n        internal static unsafe bool TryGetUtf8Bytes(ReadOnlySpan<char> source, Span<byte> destination, out int bytesWritten)","sourceCodeStart":22,"sourceCodeEnd":58,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Utils/Helpers.netcoreapp.cs#L22-L58","documentation":"Helpers.FromBase64String validates the requested substring with Base64.IsValid before decoding, and throws this FormatException when the slice is not valid base64 (bad characters, wrong length/padding, embedded whitespace). The method decodes a portion of a larger string (e.g. a vocab entry), so an offset/length that doesn't align to a complete base64 blob triggers this.","triggerScenarios":"Calling FromBase64String(s, offset, length) with a slice containing non-base64 characters, missing padding, or a length not a multiple of 4 — commonly when the offset/length arithmetic over the containing string is off by a delimiter or quote character.","commonSituations":"Parsing a byte-level BPE vocab where each token is stored as base64, but the file format changed (extra delimiters, JSON-escaped content) so offsets are computed incorrectly; hand-editing vocab files; mixing vocab formats across library versions.","solutions":["Fix the offset/length calculation so the slice covers exactly the base64 payload (excluding delimiters/quotes).","Pre-validate with Base64.IsValid(span) and log/repair the offending entry instead of crashing.","Check whether the vocab file format matches what the tokenizer version expects; regenerate the vocab.","Trim stray whitespace/quotes from the slice before decoding."],"exampleFix":"// before: length includes the trailing delimiter\nint len = lineEnd - start; // includes ','\nbyte[] b = Helpers.FromBase64String(line, start, len);\n// after\nint len = lineEnd - start;\nif (line[len-1] == ',') len--;\nbyte[] b = Helpers.FromBase64String(line, start, len);","handlingStrategy":"validation","validationCode":"if (!Base64.IsValid(textSpan)) throw new FormatException($\"Invalid base64 at offset {offset}, length {length}\");\nbyte[] bytes = Helpers.FromBase64String(s, offset, length);","typeGuard":"static bool IsValidBase64Slice(string s, int offset, int length) => offset >= 0 && length >= 0 && offset + length <= s.Length && Base64.IsValid(s.AsSpan(offset, length));","tryCatchPattern":"try { bytes = Helpers.FromBase64String(s, offset, length); }\ncatch (FormatException ex)\n{ log.LogError(ex, \"Bad base64 token in vocab\"); bytes = Array.Empty<byte>(); }","preventionTips":["Compute slice bounds excluding delimiters/quotes","Pre-validate with Base64.IsValid before decoding","Keep vocab file format and parser offsets in sync per library version"],"tags":["base64","format","parsing","tokenizer"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}