dotnet/machinelearning · error · FormatException

Invalid base64 string '{base64String.Substring(offset, lengt

Error message

Invalid base64 string '{base64String.Substring(offset, length)}'

What it means

Helpers.FromBase64String validates the requested substring with Base64.IsValid before decoding, and throws this FormatException when the slice is not valid base64 (bad characters, wrong length/padding, embedded whitespace). The method decodes a portion of a larger string (e.g. a vocab entry), so an offset/length that doesn't align to a complete base64 blob triggers this.

Source

Thrown at src/Microsoft.ML.Tokenizers/Utils/Helpers.netcoreapp.cs:40

    {
        public static ValueTask<string?> ReadLineAsync(StreamReader reader, CancellationToken cancellationToken) =>
            reader.ReadLineAsync(cancellationToken);

        public static Task<Stream> GetStreamAsync(HttpClient client, string url, CancellationToken cancellationToken = default) =>
            client.GetStreamAsync(url, cancellationToken);

        public static Stream GetStream(HttpClient client, string url)
        {
            HttpResponseMessage response = client.Send(new HttpRequestMessage(HttpMethod.Get, url), HttpCompletionOption.ResponseHeadersRead);
            response.EnsureSuccessStatusCode();
            return response.Content.ReadAsStream();
        }

        public static byte[] FromBase64String(string base64String, int offset, int length)
        {
            if (!Base64.IsValid(base64String.AsSpan(offset, length), out int decodedLength))
            {
                throw new FormatException($"Invalid base64 string '{base64String.Substring(offset, length)}'");
            }

            byte[] bytes = new byte[decodedLength];
            bool success = Convert.TryFromBase64Chars(base64String.AsSpan(offset, length), bytes, out int bytesWritten);
            Debug.Assert(success);
            Debug.Assert(bytes.Length == bytesWritten);
            return bytes;
        }

        internal static bool TryParseInt32(string s, int offset, out int result)
            => int.TryParse(s.AsSpan().Slice(offset), NumberStyles.None, CultureInfo.InvariantCulture, out result);

        internal static int GetHashCode(ReadOnlySpan<char> span) => string.GetHashCode(span);

        internal static unsafe int GetUtf8Bytes(ReadOnlySpan<char> source, Span<byte> destination)
            => Encoding.UTF8.GetBytes(source, destination);

        internal static unsafe bool TryGetUtf8Bytes(ReadOnlySpan<char> source, Span<byte> destination, out int bytesWritten)

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Fix the offset/length calculation so the slice covers exactly the base64 payload (excluding delimiters/quotes).
  2. Pre-validate with Base64.IsValid(span) and log/repair the offending entry instead of crashing.
  3. Check whether the vocab file format matches what the tokenizer version expects; regenerate the vocab.
  4. Trim stray whitespace/quotes from the slice before decoding.

Example fix

// before: length includes the trailing delimiter
int len = lineEnd - start; // includes ','
byte[] b = Helpers.FromBase64String(line, start, len);
// after
int len = lineEnd - start;
if (line[len-1] == ',') len--;
byte[] b = Helpers.FromBase64String(line, start, len);
Defensive patterns

Strategy: validation

Validate before calling

if (!Base64.IsValid(textSpan)) throw new FormatException($"Invalid base64 at offset {offset}, length {length}");
byte[] bytes = Helpers.FromBase64String(s, offset, length);

Type guard

static bool IsValidBase64Slice(string s, int offset, int length) => offset >= 0 && length >= 0 && offset + length <= s.Length && Base64.IsValid(s.AsSpan(offset, length));

Try / catch

try { bytes = Helpers.FromBase64String(s, offset, length); }
catch (FormatException ex)
{ log.LogError(ex, "Bad base64 token in vocab"); bytes = Array.Empty<byte>(); }

Prevention

When it happens

Trigger: Calling FromBase64String(s, offset, length) with a slice containing non-base64 characters, missing padding, or a length not a multiple of 4 — commonly when the offset/length arithmetic over the containing string is off by a delimiter or quote character.

Common situations: Parsing a byte-level BPE vocab where each token is stored as base64, but the file format changed (extra delimiters, JSON-escaped content) so offsets are computed incorrectly; hand-editing vocab files; mixing vocab formats across library versions.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e93c6b2f373829c3. Report an issue: GitHub.