dotnet/machinelearning · error · ArgumentException

The length of inputIds and labelIds should be the same.

Error message

The length of inputIds and labelIds should be the same.

What it means

The token-id overload CausalLMDataset.Create(IEnumerable<IReadOnlyList<int>> inputIds, IEnumerable<IReadOnlyList<int>> labelIds) throws ArgumentException when the number of id sequences differs between the two collections. It zips the sequences pairwise into training examples, so the outer counts must match.

Solutions

1. Ensure both collections are produced from the same source rows in the same order and have equal counts.
2. Tokenize inputs and labels together from the same record list instead of two independent loops.
3. Count both collections before the call and fail fast with the offending index.
4. Regenerate the tokenized cache if it was produced by an older pipeline version.

Example fix

// before
var ds = CausalLMDataset.Create(inputIds, labelIds);
// after
if (inputIds.Count() != labelIds.Count()) throw new InvalidDataException("tokenized rows mismatch");
var ds = CausalLMDataset.Create(inputIds, labelIds);

if (inputIds.Count() != labelIds.Count()) throw new InvalidDataException("tokenized inputs/labels row count mismatch");

bool SameTokenizedRows(IEnumerable<IReadOnlyList<int>> a, IEnumerable<IReadOnlyList<int>> b) => a.Count() == b.Count();

try { var ds = CausalLMDataset.Create(inputIds, labelIds); }
catch (ArgumentException ex) when (ex.Message.Contains("inputIds and labelIds")) { logger.LogError("Tokenized dataset rows misaligned: {Msg}", ex.Message); throw; }

When it happens

Trigger: Calling Create with pre-tokenized inputIds and labelIds collections of different lengths, e.g. one tokenized batch that was truncated or filtered independently of the labels.

Common situations: Tokenizing prompts and labels in separate passes where one pass failed silently or skipped an input; passing cached tokenized data from a different dataset version.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• The length of inputs and outputs should be the same.
• The max token count must be greater than 0.
• The maximum number of tokens must be greater than 0.
• The model type ' ' is not supported.
• Training failed with the exception

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/65cbef4b1d6af65b. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.GenAI.Core/Trainer/CausalLMDataset.cs:75

        var enumerable = inputs.Zip(outputs, (input, output) =>
        {
            var inputIds = tokenizer.EncodeToIds(input);
            var outputIds = tokenizer.EncodeToIds(input + output);
            outputIds = outputIds.Skip(inputIds.Count()).ToArray();

            return (inputIds, outputIds);
        }).ToArray();

        return Create(enumerable.Select(x => x.inputIds), enumerable.Select(x => x.outputIds));
    }

    public static CausalLMDataset Create(IEnumerable<IReadOnlyList<int>> inputIds, IEnumerable<IReadOnlyList<int>> labelIds)
    {
        // the length of inputIds and labelIds should be the same
        if (inputIds.Count() != labelIds.Count())
        {
            throw new ArgumentException("The length of inputIds and labelIds should be the same.");
        }

        var enumerable = inputIds.Zip(labelIds, Create)
            .SelectMany(x => x);

        return new CausalLMDataset(enumerable);
    }

    public static CausalLMDataset Create(IReadOnlyList<int> inputIds, IReadOnlyList<int> labelIds)
    {
        var enumerable = Enumerable.Range(0, labelIds.Count)
            .Select(i =>
            {
                var train = inputIds.Concat(labelIds.Take(i)).ToArray();
                var label = Enumerable.Repeat(-100L, train.Length).Concat([labelIds[i]]).Skip(1).ToArray();
                var mask = Enumerable.Repeat(1L, train.Length).ToArray();

                return new CausalLMModelInput(

View on GitHub (pinned to 7b76e69cf9)