{"record":{"id":"65cbef4b1d6af65b","repo":"dotnet/machinelearning","slug":"the-length-of-inputids-and-labelids-should-be-the","errorCode":null,"errorMessage":"The length of inputIds and labelIds should be the same.","messagePattern":"The length of inputIds and labelIds should be the same\\.","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.GenAI.Core/Trainer/CausalLMDataset.cs","lineNumber":75,"sourceCode":"\n        var enumerable = inputs.Zip(outputs, (input, output) =>\n        {\n            var inputIds = tokenizer.EncodeToIds(input);\n            var outputIds = tokenizer.EncodeToIds(input + output);\n            outputIds = outputIds.Skip(inputIds.Count()).ToArray();\n\n            return (inputIds, outputIds);\n        }).ToArray();\n\n        return Create(enumerable.Select(x => x.inputIds), enumerable.Select(x => x.outputIds));\n    }\n\n    public static CausalLMDataset Create(IEnumerable<IReadOnlyList<int>> inputIds, IEnumerable<IReadOnlyList<int>> labelIds)\n    {\n        // the length of inputIds and labelIds should be the same\n        if (inputIds.Count() != labelIds.Count())\n        {\n            throw new ArgumentException(\"The length of inputIds and labelIds should be the same.\");\n        }\n\n        var enumerable = inputIds.Zip(labelIds, Create)\n            .SelectMany(x => x);\n\n        return new CausalLMDataset(enumerable);\n    }\n\n    public static CausalLMDataset Create(IReadOnlyList<int> inputIds, IReadOnlyList<int> labelIds)\n    {\n        var enumerable = Enumerable.Range(0, labelIds.Count)\n            .Select(i =>\n            {\n                var train = inputIds.Concat(labelIds.Take(i)).ToArray();\n                var label = Enumerable.Repeat(-100L, train.Length).Concat([labelIds[i]]).Skip(1).ToArray();\n                var mask = Enumerable.Repeat(1L, train.Length).ToArray();\n\n                return new CausalLMModelInput(","sourceCodeStart":57,"sourceCodeEnd":93,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.GenAI.Core/Trainer/CausalLMDataset.cs#L57-L93","documentation":"The token-id overload CausalLMDataset.Create(IEnumerable<IReadOnlyList<int>> inputIds, IEnumerable<IReadOnlyList<int>> labelIds) throws ArgumentException when the number of id sequences differs between the two collections. It zips the sequences pairwise into training examples, so the outer counts must match.","triggerScenarios":"Calling Create with pre-tokenized inputIds and labelIds collections of different lengths, e.g. one tokenized batch that was truncated or filtered independently of the labels.","commonSituations":"Tokenizing prompts and labels in separate passes where one pass failed silently or skipped an input; passing cached tokenized data from a different dataset version.","solutions":["Ensure both collections are produced from the same source rows in the same order and have equal counts.","Tokenize inputs and labels together from the same record list instead of two independent loops.","Count both collections before the call and fail fast with the offending index.","Regenerate the tokenized cache if it was produced by an older pipeline version."],"exampleFix":"// before\nvar ds = CausalLMDataset.Create(inputIds, labelIds);\n// after\nif (inputIds.Count() != labelIds.Count()) throw new InvalidDataException(\"tokenized rows mismatch\");\nvar ds = CausalLMDataset.Create(inputIds, labelIds);","handlingStrategy":"validation","validationCode":"if (inputIds.Count() != labelIds.Count()) throw new InvalidDataException(\"tokenized inputs/labels row count mismatch\");","typeGuard":"bool SameTokenizedRows(IEnumerable<IReadOnlyList<int>> a, IEnumerable<IReadOnlyList<int>> b) => a.Count() == b.Count();","tryCatchPattern":"try { var ds = CausalLMDataset.Create(inputIds, labelIds); }\ncatch (ArgumentException ex) when (ex.Message.Contains(\"inputIds and labelIds\")) { logger.LogError(\"Tokenized dataset rows misaligned: {Msg}\", ex.Message); throw; }","preventionTips":["Tokenize inputs and labels from the same record loop.","Validate both id collections were produced by the same tokenizer/pipeline version.","Materialize tokenized lists before passing them in.","Version your tokenized caches and invalidate on pipeline changes."],"tags":["dataset","training","tokenization"],"backgroundTag":"schema-validation-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}