dotnet/machinelearning · error · NotImplementedException

PretrainingTp > 1 is not supported yet.

Error message

PretrainingTp > 1 is not supported yet.

What it means

LlamaMLP.forward explicitly rejects tensor-parallel pretraining configurations greater than 1 by throwing NotImplementedException. The MLP only implements the single-GPU gate/up/down projection path; tensor-parallel sharding of the weight matrices was never implemented in this C# port.

Solutions

1. Set pretraining_tp to 1 in the model config before loading (e.g. config.PretrainingTp = 1); logits quality is unaffected for inference purposes.
2. Check your config.json for "pretraining_tp": 8 (or other >1) and edit it to 1.
3. If tensor parallel inference is truly required, use a different runtime (e.g. vLLM/PyTorch) — this library does not support it.

Example fix

// before
var config = LlamaConfig.FromFile("config.json"); // pretraining_tp: 8
var model = LlamaForCausalLM.FromModelFolder(path, device);
// after
var config = LlamaConfig.FromFile("config.json");
config.PretrainingTp = 1;
var model = LlamaForCausalLM.FromModelFolder(path, device);

if (config.PretrainingTp > 1) throw new InvalidOperationException("Set pretraining_tp=1; tensor parallel inference is unsupported.");

bool SupportsTensorParallel(LlamaConfig c) => c.PretrainingTp <= 1;

try { output = model.forward(input); } catch (NotImplementedException) { /* reconfigure PretrainingTp=1 and retry, or surface unsupported-feature error */ }

When it happens

Trigger: Loading a LLaMA model whose config.json has pretraining_tp > 1 (e.g. many released LLaMA-1/2 checkpoints were trained with tp=8) and then calling LlamaMLP.forward via model forward.

Common situations: Downloading a Hugging Face LLaMA checkpoint trained with tensor parallelism and running it without overriding pretraining_tp to 1 in the model config.

Related errors

• Either input_ids or inputs_embeds must be set
• Failed to generate a reply.
• {fieldType.Name}
• inputEmbeddings is not supported
• Invalid role.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/14b8ece9c183bd1b. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.GenAI.LLaMA/Module/LlamaMLP.cs:53

        this._hiddenSize = config.HiddenSize;
        this._intermediateSize = config.IntermediateSize;
        this._hasBias = config.MlpBias;
        this._pretrainingTp = config.PretrainingTp;
        var hiddenAct = config.HiddenAct;
        this.gate_proj = new QuantizedLinear(this._hiddenSize, this._intermediateSize, hasBias: this._hasBias, dtype: config.DType);
        this.up_proj = new QuantizedLinear(this._hiddenSize, this._intermediateSize, hasBias: this._hasBias, dtype: config.DType);
        this.down_proj = new QuantizedLinear(this._intermediateSize, this._hiddenSize, hasBias: this._hasBias, dtype: config.DType);
        this.RegisterComponents();
        this.activation_fn = Core.Utils.GetActivation(hiddenAct);
    }

#pragma warning disable MSML_GeneralName // This name should be PascalCased
    public override Tensor forward(Tensor input)
#pragma warning restore MSML_GeneralName // This name should be PascalCased
    {
        if (this._pretrainingTp > 1)
        {
            throw new NotImplementedException("PretrainingTp > 1 is not supported yet.");
        }

        using var input1 = this.gate_proj.forward(input);
        using var input2 = this.activation_fn.forward(input1);
        using var input3 = input2 * this.up_proj.forward(input);
        return this.down_proj.forward(input3);
    }
}

View on GitHub (pinned to 7b76e69cf9)