What causes "Invalid stencil operation!" in MonoGame/MonoGame?

Validate stencil operation values against Enum.IsDefined(typeof(StencilOperation), value) before assigning to the state object.

MonoGame/MonoGame · error · ArgumentException

Invalid stencil operation!

Error message

Invalid stencil operation!

What it means

Thrown by the DepthStencilState DirectX backend's StencilOperation-to-SharpDX mapping switch when the input value does not match any known case. The switch converts MonoGame's StencilOperation enum to SharpDX.Direct3D11.StencilOperation; hitting the default branch means the enum held an undefined or out-of-range value, usually due to uninitialized state or a bad cast from an integer.

Source

Thrown at MonoGame.Framework/Platform/Graphics/States/DepthStencilState.DirectX.cs:102

                    return SharpDX.Direct3D11.StencilOperation.Increment;

                case StencilOperation.IncrementSaturation:
                    return SharpDX.Direct3D11.StencilOperation.IncrementAndClamp;

                case StencilOperation.Invert:
                    return SharpDX.Direct3D11.StencilOperation.Invert;

                case StencilOperation.Keep:
                    return SharpDX.Direct3D11.StencilOperation.Keep;

                case StencilOperation.Replace:
                    return SharpDX.Direct3D11.StencilOperation.Replace;

                case StencilOperation.Zero:
                    return SharpDX.Direct3D11.StencilOperation.Zero;

                default:
                    throw new ArgumentException("Invalid stencil operation!");
            }
        }

        partial void PlatformDispose()
        {
            SharpDX.Utilities.Dispose(ref _state);
        }
    }
}

View on GitHub (pinned to 1d71bbd0ff)

Solutions

Validate stencil operation values against Enum.IsDefined(typeof(StencilOperation), value) before assigning to the state object.
Correct the configuration source to use a valid StencilOperation name (Keep, Zero, Replace, DecrementSaturation, Decrement, IncrementSaturation, Increment, Invert).
Default-construct the DepthStencilState and only override known fields.

Example fix

// before
state.StencilEnable = true;
state.StencilFail = (StencilOperation)42; // invalid → throws on apply

// after — use a defined enum value
state.StencilFail = StencilOperation.Keep;

Defensive patterns

Strategy: validation

Validate before calling

// Validate stencil op before assigning
static StencilOperation SafeStencilOp(int v)
    => Enum.IsDefined(typeof(StencilOperation), v) ? (StencilOperation)v : StencilOperation.Keep;

state.StencilEnable = true;
state.StencilFail = SafeStencilOp(rawStencilFail);

Type guard

static bool IsValidStencilOp(StencilOperation op) => Enum.IsDefined(typeof(StencilOperation), op);

Try / catch

try { GraphicsDevice.DepthStencilState = state; }
catch (ArgumentException ex) when (ex.Message.Contains("stencil operation"))
{
    state.StencilFail = StencilOperation.Keep;
    state.StencilDepthBufferFail = StencilOperation.Keep;
    state.StencilPass = StencilOperation.Keep;
    GraphicsDevice.DepthStencilState = state;
}

Prevention

Always assign named enum members, never cast raw ints.
Validate deserialized state with Enum.IsDefined.
Default-construct DepthStencilState and override only known fields.
Add unit tests for all enum members in your config format.

When it happens

Trigger: Setting DepthStencilState.StencilFail / StencilDepthBufferFail / StencilPass to a value not in the StencilOperation enum (e.g. (StencilOperation)999), or default(StencilState) where a field was left as raw zero but mis-cast. Custom state objects constructed from deserialized data with an invalid enum value.

Common situations: Reading stencil op from a binary/JSON config that had a typo or unknown token; casting an int from an external tool without range-checking; copy-paste of a numeric literal.

Related errors

AI-assisted analysis of MonoGame/MonoGame@1d71bbd0ff (2026-08-13). Data as JSON: /api/errors/325113c7da5e8f52. Report an issue: GitHub.