What causes "Failed to RegisterClassExA, error {}" in rustdesk/rustdesk?

Reuse a single Magnifier capturer per process instead of constructing one per session/stream.

rustdesk/rustdesk · error · io::Error

Failed to RegisterClassExA, error {}

Error message

Failed to RegisterClassExA, error {}

What it means

RegisterClassExA failed with an error other than ERROR_CLASS_ALREADY_EXISTS (which is deliberately ignored so a class registered earlier in the process is reused). The host window class is required before creating the magnifier host window; other failures typically mean desktop heap exhaustion or a class-name collision with different attributes in the same process/session.

Source

Thrown at libs/scrap/src/dxgi/mag.rs:383

                cbSize: size_of::<WNDCLASSEXA>() as _,
                style: 0,
                lpfnWndProc: Some(DefWindowProcA),
                cbClsExtra: 0,
                cbWndExtra: 0,
                hInstance: instance,
                hIcon: 0 as _,
                hCursor: LoadCursorA(NULL as _, IDC_ARROW as _),
                hbrBackground: 0 as _,
                lpszClassName: s.magnifier_host_class.as_ptr() as _,
                lpszMenuName: 0 as _,
                hIconSm: 0 as _,
            };

            // Ignore the error which may happen when the class is already registered.
            if 0 == RegisterClassExA(&wcex) {
                let code = GetLastError();
                if code != ERROR_CLASS_ALREADY_EXISTS {
                    return Err(Error::new(
                        ErrorKind::Other,
                        format!(
                            "Failed to RegisterClassExA, error {}",
                            Error::from_raw_os_error(code as _)
                        ),
                    ));
                }
            }

            // Create the host window.
            s.host_window = CreateWindowExA(
                WS_EX_LAYERED,
                s.magnifier_host_class.as_ptr(),
                s.host_window_name.as_ptr(),
                WS_POPUP,
                0,
                0,
                0,

View on GitHub (pinned to 7aa98d43cf)

Solutions

Reuse a single Magnifier capturer per process instead of constructing one per session/stream.
Check the error code in the message: ERROR_NOT_ENOUGH_MEMORY points at desktop heap — raise the desktop heap size or move to the interactive session.
Rename the host/magnifier class names to something unique to your application.
Fall back to the DXGI capturer when class registration fails.

Defensive patterns

Strategy: try-catch

Type guard

fn is_register_class_error(e: &std::io::Error) -> bool {
    e.to_string().contains("RegisterClassExA")
}

Try / catch

let mag = match magnifier::Magnifier::new(origin, w, h) {
    Ok(m) => m,
    Err(e) if is_register_class_error(&e) => {
        log::error!("host class registration failed: {e}");
        return fallback_to_dxgi();
    }
    Err(e) => return Err(e),
};

Prevention

Create the magnifier capturer once per process and reuse it (ERROR_CLASS_ALREADY_EXISTS is already tolerated).
Check the error code: ERROR_NOT_ENOUGH_MEMORY points at desktop-heap exhaustion — reduce window churn or raise SharedSection's third value.
Use unique window-class names to avoid collisions with other software.

When it happens

Trigger: Creating many Magnifier instances (or leaking host classes) until the session desktop heap is exhausted; another component registering the same class name with different parameters; running in a restricted desktop/window station.

Common situations: Long-running capture processes repeatedly constructing and tearing down capturers; services or scheduled tasks running on a non-interactive desktop with a tiny desktop heap; co-existing software that uses the same window class name string.

Related errors

AI-assisted analysis of rustdesk/rustdesk@7aa98d43cf (2026-08-16). Data as JSON: /api/errors/c80d541e78dc0f7f. Report an issue: GitHub.