java-native-access/jna · error · X11Exception

Invalid return format

Error message

Invalid return format

What it means

Window.getProperty validates the format field returned by XGetWindowProperty. Only 32, 16, 8 and 0 bits-per-unit are handled (mapped to sizes via Native.LONG_SIZE); any other format value triggers "Invalid return format". This indicates a protocol-level anomaly or corrupted/unsupported property reply.

Solutions

1. Update JNA and the x11contrib bindings to current versions
2. Test against a standard X server (Xorg/Xvfb) to isolate proxy/nested-server quirks
3. Catch X11Exception around property reads and fall back to another property source
4. If persistent, dump the raw reply with xprop and compare formats to confirm server misbehavior

Example fix

// before
byte[] data = win.getProperty(type, name); // throws on odd format
// after
byte[] data;
try {
    data = win.getProperty(type, name);
} catch (X11Exception e) {
    if (!e.getMessage().contains("Invalid return format")) throw e;
    data = new byte[0];
}

// standard X server sanity check before bulk property reads
Process p = Runtime.getRuntime().exec(new String[]{"xdpyinfo","-display",System.getenv("DISPLAY")});
boolean healthy = p.waitFor() == 0;

try {
    data = win.getProperty(type, atom);
} catch (X11Exception e) {
    if (e.getMessage().contains("Invalid return format")) {
        data = retryOnceOrEmpty(); // suspect nonstandard server/proxy
    } else throw e;
}

When it happens

Trigger: XGetWindowProperty replying with a format value outside {32,16,8,0} — effectively never from a conforming X server, so typically it signals misuse of the library or a broken X server/serialization bug.

Common situations: Buggy or nonstandard X servers/proxies (e.g. some VNC or nested-X implementations), memory corruption in the reply when mixing JNA struct versions, or reading a property whose definition changed between X protocol revisions.

Related errors

• Cannot get property.
• Attempting to retrieve VisualID from a null Visual
• Can't open X Display
• Can't open X Display
• Can't query subwindows

AI-assisted analysis of java-native-access/jna@d036ad9781 (2026-09-12). Data as JSON: /api/errors/3dfb9997af5ddb39. Report an issue: GitHub.

Appendix: source

Thrown at contrib/x11/src/jnacontrib/x11/api/X.java:1060

                String prop_name = x11.XGetAtomName(display.x11Display, xa_prop_name);
                throw new X11Exception("Invalid type of " + prop_name + " property");
            }

            int ret_format = ret_format_ref.getValue();
            long ret_nitems = ret_nitems_ref.getValue().longValue();

            // null terminate the result to make string handling easier
            int nbytes;
            if (ret_format == 32)
                nbytes = Native.LONG_SIZE;
            else if (ret_format == 16)
                nbytes = Native.LONG_SIZE / 2;
            else if (ret_format == 8)
                nbytes = 1;
            else if (ret_format == 0)
                nbytes = 0;
            else
                throw new X11Exception("Invalid return format");
            int length = Math.min((int) ret_nitems * nbytes, MAX_PROPERTY_VALUE_LEN);

            byte[] ret = ret_prop.getByteArray(0, length);

            x11.XFree(ret_prop);
            return ret;
        }

        /**
         * Returns the property value as a byte array.
         *
         * @param xa_prop_type property type
         * @param xa_prop_name property name
         * @return property value as a byte array
         * @throws X11Exception thrown if X11 window errors occurred
         */
        public byte[] getProperty(X11.Atom xa_prop_type, String xa_prop_name) throws X11Exception {
            return getProperty(xa_prop_type, display.getAtom(xa_prop_name));

View on GitHub (pinned to d036ad9781)