{"record":{"id":"0c1a870da9e229ba","repo":"deeplearning4j/deeplearning4j","slug":"fatal-serialization-error-shape-buffer-data-type","errorCode":null,"errorMessage":"FATAL SERIALIZATION ERROR: SHAPE BUFFER DATA TYPE MISMATCH for array '%s'. Shape buffer indicates DataType: %s, but array reports DataType: %s. Full ShapeInfo: %s. This indicates CORRUPT array metadata from ONNX import or array creation process. TERMINATING PROCESS TO PREVENT DATA CORRUPTION.","messagePattern":"FATAL SERIALIZATION ERROR: SHAPE BUFFER DATA TYPE MISMATCH for array '(.+?)'\\. Shape buffer indicates DataType: (.+?), but array reports DataType: (.+?)\\. Full ShapeInfo: (.+?)\\. This indicates CORRUPT array metadata from ONNX import or array creation process\\. TERMINATING PROCESS TO PREVENT DATA CORRUPTION\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/autodiff/samediff/serde/SameDiffSerializer.java","lineNumber":2572,"sourceCode":"                // The extras value is typically encoded in the shape buffer\n                // Try to extract the data type from the shape buffer using ArrayOptionsHelper\n                try {\n                    // Use the shape info to validate the data type encoding\n                    DataType extractedType = ArrayOptionsHelper.dataType(shapeInfo);\n                    if (extractedType == null || extractedType == DataType.UNKNOWN) {\n                        throw new IllegalStateException(String.format(\n                                \"FATAL SERIALIZATION ERROR: INVALID SHAPE BUFFER DETECTED for array '%s'. \" +\n                                        \"ArrayOptionsHelper.dataType() returned NULL or UNKNOWN. \" +\n                                        \"Full ShapeInfo: %s. \" +\n                                        \"Expected DataType: %s. \" +\n                                        \"This indicates CORRUPT array metadata from ONNX import or array creation process. \" +\n                                        \"TERMINATING PROCESS TO PREVENT DATA CORRUPTION.\",\n                                varNameForLog, Arrays.toString(shapeInfo), arr.dataType()));\n                    }\n\n                    // Additional validation: extracted type should match array's reported type\n                    if (extractedType != arr.dataType()) {\n                        throw new IllegalStateException(String.format(\n                                \"FATAL SERIALIZATION ERROR: SHAPE BUFFER DATA TYPE MISMATCH for array '%s'. \" +\n                                        \"Shape buffer indicates DataType: %s, \" +\n                                        \"but array reports DataType: %s. \" +\n                                        \"Full ShapeInfo: %s. \" +\n                                        \"This indicates CORRUPT array metadata from ONNX import or array creation process. \" +\n                                        \"TERMINATING PROCESS TO PREVENT DATA CORRUPTION.\",\n                                varNameForLog, extractedType, arr.dataType(), Arrays.toString(shapeInfo)));\n                    }\n\n                    log.info(\"SERIALIZE_VALIDATION [{}]: Shape buffer validation PASSED. DataType: {}\", varNameForLog, extractedType);\n\n                } catch (ND4JUnknownDataTypeException e) {\n                    throw new IllegalStateException(String.format(\n                            \"FATAL SERIALIZATION ERROR: INVALID DATA TYPE ENCODING in shape buffer for array '%s'. \" +\n                                    \"Full ShapeInfo: %s. \" +\n                                    \"ArrayOptionsHelper.dataType() threw: %s. \" +\n                                    \"This indicates CORRUPT array metadata from ONNX import or array creation process. \" +\n                                    \"TERMINATING PROCESS TO PREVENT DATA CORRUPTION.\",","sourceCodeStart":2554,"sourceCodeEnd":2590,"githubUrl":"https://github.com/deeplearning4j/deeplearning4j/blob/4c22ac5fe4a8350d05d224e7f4499429f7f69c93/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/autodiff/samediff/serde/SameDiffSerializer.java#L2554-L2590","documentation":"SameDiffSerializer validates an INDArray's shape-info buffer before serializing it into a FlatBuffer. It extracts the DataType from the shape buffer via ArrayOptionsHelper.dataType() and requires it to equal the array's own reported dataType(); when the two disagree it throws this fatal IllegalStateException because the array metadata is inconsistent and serializing it would persist corrupt data. ND4J throws this deliberately to stop the process from writing a model file that could silently corrupt results later.","triggerScenarios":"Calling SameDiffSerializer (serializeSmallNdArrayToFlatBuffer / save) on an INDArray whose shapeInfoDataBuffer encodes a different DataType than arr.dataType() returns - typically after a buggy ONNX import path, manual mutation of the shape buffer, or an array produced by a native op that set options bits incorrectly.","commonSituations":"ONNX model import producing arrays whose dtype options in the shape buffer disagree with the Java-side reported type (e.g. model declares float but buffer bits say double); mixing ND4J backend versions or native binaries that encode DataType extras differently; arrays created or modified via reflection/unsafe code that edits shape info directly.","solutions":["Re-create the offending array from scratch with Nd4j.create(...) and the correct DataType instead of reusing the corrupt one","Re-run/check the ONNX import (SameDiff.fromONNX / ONNXOpMapper) with an updated nd4j/onnx-import version; upgrade ND4J backend jars so Java and native code agree on the DataType extras encoding","Inspect the shape buffer: print arr.shapeInfoDataBuffer().asLong() and verify the extras/dtype bits; if only the metadata is wrong, call arr.setShapeInfo or rewrap data with Nd4j.create(data, shape, order, correctType)","Verify all ND4J backend jars (nd4j-native-platform / nd4j-cuda-*) and the nd4j-api version match exactly in your build to avoid mixed native encodings"],"exampleFix":"// before: reusing a suspect imported array\nINDArray w = sameDiff.getVariable(\"onnx_weight\").array();\nSameDiffSerializer.serializeSmallNdArrayToFlatBuffer(w, builder);\n\n// after: rebuild the array with an explicit dtype before saving\nINDArray w = sameDiff.getVariable(\"onnx_weight\").array();\nINDArray wFixed = w.castTo(DataType.FLOAT); // creates fresh, consistent shape buffer\nSameDiffSerializer.serializeSmallNdArrayToFlatBuffer(wFixed, builder);","handlingStrategy":"validation","validationCode":"DataType fromBuf = null;\ntry { fromBuf = ArrayOptionsHelper.dataType(arr.shapeInfoDataBuffer().asLong()); }\ncatch (ND4JUnknownDataTypeException e) { throw new IllegalStateException(\"Corrupt shape buffer for \" + name, e); }\nif (fromBuf == null || fromBuf == DataType.UNKNOWN || fromBuf != arr.dataType()) {\n    arr = arr.castTo(DataType.FLOAT); // rebuild consistent metadata before serializing\n}","typeGuard":"boolean hasConsistentDtype(INDArray arr) {\n    if (arr == null || arr.shapeInfoDataBuffer() == null) return false;\n    try {\n        DataType t = ArrayOptionsHelper.dataType(arr.shapeInfoDataBuffer().asLong());\n        return t != null && t != DataType.UNKNOWN && t == arr.dataType();\n    } catch (Exception e) { return false; }\n}","tryCatchPattern":"try {\n    SameDiffSerializer.serializeSmallNdArrayToFlatBuffer(arr, builder);\n} catch (IllegalStateException e) {\n    if (e.getMessage().contains(\"SHAPE BUFFER DATA TYPE MISMATCH\")) {\n        // rebuild array with explicit dtype and retry once\n        INDArray fixed = arr.castTo(DataType.FLOAT);\n        SameDiffSerializer.serializeSmallNdArrayToFlatBuffer(fixed, builder);\n    } else throw e;\n}","preventionTips":["Call castTo(desiredType) on arrays coming from ONNX import before saving","Never edit shape info buffers directly; build arrays via Nd4j.create with an explicit DataType","Pin all nd4j-* dependencies to one exact version in your build","Validate freshly imported models once at load time with the same dtype check"],"tags":["nd4j","serialization","dtype-mismatch","onnx"],"backgroundTag":"dtype-mismatch","analyzedSha":"4c22ac5fe4a8350d05d224e7f4499429f7f69c93","analyzedAt":"2026-09-07T05:27:04.714Z","contentChangedAt":"2026-09-07T05:27:04.714Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}