deeplearning4j/deeplearning4j · critical · IllegalStateException
FATAL SERIALIZATION ERROR: SHAPE BUFFER DATA TYPE MISMATCH…
Error message
FATAL SERIALIZATION ERROR: SHAPE BUFFER DATA TYPE MISMATCH for array '%s'. Shape buffer indicates DataType: %s, but array reports DataType: %s. Full ShapeInfo: %s. This indicates CORRUPT array metadata from ONNX import or array creation process. TERMINATING PROCESS TO PREVENT DATA CORRUPTION.
What it means
SameDiffSerializer validates an INDArray's shape-info buffer before serializing it into a FlatBuffer. It extracts the DataType from the shape buffer via ArrayOptionsHelper.dataType() and requires it to equal the array's own reported dataType(); when the two disagree it throws this fatal IllegalStateException because the array metadata is inconsistent and serializing it would persist corrupt data. ND4J throws this deliberately to stop the process from writing a model file that could silently corrupt results later.
Solutions
- Re-create the offending array from scratch with Nd4j.create(...) and the correct DataType instead of reusing the corrupt one
- Re-run/check the ONNX import (SameDiff.fromONNX / ONNXOpMapper) with an updated nd4j/onnx-import version; upgrade ND4J backend jars so Java and native code agree on the DataType extras encoding
- Inspect the shape buffer: print arr.shapeInfoDataBuffer().asLong() and verify the extras/dtype bits; if only the metadata is wrong, call arr.setShapeInfo or rewrap data with Nd4j.create(data, shape, order, correctType)
- Verify all ND4J backend jars (nd4j-native-platform / nd4j-cuda-*) and the nd4j-api version match exactly in your build to avoid mixed native encodings
Example fix
// before: reusing a suspect imported array
INDArray w = sameDiff.getVariable("onnx_weight").array();
SameDiffSerializer.serializeSmallNdArrayToFlatBuffer(w, builder);
// after: rebuild the array with an explicit dtype before saving
INDArray w = sameDiff.getVariable("onnx_weight").array();
INDArray wFixed = w.castTo(DataType.FLOAT); // creates fresh, consistent shape buffer
SameDiffSerializer.serializeSmallNdArrayToFlatBuffer(wFixed, builder); Defensive patterns
Strategy: validation
Validate before calling
DataType fromBuf = null;
try { fromBuf = ArrayOptionsHelper.dataType(arr.shapeInfoDataBuffer().asLong()); }
catch (ND4JUnknownDataTypeException e) { throw new IllegalStateException("Corrupt shape buffer for " + name, e); }
if (fromBuf == null || fromBuf == DataType.UNKNOWN || fromBuf != arr.dataType()) {
arr = arr.castTo(DataType.FLOAT); // rebuild consistent metadata before serializing
} Type guard
boolean hasConsistentDtype(INDArray arr) {
if (arr == null || arr.shapeInfoDataBuffer() == null) return false;
try {
DataType t = ArrayOptionsHelper.dataType(arr.shapeInfoDataBuffer().asLong());
return t != null && t != DataType.UNKNOWN && t == arr.dataType();
} catch (Exception e) { return false; }
} Try / catch
try {
SameDiffSerializer.serializeSmallNdArrayToFlatBuffer(arr, builder);
} catch (IllegalStateException e) {
if (e.getMessage().contains("SHAPE BUFFER DATA TYPE MISMATCH")) {
// rebuild array with explicit dtype and retry once
INDArray fixed = arr.castTo(DataType.FLOAT);
SameDiffSerializer.serializeSmallNdArrayToFlatBuffer(fixed, builder);
} else throw e;
} Prevention
- Call castTo(desiredType) on arrays coming from ONNX import before saving
- Never edit shape info buffers directly; build arrays via Nd4j.create with an explicit DataType
- Pin all nd4j-* dependencies to one exact version in your build
- Validate freshly imported models once at load time with the same dtype check
When it happens
Trigger: Calling SameDiffSerializer (serializeSmallNdArrayToFlatBuffer / save) on an INDArray whose shapeInfoDataBuffer encodes a different DataType than arr.dataType() returns - typically after a buggy ONNX import path, manual mutation of the shape buffer, or an array produced by a native op that set options bits incorrectly.
Common situations: ONNX model import producing arrays whose dtype options in the shape buffer disagree with the Java-side reported type (e.g. model declares float but buffer bits say double); mixing ND4J backend versions or native binaries that encode DataType extras differently; arrays created or modified via reflection/unsafe code that edits shape info directly.
Related errors
- FATAL SERIALIZATION ERROR: INVALID DATA TYPE ENCODING in…
- Append loading not supported for DataType
- Cannot get native pointer for fallback copy for variable
- Cannot JSON serialize array with datatype: " +…
- Could not create parent directories
AI-assisted analysis of deeplearning4j/deeplearning4j@4c22ac5fe4 (2026-09-07).
Data as JSON: /api/errors/0c1a870da9e229ba.
Report an issue: GitHub.
Appendix: source
Thrown at nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/autodiff/samediff/serde/SameDiffSerializer.java:2572
// The extras value is typically encoded in the shape buffer
// Try to extract the data type from the shape buffer using ArrayOptionsHelper
try {
// Use the shape info to validate the data type encoding
DataType extractedType = ArrayOptionsHelper.dataType(shapeInfo);
if (extractedType == null || extractedType == DataType.UNKNOWN) {
throw new IllegalStateException(String.format(
"FATAL SERIALIZATION ERROR: INVALID SHAPE BUFFER DETECTED for array '%s'. " +
"ArrayOptionsHelper.dataType() returned NULL or UNKNOWN. " +
"Full ShapeInfo: %s. " +
"Expected DataType: %s. " +
"This indicates CORRUPT array metadata from ONNX import or array creation process. " +
"TERMINATING PROCESS TO PREVENT DATA CORRUPTION.",
varNameForLog, Arrays.toString(shapeInfo), arr.dataType()));
}
// Additional validation: extracted type should match array's reported type
if (extractedType != arr.dataType()) {
throw new IllegalStateException(String.format(
"FATAL SERIALIZATION ERROR: SHAPE BUFFER DATA TYPE MISMATCH for array '%s'. " +
"Shape buffer indicates DataType: %s, " +
"but array reports DataType: %s. " +
"Full ShapeInfo: %s. " +
"This indicates CORRUPT array metadata from ONNX import or array creation process. " +
"TERMINATING PROCESS TO PREVENT DATA CORRUPTION.",
varNameForLog, extractedType, arr.dataType(), Arrays.toString(shapeInfo)));
}
log.info("SERIALIZE_VALIDATION [{}]: Shape buffer validation PASSED. DataType: {}", varNameForLog, extractedType);
} catch (ND4JUnknownDataTypeException e) {
throw new IllegalStateException(String.format(
"FATAL SERIALIZATION ERROR: INVALID DATA TYPE ENCODING in shape buffer for array '%s'. " +
"Full ShapeInfo: %s. " +
"ArrayOptionsHelper.dataType() threw: %s. " +
"This indicates CORRUPT array metadata from ONNX import or array creation process. " +
"TERMINATING PROCESS TO PREVENT DATA CORRUPTION.",View on GitHub (pinned to 4c22ac5fe4)