apache/kafka · error · SchemaException

Missing value for field '${name}' which has no default value

Error message

Missing value for field '${name}' which has no default value.

What it means

Thrown by Schema.read when the schema was constructed with tolerateMissingFieldsWithDefaults=true, the ByteBuffer is exhausted before all fields are read, and the next field has no default value. This lenient mode lets readers accept older payloads that omit trailing optional fields, but a missing mandatory field is still fatal.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/protocol/types/Schema.java:113

     * values; otherwise, if the schema does not tolerate missing fields, or if missing fields
     * don't have a default value, a {@code SchemaException} is thrown to signify that mandatory
     * fields are missing.
     */
    @Override
    public Struct read(ByteBuffer buffer) {
        if (cachedStruct != null) {
            return cachedStruct;
        }
        Object[] objects = new Object[fields.length];
        for (int i = 0; i < fields.length; i++) {
            try {
                if (tolerateMissingFieldsWithDefaults) {
                    if (buffer.hasRemaining()) {
                        objects[i] = fields[i].def.type.read(buffer);
                    } else if (fields[i].def.hasDefaultValue) {
                        objects[i] = fields[i].def.defaultValue;
                    } else {
                        throw new SchemaException("Missing value for field '" + fields[i].def.name +
                                "' which has no default value.");
                    }
                } else {
                    objects[i] = fields[i].def.type.read(buffer);
                }
            } catch (Exception e) {
                throw new SchemaException("Error reading field '" + fields[i].def.name + "': " +
                                          (e.getMessage() == null ? e.getClass().getName() : e.getMessage()));
            }
        }
        return new Struct(this, objects);
    }

    /**
     * The size of the given record
     */
    @Override
    public int sizeOf(Object o) {

View on GitHub (pinned to c31c9215e1)

Solutions

1. Add a default value to the declared Field so omitted payloads decode (Field.for(...).withDefault(...)).
2. Ensure producers and consumers are on compatible versions; if a mandatory field was added, roll out producers before consumers.
3. If reading an older persisted payload, pin the reader's Schema to the version that wrote it instead of the latest.
4. Audit the message JSON specs and regenerate schemas (./gradlew processMessages) so nullable/optional fields carry defaults.

Example fix

// before - mandatory field added without default
new Schema(true,
    new Field("replica_id", INT32),
    new Field("new_mandatory", INT32))

// after - give it a default
new Schema(true,
    new Field("replica_id", INT32),
    new Field("new_mandatory", INT32).withDefault(0))

// You control the Schema definition. If a field may be absent on the wire,
// give it a default so read() falls back instead of throwing.
Schema schema = new Schema(
    Field.forInt32("required_field"),                       // mandatory
    Field.forInt32("optional_field").withDefault(0)         // absent-tolerant
);
// OR, if you cannot change the schema, ensure the buffer carries every field:
if (!buffer.hasRemaining()) {
    throw new EOFException("buffer ended before mandatory field");
}
// Prefer new Schema(true, fields...) so trailing optional fields with defaults
// are tolerated automatically:
Schema tolerant = new Schema(true, /* fields with defaults */ );

try {
    Struct s = (Struct) schema.read(buffer);
} catch (org.apache.kafka.common.protocol.types.SchemaException e) {
    if (e.getMessage().contains("has no default value")) {
        // buffer was shorter than the schema -> version/encoding mismatch.
        // Re-read with a tolerant schema or reject the frame.
        throw new IncompleteFrameException(e);
    }
    throw e;
}

When it happens

Trigger: Schema.read loops fields; when tolerateMissingFieldsWithDefaults is set, !buffer.hasRemaining() && !field.def.hasDefaultValue triggers the exception. Hit when a producer serializes fewer trailing fields than the reader's schema declares and one of the omitted fields lacks a default.

Common situations: Forward-incompatible schema evolution: a new mandatory field was added without a default while older clients still send the prior layout; an internal caller constructing a Schema(true, ...) for v0/v1 feature gating forgot to set defaults; test fixtures serialized with an older code path.

Related errors

• Array size ${size} cannot be negative
• Error reading array of size ${size}, only ${remaining} bytes
• This array is not nullable.
• Error writing field '${name}': ${detail}
• Error reading field '${name}': ${detail}

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/a7654e3cb2013a5f.json. Report an issue: GitHub.