apache/kafka · error · SchemaException

Error writing field '${name}': ${detail}

Error message

Error writing field '${name}': ${detail}

What it means

Thrown by Schema.write when a field's Type.write or Type.validate raises any exception while serializing a Struct. The library wraps the cause's message (or class name if null) so the failing field is identifiable. It signals bad data being handed to the serializer rather than wire-level corruption.

Source

Thrown at clients/src/main/java/org/apache/kafka/common/protocol/types/Schema.java:86

            this.fieldsByName.put(def.name, this.fields[i]);
        }
        //6 schemas have no fields at the time of this writing (3 versions each of list_groups and api_versions)
        //for such schemas there's no point in even creating a unique Struct object when deserializing.
        this.cachedStruct = this.fields.length > 0 ? null : new Struct(this, NO_VALUES);
    }

    /**
     * Write a struct to the buffer
     */
    @Override
    public void write(ByteBuffer buffer, Object o) {
        Struct r = (Struct) o;
        for (BoundField field : fields) {
            try {
                Object value = field.def.type.validate(r.get(field));
                field.def.type.write(buffer, value);
            } catch (Exception e) {
                throw new SchemaException("Error writing field '" + field.def.name + "': " +
                                          (e.getMessage() == null ? e.getClass().getName() : e.getMessage()));
            }
        }
    }

    /**
     * Read a struct from the buffer. If this schema is configured to tolerate missing
     * optional fields at the end of the buffer, these fields are replaced with their default
     * values; otherwise, if the schema does not tolerate missing fields, or if missing fields
     * don't have a default value, a {@code SchemaException} is thrown to signify that mandatory
     * fields are missing.
     */
    @Override
    public Struct read(ByteBuffer buffer) {
        if (cachedStruct != null) {
            return cachedStruct;
        }
        Object[] objects = new Object[fields.length];

View on GitHub (pinned to c31c9215e1)

Solutions

1. Inspect the named field in the exception; confirm the Struct holds a Java value matching the field's Type (INT32=Integer, INT64=Long, STRING=String, BYTES=byte[]/ByteBuffer, ARRAY=Object[]).
2. Ensure required/non-nullable fields are populated before write; use Struct.set with the correct BoundField handle, not a String lookup typo.
3. Add a Struct.validate(...) call in tests before serialization to surface type errors earlier with a clearer message.
4. If a field type changed, regenerate message classes (./gradlew processMessages) and rebuild against the updated Schema.

Example fix

// before - wrong java type for an INT32 field
struct.set("timeout_ms", 30000L);

// after
struct.set("timeout_ms", Integer.valueOf(30000));

// Validate every field BEFORE Schema.write so you fail with a clear error
// instead of the library's wrapped "Error writing field" SchemaException.
Struct r = ...; // the struct you are about to serialize
for (BoundField f : schema.fields()) {
    Object v = r.get(f);
    if (v == null && !f.def.type.isNullable()) {
        throw new IllegalArgumentException(
            "Field '" + f.def.name + "' is null but its type is non-nullable");
    }
    f.def.type.validate(v); // throws SchemaException with field-specific reason
}
schema.write(buffer, r);

// Narrow to Struct and ensure each field matches its declared Type before write.
static boolean isWriteableStruct(Object o, Schema schema) {
    if (!(o instanceof Struct)) return false;
    Struct s = (Struct) o;
    for (BoundField f : schema.fields()) {
        Object v = s.get(f);
        if (v == null) { if (!f.def.type.isNullable()) return false; continue; }
        try { f.def.type.validate(v); } catch (Exception e) { return false; }
    }
    return true;
}

try {
    schema.write(buffer, struct);
} catch (org.apache.kafka.common.protocol.types.SchemaException e) {
    if (e.getMessage().startsWith("Error writing field '")) {
        // 'detail' suffix is the underlying cause (often null on non-nullable,
        // wrong type, or index out of bounds). Re-validate the named field.
        throw new SerializationPrecheckFailedException(e.getMessage(), e);
    }
    throw e;
}

When it happens

Trigger: Schema.write loops over BoundFields; for each it calls field.def.type.validate(r.get(field)) then type.write(buffer, value). A ClassCastException (wrong Java type in the Struct), NullPointerException (null for a non-nullable type), or anything thrown by a nested Type (e.g. STRING/INT32/ArrayOf) lands here.

Common situations: Putting a Long into an INT32 field, a String into a BYTES field, null into a non-nullable type, or an Object[] of the wrong component type into an ArrayOf; building a Struct by name with a typo so a field stays unset; refactoring a Struct's field types without updating call sites.

Related errors

• Array size ${size} cannot be negative
• Error reading field '${name}': ${detail}
• Buffer underflow while parsing consumer protocol's header
• Malformed consumer protocol subscription
• Malformed consumer protocol assignment

AI-assisted analysis of apache/kafka@c31c9215e1 (2026-08-03). Data as JSON: /data/errors/4c9f226935a5d928.json. Report an issue: GitHub.