prestodb/presto · error · UncheckedIOException

Failed to append record

Error message

Failed to append record

What it means

AvroRowEncoder.toByteArray serializes the accumulated GenericRecord via an Avro DataFileWriter into an in-memory ByteArrayOutputStream and wraps any resulting IOException in UncheckedIOException('Failed to append record'). Because the streams are purely in-memory, this almost always means the row failed Avro schema validation: the record's field values are incompatible with the Avro schema supplied for the topic (wrong primitive type, null in a non-nullable field, or a name mismatch between column mappings and schema fields).

Source

Thrown at presto-kafka/src/main/java/com/facebook/presto/kafka/encoder/avro/AvroRowEncoder.java:146

    }

    @Override
    public byte[] toByteArray()
    {
        // make sure entire row has been updated with new values
        checkArgument(currentColumnIndex == columnHandles.size(), format("Missing %d columns", columnHandles.size() - currentColumnIndex + 1));

        try {
            byteArrayOutputStream.reset();
            dataFileWriter.create(parsedSchema, byteArrayOutputStream);
            dataFileWriter.append(record);
            dataFileWriter.close();

            resetColumnIndex(); // reset currentColumnIndex to prepare for next row
            return byteArrayOutputStream.toByteArray();
        }
        catch (IOException e) {
            throw new UncheckedIOException("Failed to append record", e);
        }
    }

    @Override
    public void close()
    {
        try {
            byteArrayOutputStream.close();
        }
        catch (IOException e) {
            throw new UncheckedIOException("Failed to close ByteArrayOutputStream", e);
        }
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

1. Inspect the wrapped IOException's cause (getCause() gives Avro's AvroTypeException) to identify the exact field and mismatch, then fix the Kafka topic definition's dataSchema so field names and types match the mapped columns.
2. Ensure every mapped column name matches an Avro schema field exactly (case-sensitive) and that Presto column types align with Avro types (integer->int, bigint->long, double->double, real->float, boolean->boolean, varchar->string).
3. Make nullable Presto columns map to Avro unions with 'null' (e.g. ['null','string']) or filter out NULL rows before insert.
4. Regenerate/update the Avro schema after schema registry changes and restart workers so encoders pick up the matching schema.
5. Retry the write only after fixing the schema/values; retrying alone will not help since the failure is deterministic.

Example fix

// before: Avro schema field does not accept nulls
{"name": "user_id", "type": "long"}
// after: allow null to match nullable Presto column
{"name": "user_id", "type": ["null", "long"], "default": null}

// Validate mapping vs Avro schema before creating the encoder:
for (EncoderColumnHandle col : columnHandles) {
    Schema.Field field = parsedSchema.getField(col.getName());
    if (field == null) {
        throw new IllegalArgumentException("Avro schema has no field for column: " + col.getName());
    }
    // check block nullability vs field.schema() accepting "null" in its union
}

static boolean avroFieldMatches(Schema schema, String name, Class<?> valueClass) {
    Schema.Field f = schema.getField(name);
    if (f == null) return false;
    Schema s = f.schema();
    if (s.getType() == Schema.Type.UNION) {
        return s.getTypes().stream().anyMatch(t -> t.getType().name().toLowerCase()
            .equals(valueClass.getSimpleName().toLowerCase()));
    }
    return s.getType().name().toLowerCase().equals(valueClass.getSimpleName().toLowerCase());
}

try {
    byte[] bytes = encoder.toByteArray();
    return bytes;
} catch (UncheckedIOException e) {
    Throwable cause = e.getCause();
    throw new IllegalStateException(
        "Row violates the topic's Avro dataSchema; check field names/types/nullability: "
        + (cause != null ? cause.getMessage() : e.getMessage()), e);
}

When it happens

Trigger: Calling toByteArray() after appendColumnValue filled the record; DataFileWriter.append(record) throws IOException because a field value violates the Avro schema — e.g. an INTEGER column mapped to an Avro field of type string, a NULL placed into an Avro field without a union ['null', T], or column names in the Kafka mapping not matching the Avro schema field names.

Common situations: The 'dataSchema' URI in the Kafka topic definition points to an Avro schema whose field types/names do not line up with the mapped Presto columns; schema evolution (schema registry updated to a new version) made previously writable rows invalid; inserting NULLs into required Avro fields.

Related errors

• KAFKA_SCHEMA_ERROR
• Failed to close ByteArrayOutputStream
• INVALID_FUNCTION_ARGUMENT
• NOT_SUPPORTED
• BIGQUERY_ERROR_END_OF_AVRO_BUFFER

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/ea6eea56d42892c1. Report an issue: GitHub.