{"record":{"id":"ea6eea56d42892c1","repo":"prestodb/presto","slug":"failed-to-append-record","errorCode":null,"errorMessage":"Failed to append record","messagePattern":"Failed to append record","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"presto-kafka/src/main/java/com/facebook/presto/kafka/encoder/avro/AvroRowEncoder.java","lineNumber":146,"sourceCode":"    }\n\n    @Override\n    public byte[] toByteArray()\n    {\n        // make sure entire row has been updated with new values\n        checkArgument(currentColumnIndex == columnHandles.size(), format(\"Missing %d columns\", columnHandles.size() - currentColumnIndex + 1));\n\n        try {\n            byteArrayOutputStream.reset();\n            dataFileWriter.create(parsedSchema, byteArrayOutputStream);\n            dataFileWriter.append(record);\n            dataFileWriter.close();\n\n            resetColumnIndex(); // reset currentColumnIndex to prepare for next row\n            return byteArrayOutputStream.toByteArray();\n        }\n        catch (IOException e) {\n            throw new UncheckedIOException(\"Failed to append record\", e);\n        }\n    }\n\n    @Override\n    public void close()\n    {\n        try {\n            byteArrayOutputStream.close();\n        }\n        catch (IOException e) {\n            throw new UncheckedIOException(\"Failed to close ByteArrayOutputStream\", e);\n        }\n    }\n}\n","sourceCodeStart":128,"sourceCodeEnd":161,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-kafka/src/main/java/com/facebook/presto/kafka/encoder/avro/AvroRowEncoder.java#L128-L161","documentation":"AvroRowEncoder.toByteArray serializes the accumulated GenericRecord via an Avro DataFileWriter into an in-memory ByteArrayOutputStream and wraps any resulting IOException in UncheckedIOException('Failed to append record'). Because the streams are purely in-memory, this almost always means the row failed Avro schema validation: the record's field values are incompatible with the Avro schema supplied for the topic (wrong primitive type, null in a non-nullable field, or a name mismatch between column mappings and schema fields).","triggerScenarios":"Calling toByteArray() after appendColumnValue filled the record; DataFileWriter.append(record) throws IOException because a field value violates the Avro schema — e.g. an INTEGER column mapped to an Avro field of type string, a NULL placed into an Avro field without a union ['null', T], or column names in the Kafka mapping not matching the Avro schema field names.","commonSituations":"The 'dataSchema' URI in the Kafka topic definition points to an Avro schema whose field types/names do not line up with the mapped Presto columns; schema evolution (schema registry updated to a new version) made previously writable rows invalid; inserting NULLs into required Avro fields.","solutions":["Inspect the wrapped IOException's cause (getCause() gives Avro's AvroTypeException) to identify the exact field and mismatch, then fix the Kafka topic definition's dataSchema so field names and types match the mapped columns.","Ensure every mapped column name matches an Avro schema field exactly (case-sensitive) and that Presto column types align with Avro types (integer->int, bigint->long, double->double, real->float, boolean->boolean, varchar->string).","Make nullable Presto columns map to Avro unions with 'null' (e.g. ['null','string']) or filter out NULL rows before insert.","Regenerate/update the Avro schema after schema registry changes and restart workers so encoders pick up the matching schema.","Retry the write only after fixing the schema/values; retrying alone will not help since the failure is deterministic."],"exampleFix":"// before: Avro schema field does not accept nulls\n{\"name\": \"user_id\", \"type\": \"long\"}\n// after: allow null to match nullable Presto column\n{\"name\": \"user_id\", \"type\": [\"null\", \"long\"], \"default\": null}","handlingStrategy":"try-catch","validationCode":"// Validate mapping vs Avro schema before creating the encoder:\nfor (EncoderColumnHandle col : columnHandles) {\n    Schema.Field field = parsedSchema.getField(col.getName());\n    if (field == null) {\n        throw new IllegalArgumentException(\"Avro schema has no field for column: \" + col.getName());\n    }\n    // check block nullability vs field.schema() accepting \"null\" in its union\n}","typeGuard":"static boolean avroFieldMatches(Schema schema, String name, Class<?> valueClass) {\n    Schema.Field f = schema.getField(name);\n    if (f == null) return false;\n    Schema s = f.schema();\n    if (s.getType() == Schema.Type.UNION) {\n        return s.getTypes().stream().anyMatch(t -> t.getType().name().toLowerCase()\n            .equals(valueClass.getSimpleName().toLowerCase()));\n    }\n    return s.getType().name().toLowerCase().equals(valueClass.getSimpleName().toLowerCase());\n}","tryCatchPattern":"try {\n    byte[] bytes = encoder.toByteArray();\n    return bytes;\n} catch (UncheckedIOException e) {\n    Throwable cause = e.getCause();\n    throw new IllegalStateException(\n        \"Row violates the topic's Avro dataSchema; check field names/types/nullability: \"\n        + (cause != null ? cause.getMessage() : e.getMessage()), e);\n}","preventionTips":["Keep the topic definition's dataSchema in lockstep with the schema registry version and mapped column names.","Declare Avro fields that receive nullable Presto columns as unions including \"null\" with a default.","Verify Presto-to-Avro type alignment: integer->int, bigint->long, real->float, double->double, boolean->boolean, varchar->string.","Add a smoke test that encodes one representative row (including NULLs) after any schema change."],"tags":["avro","kafka","presto","serialization","io"],"backgroundTag":"avro-schema-mismatch","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}