risingwavelabs/risingwave · error

avro parse unexpected eof

Error message

avro parse unexpected eof

What it means

In the file-writer-schema avro parser path, the payload is decoded with apache-avro's Reader bound to the declared schema. If the iterator returns None, the buffer contained no decodable record — the payload was empty or exhausted. The parser bails with 'avro parse unexpected eof' instead of returning an empty record.

Source

Thrown at src/connector/src/parser/avro/parser.rs:119

        // parse payload to avro value
        // if use confluent schema, get writer schema from confluent schema registry
        match &self.writer_schema_cache {
            WriterSchemaCache::Confluent(resolver) => {
                let (schema_id, mut raw_payload) = extract_schema_id(payload)?;
                let writer_schema = resolver.get_by_id(schema_id).await?;
                Ok(Some(from_avro_datum(
                    writer_schema.as_ref(),
                    &mut raw_payload,
                    Some(&self.schema.original_schema),
                )?))
            }
            WriterSchemaCache::File => {
                // FIXME: we should not use `Reader` (file header) here. See comment above and https://github.com/risingwavelabs/risingwave/issues/12871
                let mut reader = Reader::with_schema(&self.schema.original_schema, payload)?;
                match reader.next() {
                    Some(Ok(v)) => Ok(Some(v)),
                    Some(Err(e)) => Err(e)?,
                    None => bail!("avro parse unexpected eof"),
                }
            }
            WriterSchemaCache::Glue(resolver) => {
                // <https://github.com/awslabs/aws-glue-schema-registry/blob/v1.1.20/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L59-L61>
                // byte 0:      header version = 3
                // byte 1:      compression: 0 = no compression; 5 = zlib (unsupported)
                // byte 2..=17: 16-byte UUID as schema version id
                // byte 18..:   raw avro payload
                if payload.len() < 18 {
                    bail!("payload shorter than 18-byte glue header");
                }
                if payload[0] != 3 {
                    bail!(
                        "Only support glue header version 3 but found {}",
                        payload[0]
                    );
                }
                if payload[1] != 0 {

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the producer is emitting complete Avro records (check serialization on the producer side)
2. Inspect and fix the message size/truncation in the message queue
3. Confirm the payload actually matches the configured file schema
4. If using Confluent/Glue, ensure the correct schema-cache mode is configured rather than File

Example fix

// before
// payload: empty byte array -> error
// after: check upstream producer writes at least one record per message, or skip empty payloads in the producer

function validateAvroPayload(buf) {
  if (!buf || buf.length === 0) throw new Error('empty avro payload; producer must serialize a record');
}
// call before producing to the topic consumed by RW

function looksLikeAvroRecord(buf) {
  return buf instanceof Uint8Array && buf.length > 0;
}

match parser.parse(payload, _unused) {
  Err(e) if e.to_string().contains("unexpected eof") => {
    warn!("skipping malformed/empty avro message");
  }
  other => other?,
}

When it happens

Trigger: Feeding an empty or truncated Avro message (with schema from file, i.e. schema.registry... file-based writer schema) to an Avro parser; the Reader cannot produce even one value from the payload.

Common situations: Producer wrote empty frames; message truncation during transport; schema/payload mismatch causing zero records to decode; misconfigured topic receiving non-Avro bytes.

Related errors

• Avro error: {0}
• ser/de proto message error: {0}
• Pb decode error: {0}
• I/O error: {0}
• unrecognized {} value {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/86fcbf08c7b030a8. Report an issue: GitHub.