risingwavelabs/risingwave · error

avro parse unexpected eof

Error message

avro parse unexpected eof

What it means

In the file-writer-schema avro parser path, the payload is decoded with apache-avro's Reader bound to the declared schema. If the iterator returns None, the buffer contained no decodable record — the payload was empty or exhausted. The parser bails with 'avro parse unexpected eof' instead of returning an empty record.

Solutions

1. Verify the producer is emitting complete Avro records (check serialization on the producer side)
2. Inspect and fix the message size/truncation in the message queue
3. Confirm the payload actually matches the configured file schema
4. If using Confluent/Glue, ensure the correct schema-cache mode is configured rather than File

Example fix

// before
// payload: empty byte array -> error
// after: check upstream producer writes at least one record per message, or skip empty payloads in the producer

function validateAvroPayload(buf) {
  if (!buf || buf.length === 0) throw new Error('empty avro payload; producer must serialize a record');
}
// call before producing to the topic consumed by RW

function looksLikeAvroRecord(buf) {
  return buf instanceof Uint8Array && buf.length > 0;
}

match parser.parse(payload, _unused) {
  Err(e) if e.to_string().contains("unexpected eof") => {
    warn!("skipping malformed/empty avro message");
  }
  other => other?,
}

When it happens

Trigger: Feeding an empty or truncated Avro message (with schema from file, i.e. schema.registry... file-based writer schema) to an Avro parser; the Reader cannot produce even one value from the payload.

Common situations: Producer wrote empty frames; message truncation during transport; schema/payload mismatch causing zero records to decode; misconfigured topic receiving non-Avro bytes.

Related errors

• Avro error
• {0}
• {0}
• backfill order strategy
• BOTH, LEADING, or TRAILING

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/86fcbf08c7b030a8. Report an issue: GitHub.

Appendix: source

Thrown at src/connector/src/parser/avro/parser.rs:119

        // parse payload to avro value
        // if use confluent schema, get writer schema from confluent schema registry
        match &self.writer_schema_cache {
            WriterSchemaCache::Confluent(resolver) => {
                let (schema_id, mut raw_payload) = extract_schema_id(payload)?;
                let writer_schema = resolver.get_by_id(schema_id).await?;
                Ok(Some(from_avro_datum(
                    writer_schema.as_ref(),
                    &mut raw_payload,
                    Some(&self.schema.original_schema),
                )?))
            }
            WriterSchemaCache::File => {
                // FIXME: we should not use `Reader` (file header) here. See comment above and https://github.com/risingwavelabs/risingwave/issues/12871
                let mut reader = Reader::with_schema(&self.schema.original_schema, payload)?;
                match reader.next() {
                    Some(Ok(v)) => Ok(Some(v)),
                    Some(Err(e)) => Err(e)?,
                    None => bail!("avro parse unexpected eof"),
                }
            }
            WriterSchemaCache::Glue(resolver) => {
                // <https://github.com/awslabs/aws-glue-schema-registry/blob/v1.1.20/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L59-L61>
                // byte 0:      header version = 3
                // byte 1:      compression: 0 = no compression; 5 = zlib (unsupported)
                // byte 2..=17: 16-byte UUID as schema version id
                // byte 18..:   raw avro payload
                if payload.len() < 18 {
                    bail!("payload shorter than 18-byte glue header");
                }
                if payload[0] != 3 {
                    bail!(
                        "Only support glue header version 3 but found {}",
                        payload[0]
                    );
                }
                if payload[1] != 0 {

View on GitHub (pinned to 6469eb736d)