risingwavelabs/risingwave · error

payload shorter than 18-byte glue header

Error message

payload shorter than 18-byte glue header

What it means

AWS Glue Schema Registry frames Avro payloads with an 18-byte header: 1 byte version (3), 1 byte compression, and 16 bytes of schema-version UUID. parse_avro_value checks payload.len() < 18 before parsing the header and bails when the buffer is too short, meaning the message is not Glue-framed Avro.

Source

Thrown at src/connector/src/parser/avro/parser.rs:129

                )?))
            }
            WriterSchemaCache::File => {
                // FIXME: we should not use `Reader` (file header) here. See comment above and https://github.com/risingwavelabs/risingwave/issues/12871
                let mut reader = Reader::with_schema(&self.schema.original_schema, payload)?;
                match reader.next() {
                    Some(Ok(v)) => Ok(Some(v)),
                    Some(Err(e)) => Err(e)?,
                    None => bail!("avro parse unexpected eof"),
                }
            }
            WriterSchemaCache::Glue(resolver) => {
                // <https://github.com/awslabs/aws-glue-schema-registry/blob/v1.1.20/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L59-L61>
                // byte 0:      header version = 3
                // byte 1:      compression: 0 = no compression; 5 = zlib (unsupported)
                // byte 2..=17: 16-byte UUID as schema version id
                // byte 18..:   raw avro payload
                if payload.len() < 18 {
                    bail!("payload shorter than 18-byte glue header");
                }
                if payload[0] != 3 {
                    bail!(
                        "Only support glue header version 3 but found {}",
                        payload[0]
                    );
                }
                if payload[1] != 0 {
                    bail!("Non-zero compression {} not supported", payload[1]);
                }
                let schema_version_id = uuid::Uuid::from_slice(&payload[2..18]).unwrap();
                let writer_schema = resolver.get_by_id(schema_version_id).await?;
                let mut raw_payload = &payload[18..];
                Ok(Some(from_avro_datum(
                    writer_schema.as_ref(),
                    &mut raw_payload,
                    Some(&self.schema.original_schema),
                )?))

View on GitHub (pinned to 6469eb736d)

Solutions

1. Configure the producer to use the AWS Glue Schema Registry Avro serializer
2. Verify the topic actually carries Glue-framed messages (first byte should be 0x03)
3. Send a valid test record through the producer and re-check
4. Switch the RW source config to the matching schema registry type (e.g. Confluent) if Glue is not actually used

Example fix

// producer before: AvroEncoder without Glue framing
// producer after: AWSKafkaAvroSerializer with schemaAutoRegistration, region and registryName set

function hasGlueHeader(buf) {
  return buf && buf.length >= 18;
}
// producer side: assert frames emitted by Glue serializer are >= 18 bytes of header + body

function isGlueFramed(buf) {
  return buf instanceof Uint8Array && buf.length >= 18 && buf[0] === 3 && buf[1] === 0;
}

match parse_result {
  Err(e) if e.to_string().contains("18-byte glue header") => {
    error!("producer is not using Glue Schema Registry serializer");
  }
  other => other?,
}

When it happens

Trigger: Avro source configured with Glue Schema Registry receiving a payload shorter than 18 bytes — e.g. plain unframed Avro, an empty message, or the wrong serializer on the producer.

Common situations: Producer not using the Glue Schema Registry serializer while the RW source expects Glue framing; test messages sent manually; wrong topic wired up with non-Glue producers.

Related errors

• Only support glue header version 3 but found {}
• unused FORMAT ENCODE option: key.message='{name}'
• Invalid flag: {0:b}
• unrecognized {} value {}
• schema invalid, record type required at top level of the sch

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/1d323a8555ad28f8. Report an issue: GitHub.