apache/iceberg · error · BadHeaderException

Unrecognized header bytes: 0x%02X 0x%02X

Error message

Unrecognized header bytes: 0x%02X 0x%02X

What it means

After reading the header, decode validates the first two magic bytes against IcebergEncoder.V1_HEADER. If they don't match, it throws BadHeaderException with the offending bytes formatted as hex. The data isn't in the Iceberg binary-encoded format this decoder expects.

Source

Thrown at core/src/main/java/org/apache/iceberg/data/avro/IcebergDecoder.java:140

    }

    throw new MissingSchemaException("Cannot resolve schema for fingerprint: " + fp);
  }

  @Override
  public D decode(InputStream stream, D reuse) throws IOException {
    byte[] header = HEADER_BUFFER.get();
    try {
      if (!readFully(stream, header)) {
        throw new BadHeaderException("Not enough header bytes");
      }
    } catch (IOException e) {
      throw new IOException("Failed to read header and fingerprint bytes", e);
    }

    if (IcebergEncoder.V1_HEADER[0] != header[0] || IcebergEncoder.V1_HEADER[1] != header[1]) {
      throw new BadHeaderException(
          String.format(
              Locale.ROOT, "Unrecognized header bytes: 0x%02X 0x%02X", header[0], header[1]));
    }

    RawDecoder<D> decoder = getDecoder(FP_BUFFER.get().getLong(2));

    try {
      return decoder.decode(stream, reuse);
    } catch (UncheckedIOException e) {
      throw new AvroRuntimeException(e);
    }
  }

  /**
   * Reads a buffer from a stream, making multiple read calls if necessary.
   *
   * @param stream an InputStream to read from
   * @param bytes a buffer
   * @return true if the buffer is complete, false otherwise (stream ended)

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Confirm the data was written by IcebergEncoder (V1 header) rather than plain Avro DataFileWriter.
2. Ensure the stream starts exactly at the beginning of the encoded record — do not skip/offset bytes.
3. Check writer/decoder version compatibility (header version mismatch).
4. Log the hex bytes from the message to identify the actual format.

Example fix

// before
InputStream in = new FileInputStream(avroContainerFile);
decoder.decode(in, null); // Unrecognized header bytes: 0x4F 0x62
// after
InputStream in = new FileInputStream(rawEncodedFile); // produced by IcebergEncoder
decoder.decode(in, null);

byte[] magic = new byte[2];
if (new DataInputStream(in).read(magic) != 2 || magic[0] != IcebergEncoder.V1_HEADER[0] || magic[1] != IcebergEncoder.V1_HEADER[1]) {
  throw new IllegalArgumentException("Not Iceberg raw-encoded data");
}

null

try { return decoder.decode(stream, reuse); } catch (BadHeaderException e) { throw new IllegalStateException("Stream is not Iceberg-encoded: " + e.getMessage(), e); }

When it happens

Trigger: Decoding a stream not produced by IcebergEncoder: raw Avro files, other binary formats, a stream offset by a few bytes, or a different encoding version's header.

Common situations: Pointing the decoder at an Avro object-container file instead of Iceberg raw-encoded data; starting the read mid-stream (skipping bytes); decoding output of an incompatible writer version.

Related errors

• Cannot resolve schema for fingerprint:
• Not enough header bytes
• Failed to read header and fingerprint bytes
• Decoding datum failed
• Cannot read manifest list file: %s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f75b934108be56af. Report an issue: GitHub.