apache/iceberg · error · IllegalStateException

Invalid file: expected magic at offset

Error message

Invalid file: expected magic at offset %s: %s, but got %s

What it means

checkMagic validates that the expected Puffin magic bytes appear at a required offset (start of file, or computed footer start, depending on the caller: fileMetadata, footerSize). When the bytes do not match it throws this IllegalStateException reporting the offset, expected, and actual bytes. It means the file is not a valid Puffin file or the footer offset is wrong.

Solutions

1. Confirm the file is a genuine Puffin file (it must begin with the Puffin magic)
2. Check for truncation/corruption; re-export or rewrite the file
3. Verify the file was not modified after writing (footer offset depends on trailing bytes)
4. Inspect bytes at the reported offset with a hex viewer

byte[] head = readFirstBytes(file, 4);
if (!Arrays.equals(head, expectedPuffinMagic)) {
  throw new IllegalArgumentException("Not a puffin file: " + file);
}

try {
  reader.fileMetadata();
} catch (IllegalStateException e) {
  if (e.getMessage().contains("expected magic")) {
    // treat as non-puffin file
  }
}

When it happens

Trigger: Calling PuffinReader.fileMetadata or footerSize on a file whose leading bytes, or bytes at the computed footer start offset, do not equal the Puffin magic.

Common situations: Reading a non-Puffin file (e.g. plain Avro) as Puffin; truncated or corrupted files where the footer offset is miscalculated; files modified after writing.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• could not decode the dictionary for
• could not read levels in page for col
• could not read page in col
• Failed to read a byte
• Not an Avro file

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/88a903acc29c841a. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/puffin/PuffinReader.java:154

                    input.seek(blobMetadata.offset());
                    byte[] bytes = new byte[Math.toIntExact(blobMetadata.length())];
                    ByteStreams.readFully(input, bytes);
                    ByteBuffer rawData = ByteBuffer.wrap(bytes);
                    PuffinCompressionCodec codec =
                        PuffinCompressionCodec.forName(blobMetadata.compressionCodec());
                    ByteBuffer data = PuffinFormat.decompress(codec, rawData);
                    return Pair.of(blobMetadata, data);
                  } catch (IOException e) {
                    throw new UncheckedIOException(e);
                  }
                })
            .iterator();
  }

  private static void checkMagic(byte[] data, int offset) {
    byte[] read = Arrays.copyOfRange(data, offset, offset + MAGIC.length);
    if (!Arrays.equals(read, MAGIC)) {
      throw new IllegalStateException(
          String.format(
              "Invalid file: expected magic at offset %s: %s, but got %s",
              offset, Arrays.toString(MAGIC), Arrays.toString(read)));
    }
  }

  private int footerSize() throws IOException {
    if (knownFooterSize == null) {
      Preconditions.checkState(
          fileSize >= PuffinFormat.FOOTER_STRUCT_LENGTH,
          "Invalid file: file length %s is less than minimal length of the footer tail %s",
          fileSize,
          PuffinFormat.FOOTER_STRUCT_LENGTH);
      byte[] footerStruct =
          readInput(
              fileSize - PuffinFormat.FOOTER_STRUCT_LENGTH, PuffinFormat.FOOTER_STRUCT_LENGTH);
      checkMagic(footerStruct, PuffinFormat.FOOTER_STRUCT_MAGIC_OFFSET);

View on GitHub (pinned to 86d9c8fc54)