apache/hadoop · error · IOException

Error while reading compressed data

Error message

Error while reading compressed data

What it means

Thrown by IOUtils.wrappedReadForCompressedData when InputStream.read() raises something that is NOT an IOException (any other Throwable) while reading compressed data. The helper deliberately converts unexpected runtime errors — typically from native decompression code — into a regular IOException with the original throwable attached as the cause, because callers of Hadoop IO APIs only expect IOException.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:196

   * Utility wrapper for reading from {@link InputStream}. It catches any errors
   * thrown by the underlying stream (either IO or decompression-related), and
   * re-throws as an IOException.
   * 
   * @param is - InputStream to be read from
   * @param buf - buffer the data is read into
   * @param off - offset within buf
   * @param len - amount of data to be read
   * @return number of bytes read
   * @throws IOException raised on errors performing I/O.
   */
  public static int wrappedReadForCompressedData(InputStream is, byte[] buf,
      int off, int len) throws IOException {
    try {
      return is.read(buf, off, len);
    } catch (IOException ie) {
      throw ie;
    } catch (Throwable t) {
      throw new IOException("Error while reading compressed data", t);
    }
  }

  /**
   * Reads len bytes in a loop.
   *
   * @param in InputStream to read from
   * @param buf The buffer to fill
   * @param off offset from the buffer
   * @param len the length of bytes to read
   * @throws IOException if it could not read requested number of bytes 
   * for any reason (including EOF)
   */
  public static void readFully(InputStream in, byte[] buf,
      int off, int len) throws IOException {
    int toRead = len;
    while (toRead > 0) {
      int ret = in.read(buf, off, toRead);

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the cause chain (e.getCause()) — the wrapped Throwable names the real failure (native lib error vs data corruption).
2. Validate the input data: decompress the source file standalone (gunzip -t, or hadoop fs -cat to a local file) to confirm it is not truncated or corrupt.
3. Run 'hadoop checknative -a' to verify the native compression libraries load and match the Hadoop build.
4. If the native path is unreliable, force the pure-Java path (e.g. -Dio.compression.codecs without native gzip, or set the codec to non-native) to see whether the error follows the native library.

Example fix

// before: generic catch hides the real native failure
try { IOUtils.wrappedReadForCompressedData(in, buf, 0, len); }
catch (IOException e) { log.error("read failed"); }

// after: surface the wrapped cause to distinguish corruption from native-lib failure
try { IOUtils.wrappedReadForCompressedData(in, buf, 0, len); }
catch (IOException e) {
  Throwable root = e.getCause() != null ? e.getCause() : e;
  throw new IOException("compressed read failed, root cause: " + root, e);
}

try {
  int n = IOUtils.wrappedReadForCompressedData(is, buf, off, len);
} catch (IOException e) {
  Throwable root = e.getCause() != null ? e.getCause() : e;
  if (root instanceof Error || root instanceof RuntimeException) {
    // native codec blew up, not an ordinary I/O problem
    LOG.error("native decompressor failed", root);
    markSourceSuspect();
  }
  throw e;
}

When it happens

Trigger: Reading from a stream backed by a native codec (zlib/gzip/snappy/lz4 via JNI) where the native library throws a RuntimeException or Error instead of an IOException — e.g. ArrayIndexOutOfBoundsException inside a decompressor after bad input, or an InternalError from a broken JNI state. IOExceptions are re-thrown unchanged; only non-IO Throwables get wrapped with this message.

Common situations: Corrupted compressed blocks or truncated gzip/snappy files being fed to a decompressor; a mismatched or broken native library (libhadoop.so / libsnappy.so) loaded at runtime; JVM/native version skew after an upgrade while reading old compressed data.

Related errors

• Unable to write to output stream.
• Premature EOF from inputStream
• Premature EOF from inputStream after skipping {len-amt} byte
• File/Directory {path} does not exist
• lz4-java library is not available: Lz4Compressor has not bee

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d91d99eb641ef5a6. Report an issue: GitHub.