{"record":{"id":"d91d99eb641ef5a6","repo":"apache/hadoop","slug":"error-while-reading-compressed-data","errorCode":null,"errorMessage":"Error while reading compressed data","messagePattern":"Error while reading compressed data","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java","lineNumber":196,"sourceCode":"   * Utility wrapper for reading from {@link InputStream}. It catches any errors\n   * thrown by the underlying stream (either IO or decompression-related), and\n   * re-throws as an IOException.\n   * \n   * @param is - InputStream to be read from\n   * @param buf - buffer the data is read into\n   * @param off - offset within buf\n   * @param len - amount of data to be read\n   * @return number of bytes read\n   * @throws IOException raised on errors performing I/O.\n   */\n  public static int wrappedReadForCompressedData(InputStream is, byte[] buf,\n      int off, int len) throws IOException {\n    try {\n      return is.read(buf, off, len);\n    } catch (IOException ie) {\n      throw ie;\n    } catch (Throwable t) {\n      throw new IOException(\"Error while reading compressed data\", t);\n    }\n  }\n\n  /**\n   * Reads len bytes in a loop.\n   *\n   * @param in InputStream to read from\n   * @param buf The buffer to fill\n   * @param off offset from the buffer\n   * @param len the length of bytes to read\n   * @throws IOException if it could not read requested number of bytes \n   * for any reason (including EOF)\n   */\n  public static void readFully(InputStream in, byte[] buf,\n      int off, int len) throws IOException {\n    int toRead = len;\n    while (toRead > 0) {\n      int ret = in.read(buf, off, toRead);","sourceCodeStart":178,"sourceCodeEnd":214,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java#L178-L214","documentation":"Thrown by IOUtils.wrappedReadForCompressedData when InputStream.read() raises something that is NOT an IOException (any other Throwable) while reading compressed data. The helper deliberately converts unexpected runtime errors — typically from native decompression code — into a regular IOException with the original throwable attached as the cause, because callers of Hadoop IO APIs only expect IOException.","triggerScenarios":"Reading from a stream backed by a native codec (zlib/gzip/snappy/lz4 via JNI) where the native library throws a RuntimeException or Error instead of an IOException — e.g. ArrayIndexOutOfBoundsException inside a decompressor after bad input, or an InternalError from a broken JNI state. IOExceptions are re-thrown unchanged; only non-IO Throwables get wrapped with this message.","commonSituations":"Corrupted compressed blocks or truncated gzip/snappy files being fed to a decompressor; a mismatched or broken native library (libhadoop.so / libsnappy.so) loaded at runtime; JVM/native version skew after an upgrade while reading old compressed data.","solutions":["Inspect the cause chain (e.getCause()) — the wrapped Throwable names the real failure (native lib error vs data corruption).","Validate the input data: decompress the source file standalone (gunzip -t, or hadoop fs -cat to a local file) to confirm it is not truncated or corrupt.","Run 'hadoop checknative -a' to verify the native compression libraries load and match the Hadoop build.","If the native path is unreliable, force the pure-Java path (e.g. -Dio.compression.codecs without native gzip, or set the codec to non-native) to see whether the error follows the native library."],"exampleFix":"// before: generic catch hides the real native failure\ntry { IOUtils.wrappedReadForCompressedData(in, buf, 0, len); }\ncatch (IOException e) { log.error(\"read failed\"); }\n\n// after: surface the wrapped cause to distinguish corruption from native-lib failure\ntry { IOUtils.wrappedReadForCompressedData(in, buf, 0, len); }\ncatch (IOException e) {\n  Throwable root = e.getCause() != null ? e.getCause() : e;\n  throw new IOException(\"compressed read failed, root cause: \" + root, e);\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  int n = IOUtils.wrappedReadForCompressedData(is, buf, off, len);\n} catch (IOException e) {\n  Throwable root = e.getCause() != null ? e.getCause() : e;\n  if (root instanceof Error || root instanceof RuntimeException) {\n    // native codec blew up, not an ordinary I/O problem\n    LOG.error(\"native decompressor failed\", root);\n    markSourceSuspect();\n  }\n  throw e;\n}","preventionTips":["Run 'hadoop checknative -a' at deploy time to confirm native codec libraries load correctly.","Keep the Hadoop build and native library versions in lockstep across the cluster.","Verify integrity of compressed inputs before processing (checksums, gunzip -t spot checks).","Treat any recurrence of this error on a node as a hardware/lib signal — quarantine the node's copy of the data and retry elsewhere."],"tags":["io","compression","native-library","hadoop-common"],"backgroundTag":"corrupt-compressed-data","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}