prestodb/presto · error · IOException

Failed to read %d bytes from file %s : %d bytes read.

Error message

Failed to read %d bytes from file %s : %d bytes read.

What it means

HdfsCachedInputFile.readFully reads a file in chunkSize chunks into buffers. If a read returns fewer bytes than requested before the expected total is consumed, it throws an IOException stating how many of the requested bytes were actually read. This surfaces as an under-read from the underlying HDFS/object stream — the file is shorter or the stream failed mid-read.

Source

Thrown at presto-iceberg/src/main/java/com/facebook/presto/iceberg/HdfsCachedInputFile.java:108

    }

    private static ManifestFileCachedContent readFully(InputFile input, long fileLength, long chunkSize)
            throws IOException
    {
        try (SeekableInputStream stream = input.newStream()) {
            long totalBytesToRead = fileLength;
            List<ByteBuffer> buffers = new ArrayList<>(
                    ((int) (fileLength / chunkSize)) +
                            (fileLength % chunkSize == 0 ? 0 : 1));

            while (totalBytesToRead > 0) {
                int bytesToRead = (int) Math.min(chunkSize, totalBytesToRead);
                byte[] buf = new byte[bytesToRead];
                int bytesRead = readRemaining(stream, buf, 0, bytesToRead);
                totalBytesToRead -= bytesRead;

                if (bytesRead < bytesToRead) {
                    throw new IOException(
                            format("Failed to read %d bytes from file %s : %d bytes read.",
                                    fileLength, input.location(), fileLength - totalBytesToRead));
                }
                else {
                    buffers.add(ByteBuffer.wrap(buf));
                }
            }
            return new ManifestFileCachedContent(buffers, fileLength);
        }
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

1. Verify the file exists and its actual size matches the length the InputFile reports (ls/stat the path).
2. Retry the read — transient HDFS/network failures often resolve; clear any stale caches.
3. Re-generate the file: rewrite manifests/metadata (rewrite_manifests) or restore from snapshot backup if truncated.
4. Check for concurrent writers deleting/overwriting the file; freeze table maintenance during reads.

Example fix

// before: cached length stale after truncation
InputFile f = new HdfsCachedInputFile(...); // length from old metadata
// after: validate length before reading
long actual = environment.getFileSystem(context, path).getFileStatus(path).getLen();
if (actual < expectedLength) { throw new PrestoException(ICEBERG_FILESYSTEM_ERROR, "File truncated: " + path); }

FileStatus st = fs.getFileStatus(path);
if (st.getLen() < expectedLength) throw new PrestoException(ICEBERG_FILESYSTEM_ERROR, "File truncated: " + path);

boolean isReadable(InputFile f) { try { return f.length() > 0 && f.newStream().read() >= 0; } catch (IOException e) { return false; } }

try { stream.readFully(buffer); } catch (IOException e) { if (isTransient(e)) retryWithBackoff(); else throw new PrestoException(ICEBERG_FILESYSTEM_ERROR, "under-read: " + path, e); }

When it happens

Trigger: Calling readFully (e.g. while reading Iceberg metadata/manifest/footer bytes through the cache layer) when the underlying stream ends early: file truncated, concurrent overwrite/delete of the file, or HDFS/network read failure returning short reads that readRemaining cannot fill.

Common situations: Files truncated by failed writes, objects deleted or overwritten concurrently by compaction/cleanup jobs, network faults to HDFS or S3, misreported file length (cached length larger than actual object size).

Related errors

• DRUID_DEEP_STORAGE_ERROR
• HIVE_FILESYSTEM_ERROR
• HIVE_FILESYSTEM_ERROR
• HIVE_FILESYSTEM_ERROR
• HIVE_BAD_DATA

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/97e27f8f989927fa. Report an issue: GitHub.