prestodb/presto · warning · PrestoException

STORAGE_ERROR

STORAGE_ERROR

Error message

Unable to delete broadcast spill file

What it means

Thrown from PrestoSparkStorageBasedBroadcastDependency.destroy() when an IOException occurs while removing a broadcast spill file from the configured TempStorage. The broadcast table was spilled to temporary storage; during destruction each disk page's storage handle is deserialized and its file removed. Failure to delete indicates the underlying temp storage is unhealthy or the file is already gone.

Source

Thrown at presto-spark-base/src/main/java/com/facebook/presto/spark/PrestoSparkStorageBasedBroadcastDependency.java:137

    }

    @Override
    public void destroy()
    {
        if (broadcastVariable == null) {
            return;
        }

        try {
            // Delete the files
            for (PrestoSparkStorageHandle diskPage : broadcastVariable.getValue()) {
                TempStorageHandle storageHandle = tempStorage.deserialize(diskPage.getSerializedStorageHandle());
                tempStorage.remove(tempDataOperationContext, storageHandle);
                log.info("Deleted broadcast spill file: " + storageHandle.toString());
            }
        }
        catch (IOException e) {
            throw new PrestoException(STORAGE_ERROR, "Unable to delete broadcast spill file", e);
        }

        // Destroy broadcast variable
        broadcastVariable.destroy();
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

1. Check the wrapped IOException cause to identify whether the file is missing, access is denied, or the storage backend is down
2. Verify the TempStorage configuration (base directory, credentials, connectivity) on all Spark workers
3. Ensure no external retention/cleanup jobs delete temp storage files while queries are active
4. Retry the query; cleanup failure usually does not affect results, only leaks the spill file

Example fix

// before: destroy aborts on missing file
TempStorageHandle storageHandle = tempStorage.deserialize(diskPage.getSerializedStorageHandle());
tempStorage.remove(tempDataOperationContext, storageHandle);
// after: tolerate already-deleted files by pre-checking/catching in storage layer (or handle at cleanup job level)
if (tempStorage.exists(tempDataOperationContext, storageHandle)) {
    tempStorage.remove(tempDataOperationContext, storageHandle);
}

// verify temp storage is reachable before the query runs
TempStorage ts = ...;
// attempt a probe write/read/delete
ts.create(new TempDataOperationContext(...), OptionalLong.empty());

null

try {
    dependency.destroy();
}
catch (PrestoException e) {
    if (STORAGE_ERROR.toErrorCode().getCode().equals(e.getErrorCode().getCode())) {
        log.warn(e, "broadcast spill cleanup failed; file may leak");
        // do not fail the query for cleanup issues
    } else { throw e; }
}

When it happens

Trigger: Calling destroy() on a storage-based broadcast dependency when tempStorage.remove() fails due to an IOException — e.g. the spill file was already deleted, the storage backend is unreachable, or permissions are wrong.

Common situations: HDFS/S3/local-disk temp storage cleanup failures at query teardown; external cleanup jobs deleting spill files while the query is still running; misconfigured temp storage credentials or disk-full conditions on workers.

Related errors

• HIVE_WRITER_DATA_ERROR
• GENERIC_SPILL_FAILURE
• could not close some single stream spillers
• GENERIC_SPILL_FAILURE
• UNSUPPORTED_STORAGE_TYPE

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/c8833159bab1a2b0. Report an issue: GitHub.