apache/flink · error · TableException

Exception in finalizeGlobal

Error message

Exception in finalizeGlobal

What it means

FileSystemOutputFormat.finalizeGlobal executes after all parallel subtasks finish, committing partitions and cleaning up the staging directory. Any exception during partition commitment (metaStoreFactory operations, commit logic, or finished-attempt checks) is wrapped in a TableException. The finally block always attempts to delete the staging directory regardless of success or failure.

Source

Thrown at flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemOutputFormat.java:160

                            stagingPath,
                            partitionColumns.length,
                            isToLocal,
                            identifier,
                            staticPartitions,
                            policies);
            committer.commitPartitions(
                    (subtaskIndex, attemptNumber) -> {
                        try {
                            if (context.getFinishedAttempt(subtaskIndex) == attemptNumber) {
                                return true;
                            }
                        } catch (IllegalArgumentException ignored) {
                            // maybe met a dir or file which does not belong to this job
                        }
                        return false;
                    });
        } catch (Exception e) {
            throw new TableException("Exception in finalizeGlobal", e);
        } finally {
            try {
                fsFactory.create(stagingPath.toUri()).delete(stagingPath, true);
            } catch (IOException ignore) {
            }
        }
    }

    @Override
    public void configure(Configuration parameters) {
        this.parameters = parameters;
    }

    @Override
    public void open(InitializationContext context) throws IOException {
        try {
            PartitionTempFileManager fileManager =
                    new PartitionTempFileManager(

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Examine the chained cause exception in the stack trace to identify the root failure (metastore, file system, or commit logic).
2. If the cause is metastore connectivity, ensure the metastore service is running and reachable from the JobManager.
3. If the cause is file system related, verify HDFS/S3 availability and that the output directory is writable.
4. Retry the job after resolving the underlying infrastructure issue; the staging directory is cleaned in finally.

try {
    outputFormat.finalizeGlobal(context);
} catch (TableException e) {
    // Examine e.getCause() for the root failure
    Throwable root = e.getCause();
    LOG.error("Finalize failed due to: {}", root.getMessage(), root);
    // Depending on root cause: retry, alert, or fail the job
    throw e;
}

When it happens

Trigger: An exception occurs during finalizeGlobal: the partition committer fails (e.g. metastore connection error in HiveMetastore), the file system operation during commit fails, or the context.getFinishedAttempt call logic encounters unexpected state. The original exception is chained as the cause of the TableException.

Common situations: Hive metastore is down or unreachable during global finalization. File system rename or commit operation fails (S3 eventual consistency, HDFS safe mode). Partition commit policy encounters an unexpected file structure. Network issue during the commit phase of a large batch write.

Related errors

• Exception in open
• Exception in writeRecord
• Exception in close
• An IO error occurred while accessing the staging FileSystem.
• Invalid time zone for '%s'. The value should be a Time Zone

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/a5e302a182933f24. Report an issue: GitHub.