apache/flink · error · ValidationException

The 'protobuf' format is not supported for the 'filesystem'

Error message

The 'protobuf' format is not supported for the 'filesystem' connector.

What it means

PbFileFormatFactory implements FileSystemFormatFactory but protobuf decoding requires a complete serialized message per record, which the filesystem (bulk) connector cannot provide. Calling createDecodingFormat therefore always throws this ValidationException. It is a deliberate, documented limitation: the protobuf format only works with messaging connectors like Kafka.

Source

Thrown at flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/PbFileFormatFactory.java:73

    @Override
    public Set<ConfigOption<?>> requiredOptions() {
        return Collections.emptySet();
    }

    @Override
    public Set<ConfigOption<?>> optionalOptions() {
        return Collections.emptySet();
    }

    @Override
    public Set<ConfigOption<?>> forwardOptions() {
        return Collections.emptySet();
    }

    @Override
    public BulkDecodingFormat<RowData> createDecodingFormat(
            DynamicTableFactory.Context context, ReadableConfig formatOptions) {
        throw new ValidationException(
                "The 'protobuf' format is not supported for the 'filesystem' connector.");
    }

    @Override
    public EncodingFormat<Factory<RowData>> createEncodingFormat(
            DynamicTableFactory.Context context, ReadableConfig formatOptions) {
        throw new ValidationException(
                "The 'protobuf' format is not supported for the 'filesystem' connector.");
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Use the protobuf format with a message connector: 'connector'='kafka' (or kinesis/pulsar) instead of 'filesystem'.
2. If you must read length-delimited protobuf files, wrap them via a custom Source and PbRowDataDeserializationSchema.
3. Convert the data to a filesystem-supported format (orc/parquet/avro/csv/json) upstream.

Example fix

-- before
CREATE TABLE t (...) WITH (
  'connector'='filesystem',
  'path'='...',
  'format'='protobuf'
);
-- after
CREATE TABLE t (...) WITH (
  'connector'='kafka',
  'topic'='...',
  'properties.bootstrap.servers'='...',
  'format'='protobuf',
  'protobuf.message-class-name'='com.example.MyMsg'
);

-- Check before creating the table:
SELECT * FROM table_options; -- ensure connector is kafka-like when format='protobuf'

When it happens

Trigger: CREATE TABLE ... WITH ('connector'='filesystem', 'format'='protobuf'): during factory discovery the filesystem connector asks the format factory for a BulkDecodingFormat and this method throws.

Common situations: Copying a Kafka protobuf table DDL and only changing the connector to 'filesystem'; expecting to read protobuf payload files from object storage.

Related errors

• Table options do not contain an option key '%s' for discover
• Format cannot define a quote character and disabled quote ch
• fail-on-missing-field and ignore-parse-errors shouldn't both
• Column
• Unexpected LogicalType:

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/0dd375bfe4a5c75f. Report an issue: GitHub.