prestodb/presto · error · IllegalArgumentException

unexpected internal column '%s'

Error message

unexpected internal column '%s'

What it means

KafkaPageSinkProvider.createPageSink refuses to build a Kafka insert handle when any column in the table handle is flagged as internal. Internal columns are metadata-only pseudo columns; they have no encoder mapping and cannot be serialized to Kafka, so the connector fails fast with an IllegalArgumentException rather than silently dropping or mis-encoding data.

Source

Thrown at presto-kafka/src/main/java/com/facebook/presto/kafka/KafkaPageSinkProvider.java:71

    }

    @Override
    public ConnectorPageSink createPageSink(ConnectorTransactionHandle transactionHandle, ConnectorSession session, ConnectorOutputTableHandle outputTableHandle, PageSinkContext pageSinkContext)
    {
        throw new UnsupportedOperationException("Table creation is not supported by the kafka connector");
    }

    @Override
    public ConnectorPageSink createPageSink(ConnectorTransactionHandle transactionHandle, ConnectorSession session, ConnectorInsertTableHandle insertTableHandle, PageSinkContext pageSinkContext)
    {
        requireNonNull(insertTableHandle, "tableHandle is null");
        KafkaTableHandle handle = (KafkaTableHandle) insertTableHandle;

        ImmutableList.Builder<EncoderColumnHandle> keyColumns = ImmutableList.builder();
        ImmutableList.Builder<EncoderColumnHandle> messageColumns = ImmutableList.builder();
        handle.getColumns().forEach(col -> {
            if (col.isInternal()) {
                throw new IllegalArgumentException(format("unexpected internal column '%s'", col.getName()));
            }
            if (col.isKeyCodec()) {
                keyColumns.add(col);
            }
            else {
                messageColumns.add(col);
            }
        });

        RowEncoder keyEncoder = encoderFactory.create(
                session,
                handle.getKeyDataFormat(),
                getDataSchema(handle.getKeyDataSchemaLocation()),
                keyColumns.build());

        RowEncoder messageEncoder = encoderFactory.create(
                session,
                handle.getMessageDataFormat(),

View on GitHub (pinned to 55bb57d202)

Solutions

1. Remove internal columns from the insert column list (SELECT only non-internal columns in the INSERT)
2. Regenerate the table description so column handles are built by the connector, not manually
3. Check for an upgraded metadata plugin producing internal column handles and file/verify against the connector version

Example fix

// before
INSERT INTO kafka_table SELECT rowkey, _partition_id, payload FROM staging;
// after
INSERT INTO kafka_table SELECT rowkey, payload FROM staging;

// before INSERT, ensure no internal columns are projected
SELECT column_name FROM information_schema.columns
WHERE table_name='kafka_table' AND column_name NOT LIKE '\_%';

boolean isWritable(KafkaColumnHandle c) { return !c.isInternal(); }

null

When it happens

Trigger: Calling ConnectorPageSinkProvider.createPageSink with a KafkaTableHandle whose columns list contains a KafkaColumnHandle with isInternal()==true, i.e. an INSERT/CREATE TABLE AS against a Kafka table where internal column handles leaked into the insert projection.

Common situations: Metadata plugins or custom table descriptors expose hidden/internal columns; a query planner bug or hand-built KafkaTableHandle includes internal columns; version drift where a column that used to be plain is now marked internal.

Related errors

• KAFKA_SCHEMA_ERROR
• KAFKA_SPLIT_ERROR
• GENERIC_INTERNAL_ERROR
• Invalid Kafka Offset start/end pair: %s - %s
• KAFKA_SPLIT_ERROR

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/a9820699e0571647. Report an issue: GitHub.