apache/iceberg · error · RuntimeIOException

Failed to encode value as UTF-8: %s

Error message

Failed to encode value as UTF-8: %s

What it means

Conversions.toByteBuffer serializes a primitive value per the Iceberg single-value serialization spec. For STRING values it UTF-8 encodes the CharSequence; if the input contains unpaired surrogates or otherwise invalid UTF-16 the encoder throws CharacterCodingException, which is wrapped in a RuntimeIOException with this message.

Source

Thrown at api/src/main/java/org/apache/iceberg/types/Conversions.java:115

        return ByteBuffer.allocate(1).put(0, (Boolean) value ? (byte) 0x01 : (byte) 0x00);
      case INTEGER:
      case DATE:
        return ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(0, (int) value);
      case LONG:
      case TIME:
      case TIMESTAMP:
      case TIMESTAMP_NANO:
        return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, (long) value);
      case FLOAT:
        return ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putFloat(0, (float) value);
      case DOUBLE:
        return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putDouble(0, (double) value);
      case STRING:
        CharBuffer buffer = CharBuffer.wrap((CharSequence) value);
        try {
          return ENCODER.get().encode(buffer);
        } catch (CharacterCodingException e) {
          throw new RuntimeIOException(e, "Failed to encode value as UTF-8: %s", value);
        }
      case UUID:
        return UUIDUtil.convertToByteBuffer((UUID) value);
      case FIXED:
      case BINARY:
        return (ByteBuffer) value;
      case DECIMAL:
        return ByteBuffer.wrap(((BigDecimal) value).unscaledValue().toByteArray());
      case VARIANT:
        // Produce a concatenated buffer of metadata and value
        Variant variant = (Variant) value;
        VariantMetadata variantMetadata = variant.metadata();
        VariantValue variantValue = variant.value();
        ByteBuffer variantBuffer =
            ByteBuffer.allocate(variantMetadata.sizeInBytes() + variantValue.sizeInBytes())
                .order(ByteOrder.LITTLE_ENDIAN);
        variantMetadata.writeTo(variantBuffer, 0);
        variantValue.writeTo(variantBuffer, variantMetadata.sizeInBytes());

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Fix the data source so strings are valid UTF-16/UTF-8 (re-decode bytes with the correct charset)
  2. Sanitize the value: encode to bytes replacing errors (new String(bytes, UTF_8) after a CharsetDecoder with REPLACE) before passing it in
  3. Catch RuntimeIOException at the write boundary and quarantine the offending record

Example fix

// before
ByteBuffer buf = Conversions.toByteBuffer(Types.StringType.get(), rawString);
// after
byte[] bytes = rawString.getBytes(StandardCharsets.UTF_8);
String clean = new String(bytes, StandardCharsets.UTF_8); // replaces malformed sequences
ByteBuffer buf = Conversions.toByteBuffer(Types.StringType.get(), clean);
Defensive patterns

Strategy: try-catch

Validate before calling

boolean valid = value.toString().chars().noneMatch(c -> Character.isSurrogate((char) c) && !Character.isHighSurrogate((char) c));

Type guard

boolean isEncodableUtf8(CharSequence s) { return StandardCharsets.UTF_8.newEncoder().canEncode(s); }

Try / catch

try { return Conversions.toByteBuffer(type, value); } catch (RuntimeIOException e) { log.error("Malformed string data", e); throw e; }

Prevention

When it happens

Trigger: Passing a String/CharSequence containing unpaired surrogates (e.g. from decoding bytes with the wrong charset or corrupted data) to Conversions.toByteBuffer(stringType, value).

Common situations: Reading string data written by systems that produced malformed UTF-16; Java strings built from byte[] with an incorrect charset like ISO-8859-1 treated as UTF-16; records ingested from binary sources.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b652df9e7b83e9e2. Report an issue: GitHub.