{"record":{"id":"b652df9e7b83e9e2","repo":"apache/iceberg","slug":"failed-to-encode-value-as-utf-8-s","errorCode":null,"errorMessage":"Failed to encode value as UTF-8: %s","messagePattern":"Failed to encode value as UTF-8: (.+?)","errorType":"exception","errorClass":"RuntimeIOException","httpStatus":null,"severity":"error","filePath":"api/src/main/java/org/apache/iceberg/types/Conversions.java","lineNumber":115,"sourceCode":"        return ByteBuffer.allocate(1).put(0, (Boolean) value ? (byte) 0x01 : (byte) 0x00);\n      case INTEGER:\n      case DATE:\n        return ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(0, (int) value);\n      case LONG:\n      case TIME:\n      case TIMESTAMP:\n      case TIMESTAMP_NANO:\n        return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putLong(0, (long) value);\n      case FLOAT:\n        return ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putFloat(0, (float) value);\n      case DOUBLE:\n        return ByteBuffer.allocate(8).order(ByteOrder.LITTLE_ENDIAN).putDouble(0, (double) value);\n      case STRING:\n        CharBuffer buffer = CharBuffer.wrap((CharSequence) value);\n        try {\n          return ENCODER.get().encode(buffer);\n        } catch (CharacterCodingException e) {\n          throw new RuntimeIOException(e, \"Failed to encode value as UTF-8: %s\", value);\n        }\n      case UUID:\n        return UUIDUtil.convertToByteBuffer((UUID) value);\n      case FIXED:\n      case BINARY:\n        return (ByteBuffer) value;\n      case DECIMAL:\n        return ByteBuffer.wrap(((BigDecimal) value).unscaledValue().toByteArray());\n      case VARIANT:\n        // Produce a concatenated buffer of metadata and value\n        Variant variant = (Variant) value;\n        VariantMetadata variantMetadata = variant.metadata();\n        VariantValue variantValue = variant.value();\n        ByteBuffer variantBuffer =\n            ByteBuffer.allocate(variantMetadata.sizeInBytes() + variantValue.sizeInBytes())\n                .order(ByteOrder.LITTLE_ENDIAN);\n        variantMetadata.writeTo(variantBuffer, 0);\n        variantValue.writeTo(variantBuffer, variantMetadata.sizeInBytes());","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/api/src/main/java/org/apache/iceberg/types/Conversions.java#L97-L133","documentation":"Conversions.toByteBuffer serializes a primitive value per the Iceberg single-value serialization spec. For STRING values it UTF-8 encodes the CharSequence; if the input contains unpaired surrogates or otherwise invalid UTF-16 the encoder throws CharacterCodingException, which is wrapped in a RuntimeIOException with this message.","triggerScenarios":"Passing a String/CharSequence containing unpaired surrogates (e.g. from decoding bytes with the wrong charset or corrupted data) to Conversions.toByteBuffer(stringType, value).","commonSituations":"Reading string data written by systems that produced malformed UTF-16; Java strings built from byte[] with an incorrect charset like ISO-8859-1 treated as UTF-16; records ingested from binary sources.","solutions":["Fix the data source so strings are valid UTF-16/UTF-8 (re-decode bytes with the correct charset)","Sanitize the value: encode to bytes replacing errors (new String(bytes, UTF_8) after a CharsetDecoder with REPLACE) before passing it in","Catch RuntimeIOException at the write boundary and quarantine the offending record"],"exampleFix":"// before\nByteBuffer buf = Conversions.toByteBuffer(Types.StringType.get(), rawString);\n// after\nbyte[] bytes = rawString.getBytes(StandardCharsets.UTF_8);\nString clean = new String(bytes, StandardCharsets.UTF_8); // replaces malformed sequences\nByteBuffer buf = Conversions.toByteBuffer(Types.StringType.get(), clean);","handlingStrategy":"try-catch","validationCode":"boolean valid = value.toString().chars().noneMatch(c -> Character.isSurrogate((char) c) && !Character.isHighSurrogate((char) c));","typeGuard":"boolean isEncodableUtf8(CharSequence s) { return StandardCharsets.UTF_8.newEncoder().canEncode(s); }","tryCatchPattern":"try { return Conversions.toByteBuffer(type, value); } catch (RuntimeIOException e) { log.error(\"Malformed string data\", e); throw e; }","preventionTips":["Sanitize strings from binary sources before writing","Always decode bytes with the correct charset","Validate sample data early in ingestion pipelines"],"tags":["java","encoding","utf-8","serialization"],"backgroundTag":"invalid-argument-format","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}