{"record":{"id":"3a6fd3081e3499ed","repo":"apache/iceberg","slug":"couldn-t-set-arrow-properties-which-may-impact-re-3a6fd3","errorCode":null,"errorMessage":"Couldn't set Arrow properties, which may impact read performance","messagePattern":"Couldn't set Arrow properties, which may impact read performance","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java","lineNumber":49,"sourceCode":"import org.apache.iceberg.spark.SparkUtil;\nimport org.apache.parquet.schema.MessageType;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class VectorizedSparkParquetReaders {\n\n  private static final Logger LOG = LoggerFactory.getLogger(VectorizedSparkParquetReaders.class);\n  private static final String ENABLE_UNSAFE_MEMORY_ACCESS = \"arrow.enable_unsafe_memory_access\";\n  private static final String ENABLE_UNSAFE_MEMORY_ACCESS_ENV = \"ARROW_ENABLE_UNSAFE_MEMORY_ACCESS\";\n  private static final String ENABLE_NULL_CHECK_FOR_GET = \"arrow.enable_null_check_for_get\";\n  private static final String ENABLE_NULL_CHECK_FOR_GET_ENV = \"ARROW_ENABLE_NULL_CHECK_FOR_GET\";\n\n  static {\n    try {\n      enableUnsafeMemoryAccess();\n      disableNullCheckForGet();\n    } catch (Exception e) {\n      LOG.warn(\"Couldn't set Arrow properties, which may impact read performance\", e);\n    }\n  }\n\n  private VectorizedSparkParquetReaders() {}\n\n  public static ColumnarBatchReader buildReader(\n      Schema expectedSchema,\n      MessageType fileSchema,\n      Map<Integer, ?> idToConstant,\n      BufferAllocator bufferAllocator) {\n    return (ColumnarBatchReader)\n        TypeWithSchemaVisitor.visit(\n            expectedSchema.asStruct(),\n            fileSchema,\n            new ReaderBuilder(\n                expectedSchema,\n                fileSchema,\n                NullCheckingForGet.NULL_CHECKING_ENABLED,","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java#L31-L67","documentation":"A LOG.warn in VectorizedSparkParquetReaders' static initializer: setting Arrow memory-access properties (unsafe memory access, disabling null checks on get) failed via reflection. The vectorized reader still works but with safety checks enabled, degrading read performance. Not fatal — execution continues.","triggerScenarios":"Class loading of VectorizedSparkParquetReaders when the reflection calls to Arrow/Spark unsafe memory APIs fail — typically due to a Spark/Arrow version mismatch, a JVM without the expected sun.misc.Unsafe access, or a JVM that blocks unsafe operations (e.g. newer JDKs with --illegal-access=deny, or --enable-native-access restrictions).","commonSituations":"Upgrading Spark or JDK without matching the Iceberg Spark runtime version; running on JDK 16+ where illegal reflective access is denied; custom Arrow builds lacking expected methods.","solutions":["Align the iceberg-spark-runtime version with your exact Spark version (e.g. use iceberg-spark-runtime-4.2 for Spark 4.2).","Add JVM flags to permit the needed access if on newer JDKs (e.g. --add-opens java.base/jdk.internal.misc=ALL-UNNAMED, --add-opens java.base/sun.nio.ch=ALL-UNNAMED), mirroring Spark's own flags.","Treat the warning as a performance-only issue: reads still work; investigate only if throughput matters.","Check the chained exception in the log to identify which specific reflective call failed."],"exampleFix":"// before\nspark-submit --conf spark.driver.extraJavaOptions=-Xmx4g ...\n// after\nspark-submit --conf \"spark.driver.extraJavaOptions=-Xmx4g --add-opens java.base/jdk.internal.misc=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED\"","handlingStrategy":"validation","validationCode":"// Ensure Spark/Iceberg/JDK combination is supported before launch\nString sparkVersion = spark.version();\nif (!supportedIcebergRuntimeFor(sparkVersion)) {\n  throw new IllegalStateException(\"Use iceberg-spark-runtime matching Spark \" + sparkVersion);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin iceberg-spark-runtime to the exact Spark major.minor version","Add the same --add-opens JVM flags Spark uses, especially on JDK 17+","Watch driver/executor logs at startup for this warning and fix configuration early","Test vectorized reads after JDK or Spark upgrades"],"tags":["spark","arrow","vectorized-reads","reflection","performance"],"backgroundTag":"reflection-init-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}