{"record":{"id":"1496cea2ba9fbad1","repo":"apache/iceberg","slug":"couldn-t-set-arrow-properties-which-may-impact-re-1496ce","errorCode":null,"errorMessage":"Couldn't set Arrow properties, which may impact read performance","messagePattern":"Couldn't set Arrow properties, which may impact read performance","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java","lineNumber":49,"sourceCode":"import org.apache.iceberg.spark.SparkUtil;\nimport org.apache.parquet.schema.MessageType;\nimport org.slf4j.Logger;\nimport org.slf4j.LoggerFactory;\n\npublic class VectorizedSparkParquetReaders {\n\n  private static final Logger LOG = LoggerFactory.getLogger(VectorizedSparkParquetReaders.class);\n  private static final String ENABLE_UNSAFE_MEMORY_ACCESS = \"arrow.enable_unsafe_memory_access\";\n  private static final String ENABLE_UNSAFE_MEMORY_ACCESS_ENV = \"ARROW_ENABLE_UNSAFE_MEMORY_ACCESS\";\n  private static final String ENABLE_NULL_CHECK_FOR_GET = \"arrow.enable_null_check_for_get\";\n  private static final String ENABLE_NULL_CHECK_FOR_GET_ENV = \"ARROW_ENABLE_NULL_CHECK_FOR_GET\";\n\n  static {\n    try {\n      enableUnsafeMemoryAccess();\n      disableNullCheckForGet();\n    } catch (Exception e) {\n      LOG.warn(\"Couldn't set Arrow properties, which may impact read performance\", e);\n    }\n  }\n\n  private VectorizedSparkParquetReaders() {}\n\n  public static ColumnarBatchReader buildReader(\n      Schema expectedSchema,\n      MessageType fileSchema,\n      Map<Integer, ?> idToConstant,\n      BufferAllocator bufferAllocator) {\n    return (ColumnarBatchReader)\n        TypeWithSchemaVisitor.visit(\n            expectedSchema.asStruct(),\n            fileSchema,\n            new ReaderBuilder(\n                expectedSchema,\n                fileSchema,\n                NullCheckingForGet.NULL_CHECKING_ENABLED,","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/VectorizedSparkParquetReaders.java#L31-L67","documentation":"VectorizedSparkParquetReaders' static initializer sets JVM-level Arrow memory and null-check properties needed for fast vectorized reads. If enabling them throws (e.g. unsupported Arrow/JDK version), the class logs this warning and falls back to safe defaults; reads still work but slower.","triggerScenarios":"Class-loading VectorizedSparkParquetReaders when the static enableUnsafeMemoryAccess()/disableNullCheckForGet() reflection calls fail — typically Arrow version incompatibility or JDK restrictions (e.g. no access to unsafe APIs).","commonSituations":"Mixed Arrow versions on the classpath from other Spark libraries; running on JDKs where sun.misc.Unsafe access is restricted; shaded/relocated Arrow conflicts.","solutions":["Align the Arrow version with the one required by the Iceberg Spark runtime (check icebergs spark build for the Arrow version)","Avoid bundling conflicting Arrow jars in the user classpath; let Spark's Iceberg runtime provide it","Verify on a supported JDK; performance warning is non-fatal, so proceed if latency is acceptable"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// detect Arrow conflicts before enabling vectorized reads\ntry (var is = getClass().getResourceAsStream(\"/META-INF/maven/org.apache.arrow/arrow-memory-core/pom.properties\")) {\n  Properties p = new Properties(); p.load(is);\n  if (!p.getProperty(\"version\").equals(expectedArrowVersion)) LOG.warn(\"Arrow version mismatch\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  runVectorizedQuery(spark);\n} catch (Exception e) {\n  LOG.warn(\"Vectorized read degraded; falling back to non-vectorized or row-based reader\", e);\n}","preventionTips":["Keep a single Arrow version on the classpath matching the Iceberg Spark runtime","Avoid shading/relocating Arrow independently","Test on the target JDK version for unsafe-memory access","Accept the warning as a performance-only degradation"],"tags":["spark","parquet","vectorized-reads","arrow","performance"],"backgroundTag":"module-init-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}