{"record":{"id":"d2346582f2833ab7","repo":"apache/beam","slug":"expected-the-engine-to-not-be-null","errorCode":null,"errorMessage":"Expected the engine to not be null","messagePattern":"Expected the engine to not be null","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaSourceDoFn.java","lineNumber":139,"sourceCode":"  public void setUp() {\n    engine = DefaultEngine.create(getConfiguration());\n  }\n\n  @ProcessElement\n  public ProcessContinuation processElement(\n      @Element DeltaReadTask task,\n      RestrictionTracker<OffsetRange, Long> tracker,\n      OutputReceiver<Row> out)\n      throws Exception {\n\n    SerializableRow scanStateRow = task.getScanStateRow();\n    StructType physicalSchema = ScanStateRow.getPhysicalDataReadSchema(scanStateRow);\n    StructType logicalSchema = ScanStateRow.getLogicalSchema(scanStateRow);\n    Schema beamSchema = DeltaIO.ReadRows.convertToBeamSchema(logicalSchema);\n\n    Engine currentEngine = engine;\n    if (currentEngine == null) {\n      throw new IllegalArgumentException(\"Expected the engine to not be null\");\n    }\n\n    // `BeamParquetHandler` takes a reference to the `RestrictionTracker` so that it\n    // can perform `getFrom`, `getTo`, `tryClaim` requests to return the correct set\n    // of row groups that map to the current restriction.\n    BeamParquetHandler parquetHandler =\n        new BeamParquetHandler(getConfiguration(), currentEngine.getParquetHandler(), tracker);\n    BeamEngine beamEngine = new BeamEngine(currentEngine, parquetHandler);\n\n    long currentStartRgIndex = 0L;\n\n    // We have to go through files in the `DeltaReadTask` in order so that the\n    // `RestrictionTracker`\n    // can correctly handle the range of the current split.\n    List<SerializableRow> scanFileRows = task.getScanFileRows();\n    List<List<Long>> rowGroupSizesPerFile = task.getRowGroupSizesPerFile();\n    for (int i = 0; i < scanFileRows.size(); i++) {\n      if (currentStartRgIndex >= tracker.currentRestriction().getTo()) {","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaSourceDoFn.java#L121-L157","documentation":"processElement() requires a Delta Kernel Engine instance to construct the scan state, parquet handler, and read data. The engine field should have been initialized in setup() from the serialized Hadoop configuration; if it is still null when elements arrive, the DoFn is in an invalid lifecycle state and throws IllegalArgumentException.","triggerScenarios":"The DoFn's setup() failed to initialize the engine (e.g. missing Hadoop configuration, classpath issue loading engine classes), or the DoFn is invoked in a context where setup was skipped.","commonSituations":"Missing/incorrect delta-kernel or Hadoop dependencies on the worker classpath so engine construction silently fails or is skipped; custom pipelines reusing DeltaSourceDoFn without running setup; serialization/deserialization problems losing the engine field.","solutions":["Ensure delta-kernel-defaults (and its Engine implementation) plus Hadoop dependencies are on the worker classpath","Check worker logs for an earlier failure/exception in setup() that left engine uninitialized","Verify the DoFn is only used through DeltaIO.ReadRows so setup()/startBundle lifecycle hooks run correctly","Upgrade/align Beam Delta IO and io.delta kernel versions to a compatible set"],"exampleFix":"// before\n// engine null at process time because setup never ran\nDoFnProcessContext ctx = ...; doFn.processElement(ctx);\n// after\ndoFn.setup(); // initializes engine before processing\nDoFnProcessContext ctx = ...; doFn.processElement(ctx);","handlingStrategy":"validation","validationCode":"// Ensure dependencies ship the kernel Engine impl on the worker classpath\n// gradle: implementation 'io.delta:delta-kernel-defaults:<version>'\n// and let DeltaIO construct DeltaSourceDoFn itself rather than reusing it manually","typeGuard":null,"tryCatchPattern":"try {\n  rows = input.apply(deltaIO);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"Expected the engine to not be null\")) {\n    // check worker logs for setup() failure; fix classpath/dependencies and rerun\n  } else { throw e; }\n}","preventionTips":["Never instantiate DeltaSourceDoFn directly; use DeltaIO.read() so setup() lifecycle runs","Verify shaded/fat jars include io.delta kernel classes and their engine defaults","Pin compatible versions of beam-sdks-java-io-delta, delta-kernel-api, and delta-kernel-defaults"],"tags":["java","apache-beam","delta-lake","lifecycle","dependency"],"backgroundTag":"null-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}