apache/beam · error · IllegalStateException

Could not create destination for extract job

Error message

Could not create destination for extract job %s

What it means

Thrown by FakeJobService's writeRowsHelper, a Beam testing fake that emulates BigQuery. When writing extracted records to a temp file, an IOException while creating/appending the destination file is wrapped in this IllegalStateException. It indicates the fake extract job could not materialize its output file.

Solutions

1. Check that the parent directory of `filename` exists and is writable before running the test
2. Verify the test environment's temp directory (java.io.tmpdir) is writable and has free space
3. Look at the cause IOException in the stack trace for the underlying filesystem error
4. Ensure no concurrent test cleanup deletes the temp file while the extract job writes it

Example fix

// before
File out = new File(unverifiedPath);
// after
File out = new File(path);
Files.createDirectories(out.getParentFile().toPath());
if (!out.canWrite()) { throw new SkipException("temp dir not writable"); }

File dest = new File(filename);
if (dest.getParentFile() != null && !dest.getParentFile().exists()) {
  Files.createDirectories(dest.getParentFile().toPath());
}
if (!Files.isWritable(dest.getParentFile() != null ? dest.getParentFile().toPath() : Paths.get("."))) {
  throw new IllegalStateException("destination dir not writable: " + filename);
}

try {
  writeRows(rows, filename);
} catch (IllegalStateException e) {
  LOGGER.severe("extract destination failed: " + filename + " cause=" + e.getCause());
  throw e;
}

When it happens

Trigger: Calling writeRows/writeRowsHelper in FakeJobService when the target file (named by `filename`) cannot be created or opened for writing — e.g. bad path, missing parent directory, or disk full.

Common situations: Running Beam BigQuery I/O integration tests on machines with restricted temp directories, read-only /tmp, or tests that clean up temp folders concurrently while the fake job service is still writing.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Failed closing channel to
• Failed to convert the row to JSON
• Failed to fetch BigQuery data.
• Primary key validation error! Multiple inserts with the…
• Upserts only allowed when using primary keys

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a295942cccf09414. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java:559

  private void writeRowsHelper(
      List<TableRow> rows, Schema avroSchema, String destinationPattern, int shard) {
    String filename = destinationPattern.replace("*", String.format("%012d", shard));
    try (WritableByteChannel channel =
            FileSystems.create(
                FileSystems.matchNewResource(filename, false /* isDirectory */), MimeTypes.BINARY);
        DataFileWriter<GenericRecord> tableRowWriter =
            new DataFileWriter<>(new GenericDatumWriter<GenericRecord>(avroSchema))
                .create(avroSchema, Channels.newOutputStream(channel))) {
      for (Map<String, Object> record : rows) {
        GenericRecordBuilder genericRecordBuilder = new GenericRecordBuilder(avroSchema);
        for (Map.Entry<String, Object> field : record.entrySet()) {
          genericRecordBuilder.set(field.getKey(), field.getValue());
        }
        tableRowWriter.append(genericRecordBuilder.build());
      }
    } catch (IOException e) {
      throw new IllegalStateException(
          String.format("Could not create destination for extract job %s", filename), e);
    }
  }
}

View on GitHub (pinned to 12126d8942)