apache/beam · error · IOException

Duplicate job id

Error message

Duplicate job id ${jobId}

What it means

FakeJobService.verifyUniqueJobId enforces, like real BigQuery, that job ids are unique within the fake's job table. When startLoadJob, startExtractJob, or startCopyJob is called with a jobId already present in allJobs, it throws IOException("Duplicate job id " + jobId). Tests that replay jobs or generate non-unique ids trip this.

Solutions

1. Generate a unique job id per submission (e.g., append a counter, UUID, or timestamp).
2. If idempotent resubmission is intended, catch the IOException or check job existence via getJob before starting.
3. Create a fresh FakeJobService (or clear allJobs) for each test to avoid cross-test id collisions.
4. Namespace job ids with the test name to guarantee uniqueness.

Example fix

// before
service.startLoadJob(jobId, config);
service.startLoadJob(jobId, config2); // duplicate
// after
service.startLoadJob(jobId + "-1", config);
service.startLoadJob(jobId + "-2", config2);

java.util.Set<String> usedJobIds = new java.util.HashSet<>();
String jobId = baseId + "-" + java.util.UUID.randomUUID();
if (!usedJobIds.add(jobId)) {
  throw new IllegalStateException("Duplicate job id generated: " + jobId);
}

try {
  service.startLoadJob(jobRef, config);
} catch (IOException e) {
  if (e.getMessage().startsWith("Duplicate job id")) {
    // treat as already-submitted or regenerate id and retry once
    jobRef = jobRef.toBuilder().setJobId(jobRef.getJobId() + "-retry").build();
    service.startLoadJob(jobRef, config);
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling startLoadJob/startExtractJob/startCopyJob with a job id already used in the same fake (job reference id reused across two starts), or re-running a test against a shared fake whose allJobs table still holds earlier jobs.

Common situations: Test retries that resubmit the same job reference without changing the id; deterministic id generation (e.g., constant strings) in test pipelines; a shared FakeJobService across test methods without cleanup.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

• ${format}
• No such stream
• A function must be provided to convert the input type into…
• BigQuery %1$s not found for table "%2$s" . Please create…
• BigQuery data contained value

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/82928f40f3b6e315. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java:336

                      .setMessage(
                          String.format(
                              "Job %s failed: %s", job.job.getConfiguration(), e.toString())));
          List<ResourceId> sourceFiles =
              filesForLoadJobs.get(jobRef.getProjectId(), jobRef.getJobId());
          if (sourceFiles != null) {
            FileSystems.delete(sourceFiles);
          }
        }
        return JSON_FACTORY.fromString(JSON_FACTORY.toString(job.job), Job.class);
      }
    } catch (IOException e) {
      return null;
    }
  }

  private void verifyUniqueJobId(String jobId) throws IOException {
    if (allJobs.containsColumn(jobId)) {
      throw new IOException("Duplicate job id " + jobId);
    }
  }

  private JobStatus runJob(Job job) throws InterruptedException, IOException {
    if (job.getConfiguration().getLoad() != null) {
      return runLoadJob(job.getJobReference(), job.getConfiguration().getLoad());
    } else if (job.getConfiguration().getCopy() != null) {
      return runCopyJob(job.getConfiguration().getCopy());
    } else if (job.getConfiguration().getExtract() != null) {
      return runExtractJob(job, job.getConfiguration().getExtract());
    } else if (job.getConfiguration().getQuery() != null) {
      return runQueryJob(job.getConfiguration().getQuery());
    }
    return new JobStatus().setState("DONE");
  }

  private boolean validateDispositions(
      Table table, CreateDisposition createDisposition, WriteDisposition writeDisposition)

View on GitHub (pinned to 12126d8942)