{"record":{"id":"5d2953d68bb115ff","repo":"apache/beam","slug":"sorter-doesn-t-support-gcs-temporary-location","errorCode":null,"errorMessage":"Sorter doesn't support GCS temporary location.","messagePattern":"Sorter doesn't support GCS temporary location\\.","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/ExternalSorter.java","lineNumber":43,"sourceCode":"public abstract class ExternalSorter implements Sorter {\n  protected final Options options;\n\n  /** {@link Options} contains configuration of the sorter. */\n  public static class Options implements Serializable {\n    private String tempLocation = \"/tmp\";\n    private int memoryMB = 100;\n    private SorterType sorterType = SorterType.HADOOP;\n\n    /** Sorter type. */\n    public enum SorterType {\n      HADOOP,\n      NATIVE\n    }\n\n    /** Sets the path to a temporary location where the sorter writes intermediate files. */\n    public Options setTempLocation(String tempLocation) {\n      if (tempLocation.startsWith(\"gs://\")) {\n        throw new IllegalArgumentException(\"Sorter doesn't support GCS temporary location.\");\n      }\n\n      this.tempLocation = tempLocation;\n      return this;\n    }\n\n    /** Returns the configured temporary location. */\n    public String getTempLocation() {\n      return tempLocation;\n    }\n\n    /**\n     * Sets the size of the memory buffer in megabytes. Must be greater than zero and less than\n     * 2048.\n     */\n    public Options setMemoryMB(int memoryMB) {\n      this.memoryMB = memoryMB;\n      checkMemoryMB();","sourceCodeStart":25,"sourceCodeEnd":61,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/ExternalSorter.java#L25-L61","documentation":"Beam's ExternalSorter (the Hadoop-based native sorter in the sorter extension) spills intermediate sorted data to a local temporary directory and cannot read/write paths on Google Cloud Storage. Options.setTempLocation rejects any path starting with gs:// with an IllegalArgumentException, because the sorter requires random-access local filesystem access that GCS does not provide.","triggerScenarios":"Calling ExternalSorter.create(...) or configuring its Options with setTempLocation(\"gs://bucket/path\") — e.g., when --tempLocation is inherited directly from a Dataflow pipeline option that is a GCS path.","commonSituations":"Running on Dataflow where the pipeline's --tempLocation is gs:// and is passed straight through to the sorter options; building sorter options programmatically from pipeline temp location defaults.","solutions":["Set a local (non-GCS) temp location: options.setTempLocation(\"/tmp/sorter\") or another local path on the worker","If on Dataflow, derive a worker-local directory (e.g., use the worker's /tmp) instead of forwarding the pipeline's gs:// tempLocation","Switch to a sorter implementation that supports GCS if remote temp storage is required, or stage via a different mechanism (e.g., write sorted output yourself)","Guard the configuration: strip or validate the tempLocation scheme before calling setTempLocation"],"exampleFix":"// before\nnew Options().setTempLocation(pipelineOptions.getTempLocation()); // \"gs://bucket/tmp\"\n// after\nnew Options().setTempLocation(Files.createTempDirectory(\"sorter\").toString());","handlingStrategy":"validation","validationCode":"if (tempLocation != null && tempLocation.startsWith(\"gs://\")) { throw new IllegalArgumentException(\"Provide a local temp dir for Sorter\"); }\noptions.setTempLocation(tempLocation);","typeGuard":"static boolean isLocalPath(String loc) { return loc != null && !loc.startsWith(\"gs://\") && !loc.startsWith(\"s3://\") && !loc.startsWith(\"abfs://\"); }","tryCatchPattern":"try { options.setTempLocation(tempLocation); } catch (IllegalArgumentException e) { options.setTempLocation(Files.createTempDirectory(\"sorter\").toString()); }","preventionTips":["Never forward pipeline --tempLocation (GCS) directly to sorter options","Use worker-local directories (/tmp) for sorter temp files","Add a startup config check that validates the temp location scheme","Document that ExternalSorter requires local disk"],"tags":["java","apache-beam","sorter","gcs","configuration"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}