{"record":{"id":"ef5e65c863eb494d","repo":"apache/beam","slug":"please-specify-either-dask-npartitions-or-dask-parition-size","errorCode":null,"errorMessage":"Please specify either `dask_npartitions` or `dask_parition_size` but not both: npartitions=%r, partition_size=%r.","messagePattern":"Please specify either `dask_npartitions` or `dask_parition_size` but not both: npartitions=%r, partition_size=%r\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/runners/dask/transform_evaluator.py","lineNumber":165,"sourceCode":"\n\nclass NoOp(DaskBagOp):\n  \"\"\"An identity on a dask bag: returns the input as-is.\"\"\"\n  def apply(self, input_bag: OpInput, side_inputs: OpSide = None) -> db.Bag:\n    return input_bag\n\n\nclass Create(DaskBagOp):\n  \"\"\"The beginning of a Beam pipeline; the input must be `None`.\"\"\"\n  def apply(self, input_bag: OpInput, side_inputs: OpSide = None) -> db.Bag:\n    assert input_bag is None, 'Create expects no input!'\n    original_transform = t.cast(_Create, self.transform)\n    items = original_transform.values\n\n    npartitions = self.bag_kwargs.get('npartitions')\n    partition_size = self.bag_kwargs.get('partition_size')\n    if npartitions and partition_size:\n      raise ValueError(\n          f'Please specify either `dask_npartitions` or '\n          f'`dask_parition_size` but not both: '\n          f'{npartitions=}, {partition_size=}.')\n    if not npartitions and not partition_size:\n      # partition_size is inversely related to `npartitions`.\n      # Ideal \"chunk sizes\" in dask are around 10-100 MBs.\n      # Let's hope ~128 items per partition is around this\n      # memory overhead.\n      default_size = 128\n      partition_size = max(default_size, math.ceil(math.sqrt(len(items)) / 10))\n      if partition_size == default_size:\n        _LOGGER.warning(\n            'The new default partition size is %d, it used to be 1 '\n            'in previous DaskRunner versions.' % default_size)\n\n    return db.from_sequence(\n        items, npartitions=npartitions, partition_size=partition_size)\n","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/runners/dask/transform_evaluator.py#L147-L183","documentation":"When evaluating a beam.Create transform, the Dask evaluator requires exactly one of `dask_npartitions` or `dask_partition_size` to control bag chunking. Passing both is ambiguous, so it raises ValueError.","triggerScenarios":"Setting both `dask_npartitions` and `dask_partition_size` options (e.g. via PipelineOptions --dask_npartitions and --dask_partition_size) for a pipeline containing beam.Create.","commonSituations":"Config mistakes where users copy an example and add both tuning knobs, or leftover flags from earlier experiments.","solutions":["Remove one of the two options from your PipelineOptions/flags.","Keep only `dask_npartitions` if you know the desired partition count.","Keep only `dask_partition_size` to let the evaluator compute partitions from total size."],"exampleFix":"// before\noptions = PipelineOptions(['--dask_npartitions=8', '--dask_partition_size=50MB'])\n// after\noptions = PipelineOptions(['--dask_npartitions=8'])","handlingStrategy":"validation","validationCode":"opts = dask_options.get_all_options(drop_default=True, current_only=True)\nif opts.get('dask_npartitions') and opts.get('dask_partition_size'):\n    raise ValueError('Set only one of dask_npartitions/dask_partition_size')","typeGuard":null,"tryCatchPattern":"try:\n    pipeline.run()\nexcept ValueError as e:\n    if 'but not both' in str(e):\n        logging.error('Remove one of dask_npartitions/dask_partition_size')\n    raise","preventionTips":["Set exactly one partitioning knob in your pipeline options.","Centralize Dask option construction in one config module.","Grep deployment flags for both keys before launching."],"tags":["python","apache-beam","dask","config-conflict"],"backgroundTag":"mutually-exclusive-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}