{"record":{"id":"d459150be344b94d","repo":"jax-ml/jax","slug":"higher-order-ad-not-supported-d45915","errorCode":null,"errorMessage":"Higher-order AD not supported","messagePattern":"Higher-order AD not supported","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"jax/experimental/pallas/ops/tpu/splash_attention/splash_attention_kernel.py","lineNumber":1283,"sourceCode":"    q: jax.Array,\n    k: jax.Array,\n    v: jax.Array,\n    segment_ids: SegmentIds | None,\n    sinks: jax.Array | None,\n    save_residuals: bool,\n    mask_value: float,\n    is_mqa: bool,\n    block_sizes: BlockSizes,\n    residual_checkpoint_name: str | None,\n    mask_function: MaskFunctionType | None,\n    attn_logits_soft_cap: float | None = None,\n    interpret: bool = False,\n) -> tuple[\n    jax.Array,\n    SplashResidualsType,\n]:\n  if save_residuals:\n    raise NotImplementedError(\"Higher-order AD not supported\")\n\n  out, (logsumexp,) = _splash_attention_forward(\n      fwd_mask_info,\n      q,\n      k,\n      v,\n      segment_ids,\n      sinks,\n      mask_value=mask_value,\n      is_mqa=is_mqa,\n      block_sizes=block_sizes,\n      residual_checkpoint_name=residual_checkpoint_name,\n      save_residuals=True,\n      mask_function=mask_function,\n      attn_logits_soft_cap=attn_logits_soft_cap,\n      interpret=interpret,\n  )\n  return out, (","sourceCodeStart":1265,"sourceCodeEnd":1301,"githubUrl":"https://github.com/jax-ml/jax/blob/1e1c6a8fc06dfcd1247076ec5cae4640cea5d7bb/jax/experimental/pallas/ops/tpu/splash_attention/splash_attention_kernel.py#L1265-L1301","documentation":"The forward pass wrapper for Splash Attention supports only first-order autodiff. If save_residuals=True is passed (which happens when differentiating a function that itself is being differentiated), higher-order AD is explicitly unsupported and raises NotImplementedError.","triggerScenarios":"Calling jax.grad twice on a loss that uses splash attention, or jax.jacfwd/hessian over a function containing splash attention; explicitly passing save_residuals=True to the internal fwd function.","commonSituations":"Meta-learning (MAML) second derivatives; computing Hessian-vector products of an attention-based loss; score-function estimators that differentiate through gradients.","solutions":["Restructure to first-order AD only, e.g. use jax.vjp/jax.jvp once, or stop_gradient on the inner gradient","Use a different attention implementation (e.g. reference jax.nn.dot_product_attention or flash attention on GPU) inside the twice-differentiated path","For Hessian-free methods, use implicit differentiation or finite differences of the gradient"],"exampleFix":"// before\nh = jax.hessian(loss_with_splash_attn)(params)\n// after\nattn_loss = remake_loss_with(attn=jax.nn.dot_product_attention)\nh = jax.hessian(attn_loss)(params)","handlingStrategy":"fallback","validationCode":"def safe_grad(fn, x):\n    try:\n        return jax.grad(fn)(x)\n    except NotImplementedError:\n        return jax.grad(lambda p: jax.grad(fn, allow_int=False)(p).sum())(x)  # not 2nd-order; see tip","typeGuard":null,"tryCatchPattern":"try:\n    h = jax.hessian(loss)(p)\nexcept NotImplementedError as e:\n    if 'Higher-order AD' in str(e):\n        h = jax.hessian(replace_splash_with_reference(loss))(p)\n    else:\n        raise","preventionTips":["Never apply grad twice through splash attention","Swap in a reference attention implementation inside twice-differentiated code paths","Use stop_gradient on inner gradients when approximate is acceptable"],"tags":["jax","pallas","tpu","splash-attention","autodiff"],"backgroundTag":"higher-order-autodiff-unsupported","analyzedSha":"1e1c6a8fc06dfcd1247076ec5cae4640cea5d7bb","analyzedAt":"2026-08-27T09:53:25.647Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}