apache/beam · error · ValueError

HF_TOKEN environment variable not set. Please set the…

Error message

HF_TOKEN environment variable not set. Please set the environment variable or pass the token as an argument.

What it means

The HuggingFace embedding handler needs an access token to create authorized requests.Session for inference calls. At model-load time, if no token was supplied via the handler config and the HF_TOKEN environment variable is also absent, it raises ValueError because unauthenticated requests to HuggingFace inference endpoints will fail.

Solutions

Set the HF_TOKEN environment variable in the runtime environment (export HF_TOKEN=hf_xxx, or pass --env or worker environment variables in your runner).
Pass the token explicitly: HuggingFaceTextEmbeddings(columns=['text'], hf_token='hf_xxx').
For containers, bake the token into the custom container env or fetch it from a secret manager at pipeline setup.

Example fix

// before
handler = HuggingFaceTextEmbeddings(columns=['text'])
// after
handler = HuggingFaceTextEmbeddings(columns=['text'], hf_token=os.environ.get('HF_TOKEN'))

Defensive patterns

Strategy: validation

Validate before calling

import os
assert os.environ.get('HF_TOKEN') or hf_token, "HF_TOKEN env var or hf_token argument required"

Type guard

def has_hf_token():
    return bool(os.environ.get('HF_TOKEN'))

Try / catch

try:
    session = handler.load_model()
except ValueError as e:
    if 'HF_TOKEN' in str(e):
        os.environ['HF_TOKEN'] = fetch_token_from_secret_manager()
        session = handler.load_model()
    else:
        raise

Prevention

Set HF_TOKEN in all runtime environments, including remote runners
Pass hf_token explicitly in the handler constructor
Inject secrets via runner environment options or a secret manager

When it happens

Trigger: Calling load_model on HuggingFaceTextEmbeddings (remote inference API handler) when the handler was constructed without hf_token and the runtime environment (e.g. Beam worker container) does not define HF_TOKEN.

Common situations: Running a Beam pipeline on Dataflow/Flink where the local HF_TOKEN env var was set only on the dev machine, not passed via worker environment options; forgetting to pass hf_token to the handler constructor.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/86b528eae71b6741. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/transforms/embeddings/huggingface.py:179

        RunInference(
            model_handler=_TextEmbeddingHandler(self),
            inference_args=self.inference_args,
        ))


class _InferenceAPIHandler(ModelHandler):
  def __init__(self, config: 'InferenceAPIEmbeddings'):
    super().__init__()
    self._config = config

  def load_model(self):
    session = requests.Session()
    # if the token is not provided during construction time, it might have
    # been provided with custom container, which we can get it during runtume.
    if not self._config.hf_token:
      hf_token = os.environ.get("HF_TOKEN")
      if not hf_token:
        raise ValueError(
            'HF_TOKEN environment variable not set. '
            'Please set the environment variable or pass the token as an '
            'argument.')
      session.headers.update({"Authorization": f"Bearer {hf_token}"})
      return session

    session.headers.update(self._config.authorization_token)
    return session

  def run_inference(
      self, batch, session: requests.Session, inference_args=None):
    response = session.post(
        self._config.api_url,
        headers=self._config.authorization_token,
        json={
            "inputs": batch, "options": inference_args
        })
    return response.json()

View on GitHub (pinned to 12126d8942)