Issue when Invoking User Data Function in Pipeline - Tenant not authorized for cluster

Hi Lukas_Ng ,

Fabric Pipelines cannot execute User Data Functions that require Spark or Lakehouse access using an Organizational account.This is a current Fabric platform limitation

The pipeline runs UDFs in a restricted, headless compute context that:

• Has no Spark cluster authorization
• Cannot access Lakehouse APIs
• Fails before your code executes

Why it works outside the pipeline

When you run it directly:

• Fabric uses your interactive Spark session

• Your user identity is authorized

• Lakehouse access works

Pipeline execution uses a different security and compute mode

What you should do instead (Fabric-recommended)

Use a Notebook activity in the Pipeline

This is the correct and supported approach.

Notebook activities:

• Run on an authorized Spark cluster

• Can write to OneLake / Lakehouse

• Accept parameters (file name, path, timestamp)

• Are designed for data engineering steps

If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!