Forum Discussion
Issue when Invoking User Data Function in Pipeline - Tenant not authorized for cluster
Hello Lukas_Ng,
We hope you're doing well. Could you please confirm whether your issue has been resolved or if you're still facing challenges? Your update will be valuable to the community and may assist others with similar concerns.
Thank you.
Hi Lukas_Ng ,
Fabric Pipelines cannot execute User Data Functions that require Spark or Lakehouse access using an Organizational account.This is a current Fabric platform limitation
The pipeline runs UDFs in a restricted, headless compute context that:
- Has no Spark cluster authorization
- Cannot access Lakehouse APIs
- Fails before your code executes
Why it works outside the pipeline
When you run it directly:
Fabric uses your interactive Spark session
Your user identity is authorized
Lakehouse access works
Pipeline execution uses a different security and compute mode
What you should do instead (Fabric-recommended)
Use a Notebook activity in the Pipeline
This is the correct and supported approach.
Notebook activities:
Run on an authorized Spark cluster
Can write to OneLake / Lakehouse
Accept parameters (file name, path, timestamp)
Are designed for data engineering steps
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
I believe this is 100% GPT generated and false.
Also, as other users have claimed, the UDF not being able to be called from the pipeline has started from 23 Jan. Some engineer messed up something.