Notebook ownership and lakehouse authentication

[Anonymous]

I have a pipeline in which I use, among other stuff, a notebook activity. The called notebook in turn executes many other notebooks via mssparutils.notebook.run(). These notebooks write to a lakehouse.

Now, when the pipeline runs scheduled at night, one of the inner notebook's lakehouse write fails with "Bad Request", 400, and "Authentication Failed with Bearer token is not present in the request". This is the latest notebook which was created by account A, as opposed to the other notebooks which were created by account B and some last modified by account A. Both of these accounts are workspace admins.

But here comes the kicker: I can't reproduce this issue. If I run the failing inner notebook manually, everything works. If I run the outer notebook, same. If I deactivate the other pipeline items and schedule the pipeline, everything works.

The error message suggests some issues with ownership, I guess. So apart from: What is going on and how can I fix this, my question would be: Is notebook ownership a thing? Can I transfer it? I can't just recreate everything, because a) that would be a lot of work and b) all items are linked within a deployment pipeline whose item bindings would be lost.

[FabianSchut]

Hi, which user scheduled the pipeline? I think the problem is there. AFAIK, the user who scheduled the pipeline runs the notebook and needs access to all items used in the notebook.

[Anonymous]

The scheduling is done by the non-creating account A. If this was the issue, I would have to wonder why of the inner executed notebooks only one consistently fails with this issue, if they were all executed by effectively the same user. Also, all relevant accounts are workspace admins and have complete access to the problematic lakehouse.

[FabianSchut]

You also mention that the items are connected to a deployment pipeline. In which stage are you running this? Is it possible that you did not set parameters for the default Lakehouse in your last notebook. This would mean that the last notebook is trying to write to another workspace then it is in. I can imagine that this potentially break something when it is scheduled. 

[Anonymous]

Good call, it's the last of three stages. However, all default lakehouses are properly set.

[Anonymous]

Hi @Anonymous ,

Have you solved your problem? If so, can you share your solution here and mark the correct answer as a standard answer to help other members find it faster? Thank you very much for your kind cooperation!

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[FabianSchut]

Okay! Did you check in the lakehouses within the workspace or did you check by looking at the deployment rules? For deployment rules to be deployed, you should deploy the artifacts again after you've set the set the rules.

[Anonymous]

I checked within the notebooks themselves and am aware of the need to deploy after setting rules.

[Anonymous]

Yooo, I checked and it says that here: Create deployment rules for Fabric's ALM - Microsoft Fabric | Microsoft Learn

However, I just deployed some new items and configured deployment rules and they were implemented instantly. Can anyone confirm this?

[Anonymous]

Hi @Anonymous ,

It sounds like you had a smooth deployment experience. According to the Microsoft Learn documentation, deployment rules should take effect the next time you deploy to that stage. However, if you create rules and then compare the stages before deploying, the comparison is based on the newly created rules, even if they haven't taken effect yet.

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.