Fabric pyspark notebook - Failed due to configuration Issue while using %%configure -f

Hi raana_86, 

This is interesting. This makes me thing that when run from the pipeline, the variable library is not working correctly. 

Can you try in the same pipeline, disabling the notebook session, and adding an email session that emails you the variables from the variable library? 

Then if that works, can you make a new notebook that just logs the output of the variable library to a file in a lakehouse somewhere? 

THis will help troubleshoot where exactly the breakdown is. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

Hi raana_86 ,

Thanks for the clarification.

From the behaviour you’re seeing, it looks like the Variable Library tokens such as $(/**/VL_xxx/Value) might not be getting expanded when the notebook is started from a pipeline. In pipeline-triggered runs, the Spark session is created before those values are resolved, so the %%configure block could be receiving the literal token text. Since the configure cell needs to be the first executable cell in pipeline runs, the session might not be able to restart and the run ends up failing even though the same JSON works in interactive mode.

You may try the same pattern mentioned earlier where the Variable Library values are resolved inside the pipeline and then passed into the notebook as base parameters. Using parameterName inside the configure block seems to give more consistent results in pipeline scenarios, since those parameters are available to the session right when it starts.

Similar threads and references you might find useful:

Solved: How to use variable-library values in a notebook - Microsoft Fabric Community

Solved: Re: Change Default Lakehouse - Microsoft Fabric Community

Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.

Thank you.