Forum Discussion
Fabric pyspark notebook - Failed due to configuration Issue while using %%configure -f
Hi nareshraavi086,
This is interesting. This makes me thing that when run from the pipeline, the variable library is not working correctly.
Can you try in the same pipeline, disabling the notebook session, and adding an email session that emails you the variables from the variable library?
Then if that works, can you make a new notebook that just logs the output of the variable library to a file in a lakehouse somewhere?THis will help troubleshoot where exactly the breakdown is.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi nareshraavi086 ,
Thanks for the clarification.
From the behaviour you’re seeing, it looks like the Variable Library tokens such as $(/**/VL_xxx/Value) might not be getting expanded when the notebook is started from a pipeline. In pipeline-triggered runs, the Spark session is created before those values are resolved, so the %%configure block could be receiving the literal token text. Since the configure cell needs to be the first executable cell in pipeline runs, the session might not be able to restart and the run ends up failing even though the same JSON works in interactive mode.
You may try the same pattern mentioned earlier where the Variable Library values are resolved inside the pipeline and then passed into the notebook as base parameters. Using parameterName inside the configure block seems to give more consistent results in pipeline scenarios, since those parameters are available to the session right when it starts.
Similar threads and references you might find useful:
Solved: How to use variable-library values in a notebook - Microsoft Fabric Community
Solved: Re: Change Default Lakehouse - Microsoft Fabric Community
Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn
Hope this helps. Please reach out for further assistance.
Thank you.
Super UserHi nareshraavi086,
This is interesting. This makes me thing that when run from the pipeline, the variable library is not working correctly.
Can you try in the same pipeline, disabling the notebook session, and adding an email session that emails you the variables from the variable library?
Then if that works, can you make a new notebook that just logs the output of the variable library to a file in a lakehouse somewhere?
THis will help troubleshoot where exactly the breakdown is.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Thanks for the suggestions.
In my Pipeline, I have created Library variables and which referenced to my Variable Library parameters.
I have used these Library variables in Email activity. Here its working fine and variables are resolved.
And then, I tried with my notebook activity.
I have created base parameters too in my notebook activity which are pointed to the Library variables.
And then I ran my pipeline but didnt give positive results.
Pipeline got failed with same error.
v-veshwara-msftCommunity Support
Hi nareshraavi086 ,
Thanks for the update and thanks to tayloramy for the suggestions.
Since the variables resolve correctly in the email activity but the notebook still fails to start, it indicates the values are not reaching the notebook session during the pipeline run.
This also fits with the note in the documentation that the configure cell must be the first code cell in a pipeline run, otherwise the session cannot be created.
A reliable way to pass values into the configure block during pipeline execution is to send them as base parameters from the pipeline and reference those parameters inside the configure JSON. This avoids issues with variable libraries not being picked up at session start.
The following configuration pattern is an example that works reliably when run through a pipeline:
%%configure -f { "defaultLakehouse": { "name": { "parameterName": "lh_silver_name", "defaultValue": "LH_SILVER" }, "id": { "parameterName": "lh_silver_id", "defaultValue": "" }, "workspaceId": { "parameterName": "lh_silver_workspace_id", "defaultValue": "" } } }Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn
You only need to define these base parameters in the notebook activity. They are available to the session even without a parameters cell, so the configure block can resolve them when the notebook starts.
If you can try this pattern in your test notebook within the same pipeline, it should confirm whether the values are being passed through correctly.
Hope this helps. Please reach out for further assistance.
Thank you.
- v-veshwara-msft
Hi nareshraavi086 ,
Just wanted to check if the response provided was helpful. If further assistance is needed, please reach out.
Thank you. Hi,
Thanks for the suggessions.
Whatever the configureation mentioned in %%Configure -f works fine.
However, it is using direct variables.
All my variables/parameters are defined in Variable Library.
If I use Variable Library, Then it is not able to expan variables and not able to Initiate the notebook run.%%configure -f{"defaultLakehouse": {"name": {"parameterName": "$(/**/VL_Woxxxeters/DefaultLHName)","defaultValue": "xxxx"},"workspaceId": {"parameterName": "$(/**/VL_Woxxxers/DefaultWorkspaceID)","defaultValue": "xxxxx"}}}- v-veshwara-msft
Hi nareshraavi086 ,
Thanks for the clarification.
From the behaviour you’re seeing, it looks like the Variable Library tokens such as $(/**/VL_xxx/Value) might not be getting expanded when the notebook is started from a pipeline. In pipeline-triggered runs, the Spark session is created before those values are resolved, so the %%configure block could be receiving the literal token text. Since the configure cell needs to be the first executable cell in pipeline runs, the session might not be able to restart and the run ends up failing even though the same JSON works in interactive mode.
You may try the same pattern mentioned earlier where the Variable Library values are resolved inside the pipeline and then passed into the notebook as base parameters. Using parameterName inside the configure block seems to give more consistent results in pipeline scenarios, since those parameters are available to the session right when it starts.
Similar threads and references you might find useful:
Solved: How to use variable-library values in a notebook - Microsoft Fabric Community
Solved: Re: Change Default Lakehouse - Microsoft Fabric Community
Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn
Hope this helps. Please reach out for further assistance.
Thank you.