Fabric pyspark notebook - Failed due to configuration Issue while using %%configure -f

[nareshraavi086]

Hi,

I have a Fabric Pyspark notebook with First cell using %%configure -f to set default lakehouse.

%%configure -f

{

  "defaultLakehouse": {

    "name": {

      "variableName": "$(/**/VL_Wxxxx_Paxxx/DefaultLHName)"

    },

    "id": {

      "variableName": "$(/**/VL_Wxxxx_Paxxx/DefaultLHID)"

    }

  }

}

I have a variable Library where these Default variables are defined. 
If I run the notebook directly then it is working fine. where as if i run the notebook through data pipelines with notebook activity then I am getting below error.
""
Notebook execution failed at Notebook service with http status code - 'NotFound', please check the Run logs on Notebook, additional details - '{
"code": 404,
"message": "No notebook execution state found in database for the runId - exxxxxxxxxxxxxxxxxxxxx5",
"result": {
"errorMessage": null,
"details": null
}
}' :

""

Could you please help me out on this why this error is popping up.?

[Youraj8271]

Thank you @tayloramy for the helpful guidance.
I truly appreciate the clarity you provided.
As someone starting out in Fabric Data Engineering, insights like yours make the learning path much easier.
Thanks again for taking the time to reply.

[tayloramy]

Hi @nareshraavi086, 

This is interesting. This makes me thing that when run from the pipeline, the variable library is not working correctly. 

Can you try in the same pipeline, disabling the notebook session, and adding an email session that emails you the variables from the variable library? 

Then if that works, can you make a new notebook that just logs the output of the variable library to a file in a lakehouse somewhere? 

THis will help troubleshoot where exactly the breakdown is. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

[nareshraavi086]

Thanks for the suggestions.
In my Pipeline, I have created Library variables and which referenced to my Variable Library parameters.
I have used these Library variables in Email activity. Here its working fine and variables are resolved.

And then, I tried with my notebook activity.
I have created base parameters too in my notebook activity which are pointed to the Library variables.
And then I ran my pipeline but didnt give positive results.

Pipeline got failed with same error.

[v-veshwara-msft]

Hi @nareshraavi086 ,

Thanks for the update and thanks to @tayloramy  for the suggestions.

Since the variables resolve correctly in the email activity but the notebook still fails to start, it indicates the values are not reaching the notebook session during the pipeline run.

This also fits with the note in the documentation that the configure cell must be the first code cell in a pipeline run, otherwise the session cannot be created.

https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#spark-session-conf... 

A reliable way to pass values into the configure block during pipeline execution is to send them as base parameters from the pipeline and reference those parameters inside the configure JSON. This avoids issues with variable libraries not being picked up at session start.

The following configuration pattern is an example that works reliably when run through a pipeline:

%%configure -f
{
"defaultLakehouse": {
"name": {
"parameterName": "lh_silver_name",
"defaultValue": "LH_SILVER"
},
"id": {
"parameterName": "lh_silver_id",
"defaultValue": ""
},
"workspaceId": {
"parameterName": "lh_silver_workspace_id",
"defaultValue": ""
}
}
}

Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn

You only need to define these base parameters in the notebook activity. They are available to the session even without a parameters cell, so the configure block can resolve them when the notebook starts.

If you can try this pattern in your test notebook within the same pipeline, it should confirm whether the values are being passed through correctly.

Hope this helps. Please reach out for further assistance.

Thank you.