Fabric pyspark notebook - Failed due to configuration Issue while using %%configure -f

[nareshraavi086]

Hi,

I have a Fabric Pyspark notebook with First cell using %%configure -f to set default lakehouse.

%%configure -f

{

  "defaultLakehouse": {

    "name": {

      "variableName": "$(/**/VL_Wxxxx_Paxxx/DefaultLHName)"

    },

    "id": {

      "variableName": "$(/**/VL_Wxxxx_Paxxx/DefaultLHID)"

    }

  }

}

I have a variable Library where these Default variables are defined. 
If I run the notebook directly then it is working fine. where as if i run the notebook through data pipelines with notebook activity then I am getting below error.
""
Notebook execution failed at Notebook service with http status code - 'NotFound', please check the Run logs on Notebook, additional details - '{
"code": 404,
"message": "No notebook execution state found in database for the runId - exxxxxxxxxxxxxxxxxxxxx5",
"result": {
"errorMessage": null,
"details": null
}
}' :

""

Could you please help me out on this why this error is popping up.?

[tayloramy · Accepted solution]

Hi @nareshraavi086, 

This is interesting. This makes me thing that when run from the pipeline, the variable library is not working correctly. 

Can you try in the same pipeline, disabling the notebook session, and adding an email session that emails you the variables from the variable library? 

Then if that works, can you make a new notebook that just logs the output of the variable library to a file in a lakehouse somewhere? 

THis will help troubleshoot where exactly the breakdown is. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

[v-veshwara-msft]

Hi @nareshraavi086 ,
We wanted to kindly follow up regarding your query. If you need any further assistance, please reach out.
Thank you.

[Youraj8271]

Thank you @tayloramy for the helpful guidance.
I truly appreciate the clarity you provided.
As someone starting out in Fabric Data Engineering, insights like yours make the learning path much easier.
Thanks again for taking the time to reply.

[tayloramy · Accepted solution]

Hi @nareshraavi086, 

This is interesting. This makes me thing that when run from the pipeline, the variable library is not working correctly. 

Can you try in the same pipeline, disabling the notebook session, and adding an email session that emails you the variables from the variable library? 

Then if that works, can you make a new notebook that just logs the output of the variable library to a file in a lakehouse somewhere? 

THis will help troubleshoot where exactly the breakdown is. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

[nareshraavi086]

Thanks for the suggestions.
In my Pipeline, I have created Library variables and which referenced to my Variable Library parameters.
I have used these Library variables in Email activity. Here its working fine and variables are resolved.

And then, I tried with my notebook activity.
I have created base parameters too in my notebook activity which are pointed to the Library variables.
And then I ran my pipeline but didnt give positive results.

Pipeline got failed with same error.

[v-veshwara-msft]

Hi @nareshraavi086 ,

Thanks for the update and thanks to @tayloramy  for the suggestions.

Since the variables resolve correctly in the email activity but the notebook still fails to start, it indicates the values are not reaching the notebook session during the pipeline run.

This also fits with the note in the documentation that the configure cell must be the first code cell in a pipeline run, otherwise the session cannot be created.

https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#spark-session-conf... 

A reliable way to pass values into the configure block during pipeline execution is to send them as base parameters from the pipeline and reference those parameters inside the configure JSON. This avoids issues with variable libraries not being picked up at session start.

The following configuration pattern is an example that works reliably when run through a pipeline:

%%configure -f
{
"defaultLakehouse": {
"name": {
"parameterName": "lh_silver_name",
"defaultValue": "LH_SILVER"
},
"id": {
"parameterName": "lh_silver_id",
"defaultValue": ""
},
"workspaceId": {
"parameterName": "lh_silver_workspace_id",
"defaultValue": ""
}
}
}

Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn

You only need to define these base parameters in the notebook activity. They are available to the session even without a parameters cell, so the configure block can resolve them when the notebook starts.

If you can try this pattern in your test notebook within the same pipeline, it should confirm whether the values are being passed through correctly.

Hope this helps. Please reach out for further assistance.

Thank you.

[nareshraavi086]

Hi,

Thanks for the suggessions.
Whatever the configureation mentioned in %%Configure -f works fine.
However, it is using direct variables.

All my variables/parameters are defined in Variable Library.
If I use Variable Library, Then it is not able to expan variables and not able to Initiate the  notebook run.

%%configure -f

{

  "defaultLakehouse": {

    "name": {

      "parameterName": "$(/**/VL_Woxxxeters/DefaultLHName)",

      "defaultValue": "xxxx"

    },

    "workspaceId": {

      "parameterName": "$(/**/VL_Woxxxers/DefaultWorkspaceID)",

      "defaultValue": "xxxxx"

    }

  }

}

[v-veshwara-msft · Accepted solution]

Hi @nareshraavi086 ,

Thanks for the clarification.

From the behaviour you’re seeing, it looks like the Variable Library tokens such as $(/**/VL_xxx/Value) might not be getting expanded when the notebook is started from a pipeline. In pipeline-triggered runs, the Spark session is created before those values are resolved, so the %%configure block could be receiving the literal token text. Since the configure cell needs to be the first executable cell in pipeline runs, the session might not be able to restart and the run ends up failing even though the same JSON works in interactive mode.

You may try the same pattern mentioned earlier where the Variable Library values are resolved inside the pipeline and then passed into the notebook as base parameters. Using parameterName inside the configure block seems to give more consistent results in pipeline scenarios, since those parameters are available to the session right when it starts.

Similar threads and references you might find useful:

Solved: How to use variable-library values in a notebook - Microsoft Fabric Community

Solved: Re: Change Default Lakehouse - Microsoft Fabric Community

Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.

Thank you.

[v-veshwara-msft]

Hi @nareshraavi086 ,
We wanted to kindly follow up regarding your query. If you need any further assistance, please reach out.
Thank you.

[v-veshwara-msft]

Hi @nareshraavi086 ,
Just wanted to check if the response provided was helpful. If further assistance is needed, please reach out.
Thank you.