Notebook Activities with a custom environment stopped working when run from a Data Pipeline suddenly

[JosueMolina]

I have a Pipeline that runs 9 notebooks, each notebook in a different workspace with it's own lakehouse and it's own nearly identical custom environment. 

This pipeline had been running with no issue for months now, and it suddenly stopped working this past 21st with the following error on all 9 notebooks:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook. Error: [InvalidEnvironmentArtifact] Invalid provided environment artifact The attached environment 1ba2fec9-b664-4bcb-bac7-b9ce72d4caf6 should be located in the same capacity of SynapseNotebook artifact 40fbbf94-18aa-40a2-a937-00d4ab7cae9e. Please check your artifact information. [Root activity id: e5fe33af-0b88-4311-ad33-a1f029f80b2d] HTTP status code: 400.' : 

 Remember that all 9 notebooks are in a different workspace with a different environment. Some of these environment-notebook combinations have not been modified recently, but all 9 show the same error. 

If I open the Notebook and Run it directly, it will run without issue. 

All 9 workspaces do have HC enabled for Notebooks executed by Pipelines.

[v-prasare]

We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding and participation.

[v-prasare]

Hi @JosueMolina, @OZaharia, @Tough 

Sorry for the inconvenience caused here.

As @JosueMolina mentioned this was previously working, @joanfatz please do raise a support ticket this will allow the a dedicated team to investigate and provide a resolution. Please include details of your setup, test results, and the impact on your workflows when submitting the ticket.

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thanks,

Prashanth Are

MS Fabric Community Support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

[OZaharia]

This just started to happen to me also. In a workspace, I have a brand new notebook, brand new pipeline and a brand new environment without any custom configuration inside. If I run the notebook manually, it works, if I run the notebook with the scheduler it works, if I run the notebook from pipeline, it fails.

On other workspaces with the same licence attached with the same setup, it works even from pipeline, so this is realy strange. 

[OZaharia]

Hi, yesterday evening this started to happen to me also. I've created a new notebook with just a print statement inside, a new environment - just the "containter" - I haven't added custom libraries or change something in the environment - then I setup the notebook to use the environment. 

If I run the notebook manual, it works, however, the session start time was considerable (over 2 minutes), if I run the notebook from the pipeline it fails with the exact same message, if I run the notebook with scheduler also, it works.

On another workpaces, the same thing works perfect, no issues, in the same tenant, in the same capacity, so this is really strange

What happened with Fabric in the past weeks, first Lakehouses are not working with Git, then Lakehouses are not working with Deployment Pipelines, and currently my deployment pipeline is down (can't move anything from one stage to another), now this.

[saxn3]

I have the exact same problem for one of my notebooks. Did you manage to fix it?

[joanfatz]

I am getting the same error, but the environment, pipelines and notebooks are all in the same workspace.  Process ran find last Sunday.

[v-prasare]

@JosueMolina,

Thanks for actively participating in MS Fabric community supprt.

please refer to this how to run notebooks as pipeline activities and discusses the security context of running notebooks. It emphasizes the importance of understanding the execution context, which can be influenced by environment and capacity configurations

This article details how to create, configure, and use environments in Fabric and It highlights the significance of environment configurations and their impact on notebook executions

Let me know if above resouces helps you in resolving the error your facing?

Thanks,

Prashanth Are

MS Fabric community support

Did we answer your question? Mark post as a solution, this will help others!

If our response(s) assisted you in any way, don't forget to drop me a "Kudos"

[JosueMolina]

Thanks @v-prasare , 

But this is a configuration that had been working perfectly for months, and without any modification on my end, stopped working on the 21st. 

[JosueMolina]

I did some tests. The Pipeline is in Workspace A and each pair of Notebook and Environment is in a different Workspace; let's call one of these Workspace B. 

Workspace A has an F16 Capacity and Workspace B has an F64 Capacity. Doing a test of moving Workspace B to the F16 Capacity makes the error disappear.

Sadly, whatever was changed internally broke a previously working pipeline and moving the Workspace B to an F16 is not a solution for us.

Workspace B is a Gold layer workspace with a Lakehouse and we require an F64 to exist so users do not require a Pro license to consume our reports using this data.

[Tough]

Hey, just commenting to let you know you aren't alone. I support a Fabric environment running on an F64 capacity. About two weeks ago the pipeline completely stopped running and fails on the first step with the same error you are encountering. 

I've been trying to troubleshoot it myself, as the pipeline creator is currently on leave. I haven't made any progress and don't have another capacity to easily swap the pipeline to. Short term I have been running each step manually to keep the data fresh... But this is terrible. 

It was working perfectly fine and only minor changes have been made to the notebooks. 

Every notebook step and environment used in the pipeline is on F64 capacity out of Canada Central.

I haven't had dedicated time to call submit a support request, but looks like I'll be doing that tomorrow.

Thanks for posting this, was tough to find any info on this error.