Data Pipeline Fail on Notebook due to unexplainable Spark Error

Hi SergioTorrinha 

Thanks for using Fabric Community.

Apologies for the issue that you are facing here.

This might require a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Thanks.

Hi everyone!

This issue is still occourring on my end and I have not been contacted from support so far.
Can someone from Microsoft have a look at this, please? Or, at least, help with the support ticket 2401110020000508 ?

I have a pipeline developed since last month, which was working fine till this error been throwned out. I am trying to sell Fabric internally, and for that I need this pipeline to be operational, otherwise my 'internal selling operation' will be a failure.


Thank you.

I am experiencing exactly the same behaviour;

I have an error envolving data pipelines and notebook. I write raw logs to a table in the bronze lakehouse. After that in the notebook i read the data incrementally, clean it and process it to silver and gold lakehouse. 

For some time this always works without problems but after some runs, spark cannot read the logs delta table in the notebook anymore. On read the following error appears:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - SparkRuntimeException, Error value - Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures.
newInstance(class scala.Tuple3).' :

I use the spark and delta table runtime default settings of the workspace, also tested different ones. I executed the data pipeline and notbook independently. 

Only thing i found was an error where Dataflow Gen2 caused a problem, but the delta log looks fine in my case:

Re: Error using data imported through DataFlow whe... - Microsoft Fabric Community

I hope i am not doing something stupid here and I am thankful for any guidance and support.

Hi v-cboorla-msft,

is there any information if there was a fix and this is rolled out globally?

Error ist unfortunately still happing for me, last time 2min ago and i need to recreate the table every time.

Best

Any news on that? I got same error. I have 3 identical tables, notebook script was not working for one of them. Got that table recreated, everything was fine for one day. Next day, other two tables got the same error. 

This is marked as solved, but i have same error randomly. I have recreated datalake tables that notebook was not running for, they work for one time and then start showing same errors, so the bug is not fixed? I have created support ticket, but i can see in how i get the replies that it will take months to solve?? How is it again, that trial has started counting days till the end, but Fabric as a product seems is still so not finished product????