Forum Discussion
Resolver IIData Pipeline Fail on Notebook due to unexplainable Spark Error
Hi Corar !
I just got my issue solved today. I was informed by microsoft support that there was a bug in Spark runtime 1.2 that is now fixed.
In my case, the way I resolved the issue was to completly delete the table that I was not able to query via SparK SQL, and then re-run the pipeline that was generating the table.I hope this helps fixing your issue.
I am experiencing exactly the same behaviour;
I have an error envolving data pipelines and notebook. I write raw logs to a table in the bronze lakehouse. After that in the notebook i read the data incrementally, clean it and process it to silver and gold lakehouse.
For some time this always works without problems but after some runs, spark cannot read the logs delta table in the notebook anymore. On read the following error appears:
Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - SparkRuntimeException, Error value - Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures.
newInstance(class scala.Tuple3).' :
I use the spark and delta table runtime default settings of the workspace, also tested different ones. I executed the data pipeline and notbook independently.
Only thing i found was an error where Dataflow Gen2 caused a problem, but the delta log looks fine in my case:
Re: Error using data imported through DataFlow whe... - Microsoft Fabric Community
I hope i am not doing something stupid here and I am thankful for any guidance and support.
Resolver IIHi Corar !
I just got my issue solved today. I was informed by microsoft support that there was a bug in Spark runtime 1.2 that is now fixed.
In my case, the way I resolved the issue was to completly delete the table that I was not able to query via SparK SQL, and then re-run the pipeline that was generating the table.
I hope this helps fixing your issue.
Hi SergioTorrinha ,
thank you for the update.
I have resolved the problem in a similar way, multiple times since the weekend, by creating a backup as parquet and recreating the delta table afterward.
Unfortunately after multiple runs it reappears and the table cannot be read anymore, last time 2 hours ago (Mismatched Version error)
Nevertheless, maybe the fix is not yet rolled out in my region, so I am waiting a bit.
Thank you again !
- SergioTorrinha
Corar
No problem, glad I could help.
In your case I think it would be best to open a support ticket then. Perhaps the solution for your issue is somewhat distinct than mine. I don't know for sure when microsoft rolled out the bug fix, but I would dare to say it was a couple days ago, hence my sugestion.
Hope you see your issue fixed. 🙂
Thanks for chipping in.
Hello,
Did You resolve the problem? We have same similiar problem - we are using Copy Activity to copy data from on-prem to lakehouse. Every five runs we get same error: Mismatched minReaderVersion and readerFeatures.
After 5 runs, there is a file created in _delta_log catalog: 00000000000000000010.checkpoint.parquet. It consits of a column protocol with minReaderVersion:1 and readerFeatures: [] (and similiar settings for writer). According do delta.io docs its not correct, readerFeatures can be used with minReaderVersion >=3.
One solution is to delete whole table. You can remove checkpoint, but it will be recreated during load activity. We set overwrite option for destination table in Lakehouse, in Copy Activity, but checkpoints are still generated. There should be option to let sb to replace delta table (with delta log removed) in Copy Act. We are using Pippelines to copy staging tables from on premises. We don't want to mix parquet tables with delta for performance reasons.
Also played with parameter delta.checkpointInterval (default 10?) for the table. Its getting removed every checkpoint from tblproperties, besides its not documented in fabric.
Best Regards