Forum Discussion
Error using data imported through DataFlow when using pyspark
So, I found the root of the bug and also a temporary workaround.
The bug is that Dataflows gen2 writes this in the json delta log files :
- {"protocol":{"minReaderVersion":1,"minWriterVersion":2,"readerFeatures":[],"writerFeatures":[]}}
That's the reason for this error message : requirement failed: Mismatched minReaderVersion and readerFeatures
A temporary fix is to change this string in all the _delta_log json files for the table to:
- {"protocol":{"minReaderVersion":1,"minWriterVersion":2}}
You can change the log files within the onelake in your files browser:
Thansk for the workarround.. I had the same issue.. How can I edit these files?
You can change the files with the Onelake plugin that you can install in your windows file explorer :https://www.microsoft.com/en-us/download/details.aspx?id=105222
Keep in mind that every time you execute the dataflow, you'll have to modify the new log file that has been created for this workaround. Hopefully they'll quickly fix this bug.
Thanks very much... I suppose that Microsoft is working on this.. it doesn't make sense every day doing this process (if the DF runs each day)
I'm still having this issue.
I load data from my on-prem sql server using the `copy data` option, then read this on the Fabric Notebooks.
I frequently get this error:
SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures.
When running a pipeline with the:copy data (which reads from on-prem and copies to LH Table with overwrite) --> Notebook --> (anything else)
This will break when runing the 2nd or 3rd time. Super frustrating, and feels like this shouldn't really be happening at this point.
I just started having the same issue.
