Forum Discussion
Error using data imported through DataFlow when using pyspark
Thansk for the workarround.. I had the same issue.. How can I edit these files?
You can change the files with the Onelake plugin that you can install in your windows file explorer :https://www.microsoft.com/en-us/download/details.aspx?id=105222
Keep in mind that every time you execute the dataflow, you'll have to modify the new log file that has been created for this workaround. Hopefully they'll quickly fix this bug.
Thanks very much... I suppose that Microsoft is working on this.. it doesn't make sense every day doing this process (if the DF runs each day)
I'm still having this issue.
I load data from my on-prem sql server using the `copy data` option, then read this on the Fabric Notebooks.
I frequently get this error:
SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures.
When running a pipeline with the:copy data (which reads from on-prem and copies to LH Table with overwrite) --> Notebook --> (anything else)
This will break when runing the 2nd or 3rd time. Super frustrating, and feels like this shouldn't really be happening at this point.
I just started having the same issue.
Hi,
We managed to resolve this issue by changing the spark runtime version. You'll need to restart your notebook after changing the environment.
