Forum Discussion
m_cherriman
2 years agoAdvocate I
Error using data imported through DataFlow when using pyspark
Since Sunday (13-11-2023) I have been getting this error message on data imported through DataFlows. Previously they have been working. Has anyone else experienced this issue? SparkRuntimeExcep...
jeremycolson1
2 years agoAdvocate I
Has there been a resolution to this issue? We are doing a POC w/ Fabric... and following Microsoft's own instructions we are getting this error. Doing the most basic of use cases... using a one step ADF pipeline w/ a Copy task to load an example dataset (NYC Taxi) into the lakehouse (no on-premises complications, no data gateways, etc.). Then attempting to access that table in a Notebook... fails w/ this same error.
Error:
SparkRuntimeException:
Error while decoding: java.lang.IllegalArgumentException: requirement failed:
Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3)
Delta log appears fine:
{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}
Reidy
2 years agoRegular Visitor
Hey,
We got around this by downgrading the spark version in the environment used for our Notebooks. 1.1 works fine, 1.2 has the issue.