Forum Discussion

m_cherriman's avatar
m_cherriman
Advocate I
2 years ago

Error using data imported through DataFlow when using pyspark

Since Sunday (13-11-2023) I have been getting this error message on data imported through DataFlows.  Previously they have been working.  Has anyone else experienced this issue?   SparkRuntimeExcep...
Dataflows
Need Help
v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi m_cherriman 

 

Thanks for using the Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.

 

Thanks

jeremycolson1's avatar
jeremycolson1
Advocate I
2 years ago

Has there been a resolution to this issue? We are doing a POC w/ Fabric... and following Microsoft's own instructions we are getting this error. Doing the most basic of use cases... using a one step ADF pipeline w/ a Copy task to load an example dataset (NYC Taxi) into the lakehouse (no on-premises complications, no data gateways, etc.). Then attempting to access that table in a Notebook... fails w/ this same error.

 

Error:

SparkRuntimeException: 
Error while decoding: java.lang.IllegalArgumentException: requirement failed: 
Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3)

 

Delta log appears fine:

{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}

 

 

  • Reidy's avatar
    Reidy
    Regular Visitor
    2 years ago

    Hey,

     

    We got around this by downgrading the spark version in the environment used for our Notebooks. 1.1 works fine, 1.2 has the issue.