Error using data imported through DataFlow when using pyspark

[m_cherriman]

Since Sunday (13-11-2023) I have been getting this error message on data imported through DataFlows.  Previously they have been working.  Has anyone else experienced this issue?

SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3).

The Data Flows are linking to an on prem sql server database and I'm able to query the table in the sql endpoint, but cannot read the table into a dataframe.  I've tried restricting the data to a couple of lines to see if there is unusual item in any of the queries, but nothing obvious and they previously worked.  To get round this I've had to extract the data into csv files (using a python script in vs code) and then uploading the csvs to Files folders and then importing those files into a new table, which can be read using pyspark.

I've had a couple of issues with Data Flows over the last week which makes me wonder how reliable they are.

[KA78]

Any news on this issue? We're experiencing the same problem. This is a real show stopper.

[v-cboorla-msft]

Hi @m_cherriman 

Thanks for using the Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.

Thanks

[jeremycolson1]

Has there been a resolution to this issue? We are doing a POC w/ Fabric... and following Microsoft's own instructions we are getting this error. Doing the most basic of use cases... using a one step ADF pipeline w/ a Copy task to load an example dataset (NYC Taxi) into the lakehouse (no on-premises complications, no data gateways, etc.). Then attempting to access that table in a Notebook... fails w/ this same error.

Error:

SparkRuntimeException: 
Error while decoding: java.lang.IllegalArgumentException: requirement failed: 
Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3)

Delta log appears fine:

{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}

[Reidy]

Hey,

We got around this by downgrading the spark version in the environment used for our Notebooks. 1.1 works fine, 1.2 has the issue.

[v-cboorla-msft]

Hi @m_cherriman 

Thanks for using Fabric Community.
Apologies for the issue you have been facing.  

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

[souldish]

We are experiencing same issue with csv data coming from dataflow gen 2