Forum Discussion

m_cherriman's avatar
m_cherriman
Advocate I
2 years ago

Error using data imported through DataFlow when using pyspark

Since Sunday (13-11-2023) I have been getting this error message on data imported through DataFlows.  Previously they have been working.  Has anyone else experienced this issue?   SparkRuntimeExcep...
Dataflows
Need Help
v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi m_cherriman 

 

Thanks for using the Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.

 

Thanks

  • jeremycolson1's avatar
    jeremycolson1
    Advocate I
    2 years ago

    Has there been a resolution to this issue? We are doing a POC w/ Fabric... and following Microsoft's own instructions we are getting this error. Doing the most basic of use cases... using a one step ADF pipeline w/ a Copy task to load an example dataset (NYC Taxi) into the lakehouse (no on-premises complications, no data gateways, etc.). Then attempting to access that table in a Notebook... fails w/ this same error.

     

    Error:

    SparkRuntimeException: 
    Error while decoding: java.lang.IllegalArgumentException: requirement failed: 
    Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3)

     

    Delta log appears fine:

    {"protocol":{"minReaderVersion":1,"minWriterVersion":2}}

     

     

    • Reidy's avatar
      Reidy
      Regular Visitor
      2 years ago

      Hey,

       

      We got around this by downgrading the spark version in the environment used for our Notebooks. 1.1 works fine, 1.2 has the issue.

       

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi m_cherriman 

     

    Thanks for using Fabric Community.
    Apologies for the issue you have been facing.  

    Please go ahead and raise a support ticket to reach our support team:

    https://support.fabric.microsoft.com/support

    After creating a Support ticket please provide the ticket number as it would help us to track for more information.

     

    Thank you.

    • souldish's avatar
      souldish
      Frequent Visitor
      2 years ago

      We are experiencing same issue with csv data coming from dataflow gen 2