Forum Discussion

m_cherriman's avatar
m_cherriman
Advocate I
2 years ago

Error using data imported through DataFlow when using pyspark

Since Sunday (13-11-2023) I have been getting this error message on data imported through DataFlows.  Previously they have been working.  Has anyone else experienced this issue?   SparkRuntimeExcep...
Dataflows
Need Help
KA78's avatar
KA78
Advocate I
2 years ago

Hi, 

v-cboorla-msft is there perhaps an update you can give us on this issue?

Also, if anyone should have a workaround, we would aprecieate it greatly. This is a real showstopper now. 

 

Our ERP data resides in an on premise database that we can only access with ODBC. So the only way to get this data into Fabric (when using only fabric - we want to stay with SaaS) is by using Dataflow gen2. And now the tables created by Dataflow gen2, can't be used in a notebook. So we can't use a notebook / pyspark to tranform our data...

 

Does anyone know of a workaround, inside Fabric, so that we can combine ODBC on prem data + notebooks? Any help is greatly appreciated. 

 

m_cherriman , did you perhaps find a solution?

 

Thanks in advance for any update or insight into this issue.

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi KA78 

     

    Thanks for using Fabric Community and posting your question.

    Can you please create a new post as the initial ask is different from your issue? We will definitely look into the issue and help.

     

    Thanks for understanding.

    • KA78's avatar
      KA78
      Advocate I
      2 years ago

      Thanks @v-cboorla-msft for your reply. I'll create a new post for my question regarding the workaround. 

      Although, the root of the problem is the same as the initial ask. Is there perhaps something you can share about the status of the inital ask : "Error using data imported through DataFlow when using pyspark". Is this a known issue?


      (related to this error : "SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3)." )