topic Re: Error using data imported through DataFlow when using pyspark in Dataflow

Error using data imported through DataFlow when using pyspark

m_cherriman — Mon, 13 Nov 2023 10:52:39 GMT

Since Sunday (13-11-2023) I have been getting this error message on data imported through DataFlows. Previously they have been working. Has anyone else experienced this issue?

SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3).

The Data Flows are linking to an on prem sql server database and I'm able to query the table in the sql endpoint, but cannot read the table into a dataframe. I've tried restricting the data to a couple of lines to see if there is unusual item in any of the queries, but nothing obvious and they previously worked. To get round this I've had to extract the data into csv files (using a python script in vs code) and then uploading the csvs to Files folders and then importing those files into a new table, which can be read using pyspark.

I've had a couple of issues with Data Flows over the last week which makes me wonder how reliable they are.

Re: Error using data imported through DataFlow when using pyspark

v-cboorla-msft — Mon, 13 Nov 2023 11:14:45 GMT

Hi m_cherriman

Thanks for using the Microsoft Fabric Community.

Apologies for the inconvenience that you are facing here.

At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.

Thanks

Re: Error using data imported through DataFlow when using pyspark

v-cboorla-msft — Tue, 14 Nov 2023 08:00:05 GMT

Hi m_cherriman

Thanks for using Fabric Community.
Apologies for the issue you have been facing.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

Re: Error using data imported through DataFlow when using pyspark

KA78 — Sat, 18 Nov 2023 07:02:42 GMT

Any news on this issue? We're experiencing the same problem. This is a real show stopper.

Re: Error using data imported through DataFlow when using pyspark

souldish — Wed, 22 Nov 2023 21:10:46 GMT

We are experiencing same issue with csv data coming from dataflow gen 2

Re: Error using data imported through DataFlow when using pyspark

KA78 — Mon, 27 Nov 2023 09:37:22 GMT

Hi,

v-cboorla-msft is there perhaps an update you can give us on this issue?

Also, if anyone should have a workaround, we would aprecieate it greatly. This is a real showstopper now.

Our ERP data resides in an on premise database that we can only access with ODBC. So the only way to get this data into Fabric (when using only fabric - we want to stay with SaaS) is by using Dataflow gen2. And now the tables created by Dataflow gen2, can't be used in a notebook. So we can't use a notebook / pyspark to tranform our data...

Does anyone know of a workaround, inside Fabric, so that we can combine ODBC on prem data + notebooks? Any help is greatly appreciated.

m_cherriman , did you perhaps find a solution?

Thanks in advance for any update or insight into this issue.

Re: Error using data imported through DataFlow when using pyspark

v-cboorla-msft — Mon, 27 Nov 2023 11:30:42 GMT

Hi KA78

Thanks for using Fabric Community and posting your question.

Can you please create a new post as the initial ask is different from your issue? We will definitely look into the issue and help.

Thanks for understanding.

Re: Error using data imported through DataFlow when using pyspark

KA78 — Tue, 28 Nov 2023 09:47:26 GMT

Thanks @v-cboorla-msft for your reply. I'll create a new post for my question regarding the workaround.

Although, the root of the problem is the same as the initial ask. Is there perhaps something you can share about the status of the inital ask : "Error using data imported through DataFlow when using pyspark". Is this a known issue?

(related to this error : "SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3)." )

Re: Error using data imported through DataFlow when using pyspark

KA78 — Wed, 29 Nov 2023 12:16:17 GMT

So, I found the root of the bug and also a temporary workaround.

The bug is that Dataflows gen2 writes this in the json delta log files :

{"protocol":{"minReaderVersion":1,"minWriterVersion":2,"readerFeatures":[],"writerFeatures":[]}}

That's the reason for this error message : requirement failed: Mismatched minReaderVersion and readerFeatures

A temporary fix is to change this string in all the _delta_log json files for the table to:

{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}

You can change the log files within the onelake in your files browser:

Re: Error using data imported through DataFlow when using pyspark

jcvega — Thu, 30 Nov 2023 17:57:48 GMT

Thansk for the workarround.. I had the same issue.. How can I edit these files?

Re: Error using data imported through DataFlow when using pyspark

KA78 — Fri, 01 Dec 2023 09:48:04 GMT

You can change the files with the Onelake plugin that you can install in your windows file explorer :https://www.microsoft.com/en-us/download/details.aspx?id=105222

Keep in mind that every time you execute the dataflow, you'll have to modify the new log file that has been created for this workaround. Hopefully they'll quickly fix this bug.

Re: Error using data imported through DataFlow when using pyspark

KA78 — Fri, 01 Dec 2023 14:00:45 GMT

UPDATE:

I also created a ticket for this bug. I just received an email from support telling me it will be fixed this week ! 😀

"Would like to inform you that I have received an update from engineering team. The update says it was a dataflow bug whose fix will be done and rolled out to production by the end of this week.

For Dataflows Gen2, the fix should reach all production regions by the end of this week."

Thanks Microsoft for the great support!

It would be nice if we could see this kind of major issue in the known issues list in the future : Microsoft Fabric Known Issues

Re: Error using data imported through DataFlow when using pyspark

KA78 — Fri, 01 Dec 2023 11:31:39 GMT

jcvega I'm just tagging you here so you'll get a notice that you don't have to put effort in the workaround. Should be fixed this week.

Re: Error using data imported through DataFlow when using pyspark

jcvega — Fri, 01 Dec 2023 11:37:29 GMT

Thanks very much... I suppose that Microsoft is working on this.. it doesn't make sense every day doing this process (if the DF runs each day)

Re: Error using data imported through DataFlow when using pyspark

naanii — Tue, 12 Dec 2023 14:22:55 GMT

Hello,

I am still experiencing the same issue.

Any update about this case?

Re: Error using data imported through DataFlow when using pyspark

KA78 — Tue, 12 Dec 2023 14:43:44 GMT

Hi naanii, You'll also have to update your power bi gateway.

Re: Error using data imported through DataFlow when using pyspark

naanii — Tue, 12 Dec 2023 15:16:10 GMT

Thanks for the solution!

Re: Error using data imported through DataFlow when using pyspark

souldish — Thu, 28 Dec 2023 05:28:40 GMT

Still seeing issues with this....not connected to gateway and using DFG2 to connect to REST API and copy the data to a table in the lakehouse.

Any updates on this issue? it makes DFG2 completely unusable.

Re: Error using data imported through DataFlow when using pyspark

Reidy — Thu, 28 Dec 2023 07:03:47 GMT

Hi,

We are still having this issue. What do you mean by "update your PowerBI gateway"? Any help would be greatly appreciated.

Re: Error using data imported through DataFlow when using pyspark

miguel — Thu, 28 Dec 2023 10:27:15 GMT

Hey! I'd suggest starting a new thread so we can talk about your particular scenario in more detail. If your dataflow is not using a gateway then the suggestion made by other users in this thread won't apply to your scenario.

You can also raise a support ticket so you can engage directly with our support team to troubleshoot the issue.