Data Pipeline Error while decoding

[RobertCressey]

Data that is written to my Datalake through a data pipeline can not be read using Spark. (even the autogenerated code when clicking open in notebook from the datalake)

Tables that have been written using a Data pipeline to a database table are then getting the following error message when opened in spark runtime 1.2 or 1.3. (the tables are readable in 1.1 however setting this causes other issues)

SparkRuntimeException: Error while decoding: java.lang.IllegalArgumentException: requirement failed: Mismatched minReaderVersion and readerFeatures. newInstance(class scala.Tuple3).

the data is coming from an on-prem SQL server, I see that other people had this issue with dataflow gen2 when it was first introduced before it was fixed for.

I believe this is a bug however if anyone has a workaround (other than using 1.1) that would be appreciated.

[Nikos83]

We have exactly the same situation. We load data from Microsoft Dynamics 2012 on-premises using a pipeline (copy cmd). There is a gateway in between. The copy cmd ensures that the data is loaded into our data lakehouse (with the overwrite option). We use notebooks to transform the data. The error message does not occur immediately for us, nor is it reproducible. It just happens suddenly. I tried to execute the statement: ALTER TABLE MDDP_brons.bta_tob_fiscalcalendarperiod SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3') on the table. However, this yields the same error message. When I drop the table and go through the pipeline again, the problem disappears. But it can also come back without any explanation on my end.

[RobertCressey · Accepted solution]

From MS support ticket:

Issue fixed - 

 updated to data gateway 3000.218.7 or newer.

Some tables issues persisted after a refresh however we assumed this was due to the way that checkpoints are saved in the json , so we dropped all the tables and re-imported and this fixed the issues.

[Anonymous]

Hi @RobertCressey ,

It was great to know that you were able to get to a resolution . We expect you to keep using this forum and also motivate others to do that same . You can always help other community members by answering to their queries

[RobertCressey]

Hi, 

I have raised a ticket with Microsoft support.

however, this is the error message I'm getting. 

[Anonymous]

Hi @RobertCressey ,

I appreciate for opening a support ticket. 

Can you please provide the support ticket number as it would help us to track for more information.

In meantime can you please take a backup of the table and try to alter the property by running the SQL command below in Spark notebook:

ALTER TABLE <table-identifier> SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3')

This might help to fix the issue. 

Can you please try and let me know?

[Anonymous]

Hi @RobertCressey ,

Thanks for using Fabric Community.
Can you please share the screenshot if what you are doing and error you are facing?

I can help you better if you can provide these details.