Forum Discussion

RobertCressey's avatar
RobertCressey
Frequent Visitor
2 years ago
Solved

Data Pipeline Error while decoding

Data that is written to my Datalake through a data pipeline can not be read using Spark. (even the autogenerated code when clicking open in notebook from the datalake)   Tables that have been writt...
pipelines
  • RobertCressey's avatar
    RobertCressey
    2 years ago

    From MS support ticket:

     

    Issue fixed - 

     updated to data gateway 3000.218.7 or newer.

     

    Some tables issues persisted after a refresh however we assumed this was due to the way that checkpoints are saved in the json , so we dropped all the tables and re-imported and this fixed the issues.

RobertCressey's avatar
RobertCressey
Frequent Visitor
2 years ago

Hi, 

I have raised a ticket with Microsoft support.

 

however, this is the error message I'm getting. 

 

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi RobertCressey ,

I appreciate for opening a support ticket. 

Can you please provide the support ticket number as it would help us to track for more information.


In meantime can you please take a backup of the table and try to alter the property by running the SQL command below in Spark notebook:

ALTER TABLE <table-identifier> SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3')
This might help to fix the issue. 

Can you please try and let me know?