Forum Discussion

RobertCressey's avatar
RobertCressey
Frequent Visitor
2 years ago
Solved

Data Pipeline Error while decoding

Data that is written to my Datalake through a data pipeline can not be read using Spark. (even the autogenerated code when clicking open in notebook from the datalake)   Tables that have been writt...
pipelines
  • RobertCressey's avatar
    RobertCressey
    2 years ago

    From MS support ticket:

     

    Issue fixed - 

     updated to data gateway 3000.218.7 or newer.

     

    Some tables issues persisted after a refresh however we assumed this was due to the way that checkpoints are saved in the json , so we dropped all the tables and re-imported and this fixed the issues.

Nikos83's avatar
Nikos83
Icon for Advocate I rankAdvocate I
2 years ago

We have exactly the same situation. We load data from Microsoft Dynamics 2012 on-premises using a pipeline (copy cmd). There is a gateway in between. The copy cmd ensures that the data is loaded into our data lakehouse (with the overwrite option). We use notebooks to transform the data. The error message does not occur immediately for us, nor is it reproducible. It just happens suddenly. I tried to execute the statement: ALTER TABLE MDDP_brons.bta_tob_fiscalcalendarperiod SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3') on the table. However, this yields the same error message. When I drop the table and go through the pipeline again, the problem disappears. But it can also come back without any explanation on my end.