Forum Discussion

SergioTorrinha's avatar
SergioTorrinha
Icon for Resolver II rankResolver II
2 years ago
Solved

Data Pipeline Fail on Notebook due to unexplainable Spark Error

Hi everyone!   Today my datapipeline failed, because a Spark SQL notebook I have developed (and tested it was running whithout any issues) last month, failed to run. The fail reason, aparently, is...
Need Help
pipelines
  • SergioTorrinha's avatar
    SergioTorrinha
    2 years ago

    Hi Corar !

    I just got my issue solved today. I was informed by microsoft support that there was a bug in Spark runtime 1.2 that is now fixed.
    In my case, the way I resolved the issue was to completly delete the table that I was not able to query via SparK SQL, and then re-run the pipeline that was generating the table.

     

    I hope this helps fixing your issue.

matkvaid's avatar
matkvaid
Icon for Helper III rankHelper III
2 years ago

This is marked as solved, but i have same error randomly. I have recreated datalake tables that notebook was not running for, they work for one time and then start showing same errors, so the bug is not fixed? I have created support ticket, but i can see in how i get the replies that it will take months to solve?? How is it again, that trial has started counting days till the end, but Fabric as a product seems is still so not finished product????

prom's avatar
prom
Frequent Visitor
2 years ago

Hello

This is a bug. As a workaround, You can try:

  • use runtime 1.1 
  • manually delete checkpoint file in _delta_log 
  • switch to parquet/ext tables as a destination for CP

.