Forum Discussion

fieldsl's avatar
fieldsl
Frequent Visitor
1 year ago
Solved

ErrorCode: LakehouseOperationFailed

We have a Master Pipeline that runs overnight, and we’re encountering the LakehouseOperationFailed error several times a week. The failure doesn’t consistently occur on the same Copy Data activity, b...
data engineering
Data Pipeline
Lakehouse
  • v-saisrao-msft's avatar
    v-saisrao-msft
    1 year ago

    Hi fieldsl,

    The issue may be due to several pipelines writing .parquet files to the Lakehouse at the same time. Consider staging the Copy Data activities with dependencies to prevent parallel writes, and review the Monitoring Hub for more information about pipeline activity and resource usage.

     

    Thank you.

  • BalajiL's avatar
    BalajiL
    1 year ago

    fieldsl  As I said earlier, there are many concurrent threads running in parallel. Try to optimize the concurrency at the pipeline level, that would resolve the issue. 

     

     

fieldsl's avatar
fieldsl
Frequent Visitor
1 year ago

It’s been a little over a week, and we haven’t encountered the issue again. At this point, I believe any one of the three solutions could have resolved it individually:

  • Updating the pipeline to run sequentially and prevent parallel writing of .parquet files

  • Adjusting the concurrency

  • Adding a retry step

That said, this still doesn’t explain the root cause. The limited information in the error makes it difficult to pinpoint exactly what happened. While any of the above solutions works, they feel more like temporary fixes to an underlying issue I’d still like to understand. Knowing the cause would help us explain what’s happening and choose the most appropriate long-term solution.

 

For now, I’ll consider this resolved.