Forum Discussion
ErrorCode: LakehouseOperationFailed
Hi fieldsl,
The issue may be due to several pipelines writing .parquet files to the Lakehouse at the same time. Consider staging the Copy Data activities with dependencies to prevent parallel writes, and review the Monitoring Hub for more information about pipeline activity and resource usage.
Thank you.
My appologies I was off at a conference this week and had the oppertunity to ask this question to a few people. My understanding is that there shouldn't be any issue writing to files in parallel to the lakehouse at the same time. As such, I have not updated the pipeline to run sequential.
That said, I did update both of the Copy data Activities to do a Retry but my understanding is that this shouldn't be necessary either. This issue also doesn't occer ever day, it's kind of hit or miss through out the week, so it's hard to say if this actually addressed the issue or not. I'll monitor through the rest of the week and see what happens. Should I encounter this again I will consider spliting the pipeline to run Sequential but would like to understand what and why it's happening before making that change.
Just to add a little more context, the one file contains 3 rows of data and the other file contains about 1,000 of records and we are on a F64 Capacity. The files that are being written to the lakehouse are never writing to the same location. Every pipeline has it's own folder that it's writing to.