topic Re: ErrorCode: LakehouseOperationFailed in Data Engineering

ErrorCode: LakehouseOperationFailed

fieldsl — Tue, 05 Aug 2025 13:15:59 GMT

We have a Master Pipeline that runs overnight, and we’re encountering the LakehouseOperationFailed error several times a week. The failure doesn’t consistently occur on the same Copy Data activity, but the ones it does fail on share a common pattern: they pull data from a SQL database and write it as a .parquet file to the Lakehouse. These errors typically happen when multiple pipelines are running simultaneously, all attempting to write .parquet files to the Lakehouse at the same time.

Here is the full error:

ErrorCode=LakehouseOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Lakehouse operation failed for: An error occurred while sending the request.. Workspace: '?'. Path: '?/Files/2025-08-05.parquet'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=The underlying connection was closed: An unexpected error occurred on a send.,Source=System,''Type=System.IO.IOException,Message=Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.,Source=System,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=System,'

Based on this, any ideas on what is happening along with possible ideas on where to look and what to chagne?

Re: ErrorCode: LakehouseOperationFailed

v-saisrao-msft — Wed, 06 Aug 2025 09:51:16 GMT

Hi @fieldsl,

The issue may be due to several pipelines writing .parquet files to the Lakehouse at the same time. Consider staging the Copy Data activities with dependencies to prevent parallel writes, and review the Monitoring Hub for more information about pipeline activity and resource usage.

Thank you.

Re: ErrorCode: LakehouseOperationFailed

v-saisrao-msft — Mon, 11 Aug 2025 03:17:07 GMT

Hi @fieldsl,

Have you had a chance to review the solution we shared earlier? If the issue persists, feel free to reply so we can help further.

Thank you.

Re: ErrorCode: LakehouseOperationFailed

BalajiL — Mon, 11 Aug 2025 04:48:10 GMT

Try to reduce concurrency to write the data into lakehouse at pipeline level.

Re: ErrorCode: LakehouseOperationFailed

fieldsl — Mon, 11 Aug 2025 13:04:45 GMT

My appologies I was off at a conference this week and had the oppertunity to ask this question to a few people. My understanding is that there shouldn't be any issue writing to files in parallel to the lakehouse at the same time. As such, I have not updated the pipeline to run sequential.

That said, I did update both of the Copy data Activities to do a Retry but my understanding is that this shouldn't be necessary either. This issue also doesn't occer ever day, it's kind of hit or miss through out the week, so it's hard to say if this actually addressed the issue or not. I'll monitor through the rest of the week and see what happens. Should I encounter this again I will consider spliting the pipeline to run Sequential but would like to understand what and why it's happening before making that change.

Just to add a little more context, the one file contains 3 rows of data and the other file contains about 1,000 of records and we are on a F64 Capacity. The files that are being written to the lakehouse are never writing to the same location. Every pipeline has it's own folder that it's writing to.

Re: ErrorCode: LakehouseOperationFailed

BalajiL — Mon, 11 Aug 2025 13:53:56 GMT

@fieldsl As I said earlier, there are many concurrent threads running in parallel. Try to optimize the concurrency at the pipeline level, that would resolve the issue.

Re: ErrorCode: LakehouseOperationFailed

fieldsl — Mon, 11 Aug 2025 14:31:30 GMT

Appologies @BalajiL I someone skipped right over your post. I will set the Concurrency today and see how this behaves this week.

Re: ErrorCode: LakehouseOperationFailed

fieldsl — Wed, 13 Aug 2025 17:27:55 GMT

It’s been a little over a week, and we haven’t encountered the issue again. At this point, I believe any one of the three solutions could have resolved it individually:

Updating the pipeline to run sequentially and prevent parallel writing of .parquet files
Adjusting the concurrency
Adding a retry step

That said, this still doesn’t explain the root cause. The limited information in the error makes it difficult to pinpoint exactly what happened. While any of the above solutions works, they feel more like temporary fixes to an underlying issue I’d still like to understand. Knowing the cause would help us explain what’s happening and choose the most appropriate long-term solution.

For now, I’ll consider this resolved.