topic Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse in Dataflow

Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

Meister_Kaio — Thu, 13 Jul 2023 14:04:40 GMT

Hello!

I am attempting to replicate a medallion architecture in fabric using DataFlows and Lakehouses.

Initially, my first Dataflow executed flawlessly and generated tables in the bronze lakehouse. However, my current objective is to transfer these tables from the bronze lakehouse to the silver lakehouse.

After publishing the flow, I encountered an issue where three out of six queries within the flow failed, resulting in an error message.

Upon inspecting my refreshed silver lakehouse, I discovered that all six tables from the bronze lakehouse had been loaded as parquet files in the silver lakehouse. Normally, the flow would load tables, not parquet files, to my lakehouse.

I am now curious as to why the flow proceeded to load the files despite the failure to publish in the correct manner.

Thanks!

Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

miguel — Thu, 13 Jul 2023 16:04:14 GMT

hey! could you please share the error messages from the refresh history of the dataflow in question that tries to load the data to your silver lakehouse?

Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

Meister_Kaio — Fri, 14 Jul 2023 07:12:47 GMT

Small Update:
Upon refreshing the silver lakehouse, I observed that all the files were successfully transferred to tables. The queries that encountered failures in the dataflow do not contain any content, while the queries that successfully passed through display their contents.

Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

Meister_Kaio — Fri, 14 Jul 2023 07:34:34 GMT

The error from the dataflow:
Mashup Exception Data Source Error Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Microsoft SQL: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) Details: DataSourceKind = Lakehouse;DataSourcePath = Lakehouse;Message = A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.);ErrorCode = -2146232060;Number = 10054;Class = 20

I encountered the same issue with other dataflows, that had an other error message:

Mashup Exception Expression Error Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Expression.Error: The column "[columnName]" from the table "[tableName]" has not been found. Details: "[columnName]"

I have already discovered solutions to the mentioned errors. However, I believe that my issue is unrelated to those errors, as it has occurred with various different errors.

Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

miguel — Fri, 14 Jul 2023 15:35:09 GMT

Are you using a gateway? if yes, please check out this article and let us know if it helps:

On-premises data gateway considerations for output destinations in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

Meister_Kaio — Wed, 19 Jul 2023 11:12:07 GMT

I'm not using a gateway.
My workound is to disable staging. Seems to work fine if i disbale staging for all queries except for one.

Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

miguel — Wed, 19 Jul 2023 23:43:13 GMT

Dataflows and the output destination create tables in a lakehouse, but not raw files.

are you able to repro this behavior in a new workspace ? If yes, could you please share the steps to repro the scenario ?

Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

SinghIsKing — Thu, 20 Jul 2023 05:20:21 GMT

I am also facing this issue. Dataflow fails but files are geneated in Lakehouse

Re: Dataflow Gen2 publish fails, but still loads parquet files to destionation lakehouse

st_0999 — Wed, 02 Aug 2023 21:20:21 GMT

I had exactly the same error.

I removed Enable Staging from all tables, but when I tried loading the last table to a Lakehouse, got this error:

MashupException.Error: DataSource.Error: Error in replacing table's content with new data in a version: #{0}. Details: Message = We can't insert null data into a non-nullable column.;Message.Format = We can't insert null data into a non-nullable column.

I've specified the datatype for all columns, but a few columns do have null values in them (on purpose). But I guess, null values should still be allowed. Otherwise that's very strange. Is this causing this? Is there anyway around this?