topic Re: Missing Rows when loading Dataflow to Lakehouse in Dataflow

Missing Rows when loading Dataflow to Lakehouse

frances_c — Fri, 22 Mar 2024 16:36:29 GMT

I've been encountering a peculiar issue during the loading process from Dataflow Gen2 to our Lakehouse. Although the data transformation in Gen2 appears to be successful, when I try to load the queries into our Lakehouse, only a fraction of the expected rows are being successfully loaded. Despite thorough checks on formatting and other potential issues, I haven't been able to identify any apparent problems. The loading process itself doesn't report any errors, which makes this limitation even more baffling. If anyone has faced a similar issue or has any insights into what might be causing this discrepancy, I would greatly appreciate your input and guidance.

Re: Missing Rows when loading Dataflow to Lakehouse

miguel — Sun, 24 Mar 2024 18:23:37 GMT

Do you happen to have a way in which we could replicate the issue? Does your query have any sort of dynamic function reference such as Date.LocalTime, Number.Random or others?

Re: Missing Rows when loading Dataflow to Lakehouse

v-cboorla-msft — Tue, 26 Mar 2024 07:56:17 GMT

Hi frances_c

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

Re: Missing Rows when loading Dataflow to Lakehouse

frances_c — Tue, 26 Mar 2024 12:05:37 GMT

Hi, i am yet to work on this again sorry. But i will table this issue for now after discussing with a colleague. I realized that the lakehouse was just showing a data preview, and was not supposed to show all the data. Thank you.