Dataflow Gen2 to Lakehouse - missing rows

[junmin]

Hi, it's my first time using fabric lakehouse. i'm not sure why i see all of my data in dataflow gen2 but some rows are missing after publishing the dataflow into lakehouse destination.

For example, this is what i see as my top 4 rows in dataflow gen2

But this is what i'm seeing in the top 4 rows in my lakehouse

why? i dont understand. i dont have any transformation in my dataflow gen2. i only imported the data source which is a txt file, not even changing data type or promote headers.

appreciate any help please.

[v-pnaroju-msft]

Hi junmin,

We are following up to see if your query has been resolved. Should you have identified a solution, we kindly request you to share it with the community to assist others facing similar issues.

If our response was helpful, please mark it as the accepted solution and provide kudos, as this helps the broader community.

Thank you.

[v-pnaroju-msft]

Hi junmin,

We wanted to check in regarding your query, as we have not heard back from you. If you have resolved the issue, sharing the solution with the community would be greatly appreciated and could help others encountering similar challenges.

If you found our response useful, kindly mark it as the accepted solution and provide kudos to guide other members.

Thank you.

[timesprayerme]

Hey! It sounds like there might be an issue with how the data is being synced to the Lakehouse. Even without transformations, check for any row limits, file formatting issues, or data type mismatches that could be causing rows to be excluded. Hope that helps!

[alouadifama]

if you didn't explicitly promote headers, sometimes Dataflow Gen2 automatically infers and promotes headers, especially for .txt or .csv files.

[v-pnaroju-msft]

Hi junmin,

We have not received a response from you regarding the query and were following up to check if you have found a resolution. If you have identified a solution, we kindly request you to share it with the community, as it may be helpful to others facing a similar issue.

If you find the response helpful, please mark it as the accepted solution and provide kudos, as this will help other members with similar queries.

Thank you.

[v-pnaroju-msft · Accepted solution]

Hi @junmin,

We sincerely appreciate your inquiry on the Microsoft Fabric Community Forum.

Based on my understanding, the reason you observe different top rows in Dataflow Gen2 as compared to the Lakehouse is due to automatic schema inference and header promotion applied by Dataflow Gen2 during the import of flat files (such as .txt). Rows containing sequences like "------------------" or label lines such as "Field name" are automatically identified as headers or disregarded as metadata. Only the structured data after header promotion is loaded into the Lakehouse, which is why those initial rows are missing. This automatic step occurs even if no transformations have been explicitly applied.

Kindly follow the steps below which may help in resolving the issue:

1. Dataflow Gen2 may automatically promote the first row as headers, which might actually be your data row. Please review and disable header promotion if it is unintended.
2. Ensure that the Lakehouse table schema matches your source schema. Missing or mismatched columns could result in skipped rows.
3. Rows containing null values or unexpected data types may fail to load silently. Please validate the rows in the Dataflow Gen2 preview before publishing.
4. Review the Dataflow Gen2 run logs for any warnings or errors related to skipped rows or schema mismatches.
5. Test with a simplified .txt file to isolate issues related to headers or data formatting.

If you find this response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.

Thank you.