Dataflow Gen2 to Lakehouse - missing rows

Hi Anonymous,

We sincerely appreciate your inquiry on the Microsoft Fabric Community Forum.

Based on my understanding, the reason you observe different top rows in Dataflow Gen2 as compared to the Lakehouse is due to automatic schema inference and header promotion applied by Dataflow Gen2 during the import of flat files (such as .txt). Rows containing sequences like "------------------" or label lines such as "Field name" are automatically identified as headers or disregarded as metadata. Only the structured data after header promotion is loaded into the Lakehouse, which is why those initial rows are missing. This automatic step occurs even if no transformations have been explicitly applied.

Kindly follow the steps below which may help in resolving the issue:

1. Dataflow Gen2 may automatically promote the first row as headers, which might actually be your data row. Please review and disable header promotion if it is unintended.
2. Ensure that the Lakehouse table schema matches your source schema. Missing or mismatched columns could result in skipped rows.
3. Rows containing null values or unexpected data types may fail to load silently. Please validate the rows in the Dataflow Gen2 preview before publishing.
4. Review the Dataflow Gen2 run logs for any warnings or errors related to skipped rows or schema mismatches.
5. Test with a simplified .txt file to isolate issues related to headers or data formatting.

If you find this response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members who may have similar queries.

Thank you.