Forum Discussion
How does Dataflow gen2 handles flat (txt) files
Hi junminn30
In Lakehouse, you will not see blank rows becuase it's has limitation (show upto 1000 rows). I replicated your scenario.
Lets's say I have a txt file(this i uploaded from my machine into dataflow gen2) called Sales_Fabric with column called SalesOrderLinenumber having blanksLater I published this into my Sample_LH lakehouse. At that I'm able to see like this. Assume that I'm not able to see blank rows.
But if i query the table in SQL analytics end point, It clearly showing blanks in SalesOrderLinenumber column
I think it's sorting in differet way in the lakehouse.
Hope you are clear now. Let me know if it works
Thank you!!
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!
Super UserHi junminn30 ,
Dataflow Gen2 automatically applies schema inference and may filter out blank rows, headers, or footers during the publishing process to the Lakehouse—even if you don’t explicitly define transformation steps.
To retain those rows, try the following:
- In the source step, disable any automatic schema detection or column promotion.
- Use the “Use first row as headers” option carefully—if your file has multiple header/footer rows, this might skip or misinterpret them.
- Add a custom transformation step to explicitly preserve all rows, including blanks, by treating the file as raw text and parsing it manually (e.g., using Power Query’s Lines.FromBinary and Table.FromList).
Let me know if you want help writing that transformation step.
Thank you so much. I've been quite busy with other projects and didn't have the time to respond. I will try and get back to this message if I face any problems.
