Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Dataflow Gen 2 Refresh Doesn't Show New Columns

Hello everyone,   I'm trying to see new added columns in Lakehouse.   I have created 2 Dataflow Gen 2 in Fabric.   First Dataflow Connects to a XLSX in SharePoint, reads "Table S"  and save it ...
Dataflow
Need Help
GilbertQ's avatar
GilbertQ
Super User
3 years ago

Hi Anonymous 

 

As far as I understand when using dataflows gen2, it only keeps record of the current columns. If you want to add new columns, you would then need to update the dataflow and add the 2 new columns.

 

The reason for the additional parquet files is that on every load it creates a new file with the new data. It then uses the delta lake log file to know which one to read from.

  • Anonymous's avatar
    Anonymous
    Not applicable
    3 years ago

    Thanks for replay GilbertQ,

     

    When I load a table into PBI from an excel file, if I add a column in source and refresh PBI, I can see column immediately in PBI, without doing manual column add step. Is this not possible while loading data with Dataflow Gen2?

    Top image is before refresh. At bottom image, I added a column in excel, saved it and refresh PBI data set and saw column. I want this happen in Dataflow Gen 2 as well.

     

     

     

     

     

    About parquet files, will they stay as it is, so there is no way to remove them automatically  or prevent creating them while refresh?