Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Dataflow Gen 2 Refresh Doesn't Show New Columns

Hello everyone,

 

I'm trying to see new added columns in Lakehouse.

 

I have created 2 Dataflow Gen 2 in Fabric.

 

First Dataflow Connects to a XLSX in SharePoint, reads "Table S"  and save it an a new Table in Lakehouse. Let's call this table as "Table A"

 

Second Dataflow connects to same XLSX in SharePoint, read "Table S" but overwrite it into "Table A", (existing table).

 

First Dataflow's goal is to create Table A in Lakehouse, Second Dataflow's goal is to refresh data periodicly in Table A .

 

I added a new column into "Table S" in SharePoint. When Second Dataflow run, I couldn't see new column in Table A.

 

I tried to run First Dataflow manually, to update Table A, instead of refresh. But still can't see new column in Table A. What should I do to see new columns in Table A?

 

For more details,

I made transformation on Python Notebook , a pipeline run this Notebook and Second Dataflow. Created a PBI report and it is connected to Table A. Everything is work as expceted, except I can't see new column that added to "Table S". 

 

Table S is as below

 

Table A is as below

 

 

Bonus question, everytime Dataflow works, I have a new parquet file in the lakehouse, how to avoid it, it's just unnecary tables, need manual effort or python code to delete them, which I believe should be a better way.

 

Thanks in advance

 

Dataflow
Need Help

2 Replies

  • GilbertQ's avatar
    GilbertQ
    Super User
    3 years ago

    Hi Anonymous 

     

    As far as I understand when using dataflows gen2, it only keeps record of the current columns. If you want to add new columns, you would then need to update the dataflow and add the 2 new columns.

     

    The reason for the additional parquet files is that on every load it creates a new file with the new data. It then uses the delta lake log file to know which one to read from.

    • Anonymous's avatar
      Anonymous
      Not applicable
      3 years ago

      Thanks for replay GilbertQ,

       

      When I load a table into PBI from an excel file, if I add a column in source and refresh PBI, I can see column immediately in PBI, without doing manual column add step. Is this not possible while loading data with Dataflow Gen2?

      Top image is before refresh. At bottom image, I added a column in excel, saved it and refresh PBI data set and saw column. I want this happen in Dataflow Gen 2 as well.

       

       

       

       

       

      About parquet files, will they stay as it is, so there is no way to remove them automatically  or prevent creating them while refresh?