Forum Discussion

christianunr's avatar
christianunr
Helper I
1 year ago
Solved

New column added from a Dataflow is not appearing when adding a Destination Source

EDIT: I seem to be facing the same issue as this person: https://community.fabric.microsoft.com/t5/Dataflow/Overwrite-a-table-with-different-schema-in-Dataflow-Gen2/m-p/3752099

 

Hello, I am trying to create a Gen2 Dataflow where I can use Power Query to do some data transformations and then load back into the Lakehouse by overwrite.

I'm new to Fabric and trying to set up our data infrastructure on here.

The current goal (or what I am trying to achieve and what makes sense in my head) is:

  1. A data pipeline that copies data from an on-prem SQL Server instance into a Lakehouse on Fabric.

    1. I got this to work

  2. Then, invoking a Dataflow Gen2 that adds a column to the table.

    1. I am having issues with this. In the screenshot below, I created the new column.

    2. Then when I choose the destination, the column is not appearing.

  3. Unless I am not understanding how Dataflows work, I am wanting this to overwrite the table in the Lakehouse with these transformations.

Copy data from on-prem SQL Server > Lakehouse > perform transformations with Power Query > overwrite same table in Lakehouse with transformations.

 

I can see the new column being added through the Applied Steps.

 

But in the destination settings, it is not appearing.

 

Here is the data pipeline I have so far.

Dataflow
Need Help
  • christianunr's avatar
    christianunr
    1 year ago

    Thanks. What I ended up doing is creating a staging lakehouse where I copy the data to first from SQL Server and then in the Dataflow, add the source destination into a "production" lakehouse.

3 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi christianunr ,

     

    Only tables created from dataflow in the lakehouse can detect dynamic changes in the output of the powery query.

    In other words, if you want to detect new columns when mapping columns, you need to select the 'create table' ub dataflow instead of selecting an existing table.

     

     

    Besides, if you already have a table in your lakehouse that was created by something other than dataflow,

    you can also create a new column via notebook.

     

    Best Regards,

    Wearsky

    • christianunr's avatar
      christianunr
      Helper I
      1 year ago

      Thanks. What I ended up doing is creating a staging lakehouse where I copy the data to first from SQL Server and then in the Dataflow, add the source destination into a "production" lakehouse.

  • NetaZimut's avatar
    NetaZimut
    New Member
    1 year ago

    Also encountered this issue with a client using a Dataflow connected to a Delta Sharing catalog from Databricks; 

    The dataflow would show the full table with up to date data but the PBIX would not pick up the missing columns but kept the data up to date for the displayed ones.

     

     

    Here's another fix that worked for us :

    • Open PowerQuery
    • Delete table that is not refreshing columns from the queries
    • Save PBIX & Close
    • Recreate query to the table

    Cheers