Forum Discussion

mcrreddyt's avatar
mcrreddyt
Microsoft Employee
2 years ago

Not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns added

I am not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns added for the existing Dataflow. The data flow has failed and not picked the additional 2 columns. Is there an option in future to dynamically add the columns to the Fabric dataflow.

 

I tried editing the existing dataflow and can see the new columns of the source file in the source table of the dataflow but tried using both new and existing table option but not able to see the newly added columns in the destination table name.  Is this a limitation in Fabric Lakehouse or am i missing anything. 

 

I am not even able to add columns in the Lakehouse delta table created using Dataflow from the SQL Analytics endpoint using alter table <tablename> add columns syntax and looks like there is a limitation.

 

I am not even able to drop the Lakehouse delta table from the SQL Analytics endpoint created using Dataflow drop table <tablename>  syntax and looks like there is a limitation here as well. 

 

Please let us know all these are limitations in the Fabric Lakehouse  or am i missing something.

 

data engineering
data pipeline
environment
lakehouse

5 Replies

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi mcrreddyt ,

      We haven’t heard from you on the last response and was just checking back to see if your query was answered.
      Otherwise, will respond back with the more details and we will try to help .

      Thanks

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi mcrreddyt ,

        We haven’t heard from you on the last response and was just checking back to see if your query was answered.
        Otherwise, will respond back with the more details and we will try to help .

        Thanks

  • jwinchell40's avatar
    jwinchell40
    Resolver III
    2 years ago

    mcrreddyt - After you pulled the 2 columns into the Dataflow UI, did you remap the Lakehouse connection and table?  I've had it pick-up new columns before going through it that way.  In my case though, I'm doing an "overwrite" vs an "append" so that may cause different behavior.

     

    I pretty much use the Notebook utilities to modify anything structural with the Lakehouse; like adding columns or repartitioning.  Its faster.

  • frithjof_v's avatar
    frithjof_v
    Community Champion
    2 years ago

    mcrreddyt I think what you need should be possible, ref. https://community.fabric.microsoft.com/t5/General-Discussion/Dropping-and-recreating-lakehouse-table/m-p/3836094/highlight/true#M6322

     

    For loading the columns from Excel to the Dataflow, maybe it helps to mark the content area in Excel as a Table, and point the Dataflow to the table, instead of referring to Excel sheet name in the dataflow. 

    Otherwise, I think in the Advanced Editor in the dataflow maybe you need to adjust the number of columns to import from the Excel sheet.