topic Not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns added in Data Engineering

Not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns added

mcrreddyt — Wed, 29 May 2024 18:08:14 GMT

I am not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns added for the existing Dataflow. The data flow has failed and not picked the additional 2 columns. Is there an option in future to dynamically add the columns to the Fabric dataflow.

I tried editing the existing dataflow and can see the new columns of the source file in the source table of the dataflow but tried using both new and existing table option but not able to see the newly added columns in the destination table name. Is this a limitation in Fabric Lakehouse or am i missing anything.

I am not even able to add columns in the Lakehouse delta table created using Dataflow from the SQL Analytics endpoint using alter table <tablename> add columns syntax and looks like there is a limitation.

I am not even able to drop the Lakehouse delta table from the SQL Analytics endpoint created using Dataflow drop table <tablename> syntax and looks like there is a limitation here as well.

Please let us know all these are limitations in the Fabric Lakehouse or am i missing something.

Re: Not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns a

Anonymous — Thu, 30 May 2024 10:58:18 GMT

Hi @mcrreddyt ,

Thanks for using Fabric Community.
Currently ALTER TABLE command is not supported for tables in Lakehouse and Warehouse.

For workaround, please refer below thread -
Solved: Re: Add or remove column in Lakehouse table - Microsoft Fabric Community

Hope this is helpful. Do let me know incase of further queries.

Re: Not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns a

Anonymous — Fri, 31 May 2024 11:19:50 GMT

Hi @mcrreddyt ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns a

jwinchell40 — Fri, 31 May 2024 12:53:07 GMT

@mcrreddyt - After you pulled the 2 columns into the Dataflow UI, did you remap the Lakehouse connection and table? I've had it pick-up new columns before going through it that way. In my case though, I'm doing an "overwrite" vs an "append" so that may cause different behavior.

I pretty much use the Notebook utilities to modify anything structural with the Lakehouse; like adding columns or repartitioning. Its faster.

Re: Not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns a

frithjof_v — Mon, 03 Jun 2024 11:00:16 GMT

@mcrreddyt I think what you need should be possible, ref. https://community.fabric.microsoft.com/t5/General-Discussion/Dropping-and-recreating-lakehouse-table/m-p/3836094/highlight/true#M6322

For loading the columns from Excel to the Dataflow, maybe it helps to mark the content area in Excel as a Table, and point the Dataflow to the table, instead of referring to Excel sheet name in the dataflow.

Otherwise, I think in the Advanced Editor in the dataflow maybe you need to adjust the number of columns to import from the Excel sheet.

Re: Not able to add 2 additional columns into Dataflow Gen2 as the source excel file has 2 columns a

Anonymous — Mon, 03 Jun 2024 08:57:02 GMT