topic Overwrite a table with different schema in Dataflow Gen2 in Dataflow

Overwrite a table with different schema in Dataflow Gen2

amaaiia — Tue, 05 Mar 2024 07:17:44 GMT

I have a T1 table in Lakehouse L1 that I have ingested with Dataflow Gen2. I have another Lakehouse L2 with a shortcut to T1 (S2).

My T1 table in source now has one more column, so I redo the ingest with Dataflow Gen2 to overwrite the T1 table in L1, but I can't, because the schema has changed (one new column). The only way I see is to delete T1 in L1 and reload it. But if I do this, all references to T1 by its id, like the shortcut itself, are unlinked.

How can I update the schema of a table by doing overwrite with Dataflow Gen2 so that the shortcut and table ids are kept?

Re: Overwrite a table with different schema in Dataflow Gen2

rubayatyasmin — Tue, 05 Mar 2024 09:24:06 GMT

Hi,

You should be able to change the schema in the destination settings.

This what I get in f64 capacity. In f2 capacity you should be able to refresh your schema.

Hope this helps.

Re: Overwrite a table with different schema in Dataflow Gen2

amaaiia — Fri, 08 Mar 2024 15:13:26 GMT

I'm using Premium F64 capacity.

I can see Dynamic Schema option when I create a NEW table, but not for Existing table:

For Existing table I can only see Refresh destination schema option, but is not working, because I click on it and the destination schema keeps the same. My source new schema has one more column, and it's not being added to destination schema:

Re: Overwrite a table with different schema in Dataflow Gen2

rubayatyasmin — Fri, 08 Mar 2024 21:06:59 GMT

You should be able to see the option. The screenshot I gave you was from existing table. From transform environment.

Re: Overwrite a table with different schema in Dataflow Gen2

amaaiia — Mon, 11 Mar 2024 12:41:52 GMT

rubayatyasmin As you can see here, I'm also in Transform environment, I'm on Exisiting table and I can't select Dynamic Schema.

Why may be this happening?

Re: Overwrite a table with different schema in Dataflow Gen2

v-cboorla-msft — Thu, 14 Mar 2024 11:51:34 GMT

Hi amaaiia

Apologize for the inconvenience that you are facing here.

Please reach out to our support team to gain deeper insights and explore potential solutions, If its a bug, we will definitely would like to know and properly address it. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

Re: Overwrite a table with different schema in Dataflow Gen2

v-cboorla-msft — Fri, 15 Mar 2024 11:22:26 GMT

Hi amaaiia

I'm following up on my previous inquiry to see if you've had a chance to create a support ticket for this issue. If a ticket has been created, I would appreciate it if you could provide the ticket number for our reference. This will allow us to track the progress of the issue.

Thanks.

Re: Overwrite a table with different schema in Dataflow Gen2

rubayatyasmin — Mon, 18 Mar 2024 05:17:48 GMT

Hi again,

I was facing the same issue today so I removed data destination, removed previous table from the lakehouse and added the data destination again and it was all good. Let's try it out and see if this works.