topic Re: Problem with semantic model refresh after data transformation in Data Flow Gen2 in Dataflow

Problem with semantic model refresh after data transformation in Data Flow Gen2

DALI007 — Fri, 16 Feb 2024 14:18:48 GMT

Hello,

I have created a DataFlow Gen2 in Microsoft Fabric. I made some transformations by adding a new column "type_site", as shown in the screenshot "image1". However, when I publish my DataFlowGen2 and try to refresh the data in my lakehouse (LakeHouse.png) , the changes are not being reflected. Note that I did not receive any error message. Can you please help me resolve this issue?

Thank you in advance for your assistance.

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

miguel — Fri, 16 Feb 2024 17:10:45 GMT

Hey!

I am not sure I completely follow. How was the table in your destination (LakeHouse) created? Are you able to check the schema to see what data types it requires and that is bound to?
is the dataflow inserting the right values according to the definition of the schema of the lakehouse table that you see in the lakehouse experience and table browser?

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

DALI007 — Sun, 18 Feb 2024 21:57:01 GMT

Hello Miguel,
I inserted the first time a table (a schema ) which does not contain the last column "type_site". But after a transformation I wanted to insert a new table that will replace the existing table with the addition of this new column "type_site". I do not know if this action is possible to do it automatically by just updating the semantic model or I must manually delete the old table and insert the new table again. I hope I was clear in my explanations.
Thank you in advance for your return

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

miguel — Mon, 19 Feb 2024 02:22:34 GMT

Hey!

Im not sure I follow. You mention a semantic model, a Lakehouse and a DataflowGen2.

The Dataflow Gen2 cannot load data directly to a semantic model. However, it can load data to a Lakehouse.

If you initially created a table in your Lakehouse to load data to, how did you create such table? Did you use the Dataflow Gen2 to create it? Assuming that such table needs to change its schema, how do you plan to change its schema? Or are you trying to create a completely new table? If you try to create a new table, how will you create such table?

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

DALI007 — Mon, 19 Feb 2024 07:58:54 GMT

Hello,
Exactly I used a Dataflow Gen2, a semantic model and a Lakehouse.
To my knowledge, with the default settings, a Direct Lake dataset is automatically refreshed when the data is loaded into a Lakehouse.
Yes, Initially I published manually the dataFlow Gen2 which made me the creation of the table in the lakeHouse.
In a second time I made a transformation on the table at DataFlow Gen2 (Adding a column "type_site" in the table).
my problem how to replace the existing table automatically or manually if the shemas of the table latter changes (in my case : addition of this new column for example) ? Thank you

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

miguel — Mon, 19 Feb 2024 14:39:43 GMT

Is the issue that the new column is reflected in the Lakehouse but not in the automatic semantic model? Or that the new column is not reflected in the Lakehouse at all?

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

DALI007 — Mon, 19 Feb 2024 14:48:04 GMT

Yes, the new column is not reflected in the Lakehouse at all

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

miguel — Mon, 19 Feb 2024 14:51:26 GMT

A Dataflow Gen2 doesn't change the schema of a table in the destination at the moment.

You can either create a new table by going through the "data destination" configuration for your query and defining a new table, or make the changes in your backend (your LakeHouse) to alter the schema of a table and then update your dataflow gen2 to load the data to the updated schema. In both scenarios you'll have to go into the dataflow gen2 and update the configuration for the data destination.

hope this helps!

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

amolt — Tue, 20 Feb 2024 16:21:34 GMT

Hi miguel ,

I have a similar problem.
I initially created a schema in a warehouse. I use this warehouse to create a report using PowerBi Desktop. From this point on, I can no longer see the schema changes I make in the warehouse. If I add a column to a dw table, I can't see it in the PowerBI model.
Is this normal?
How can I update the PowerBi semantic model schema when I make changes to the source structure?
What's more, even when I delete the table from the semantic model, and add it again, I get back exactly the same structure as before, without the new column...

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

miguel — Tue, 20 Feb 2024 16:35:29 GMT

I'd recommend posting your full scenario in the Power BI forum:

Power BI forums - Microsoft Fabric Community

It doesn't seem to be related to a Dataflow Gen2, so it might be best for someone from the Data Warehouse team or the Power BI to take a look at it

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

Anonymous — Wed, 21 Feb 2024 16:26:57 GMT

Hi DALI007
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

DALI007 — Thu, 22 Feb 2024 08:30:56 GMT

Hello Anonymous,
Not yet, actually my problem as it is indicated in the tittre of subject how to do a refresh of semantic model after an update of dataflow Gen2. To my knowledge, with the default settings, a Direct Lake dataset is automatically refreshed when the data is loaded into a Lakehouse. But in my case I have the dataflowGen2 which is updated every day (it is a parametrage that I set up) but the dataset (semantic model) is not updated.
Thank you in advance for your help.
Thanks

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

amolt — Thu, 22 Feb 2024 09:11:29 GMT

Hello DALI007 ,

check out this post, maybe it'll help :

Re: How can I update the PowerBi semantic model sc... - Microsoft Fabric Community

Re: Problem with semantic model refresh after data transformation in Data Flow Gen2

DALI007 — Thu, 22 Feb 2024 15:22:17 GMT

Hello amolt,
it worked, thanks for your help.
Thank you