Forum Discussion
Dataflow Gen2 - Dynamic schema
Hi,
The documentation for Dynamic schema in Dataflow Gen2 says:
"When the dataflow is refreshed, your table is dropped and recreated. Your dataflow refresh fails if you have any relationships or measures added to your table."
However, when I tested Dataflow Gen2 Dynamic schema on a Lakehouse table - which is part of a direct lake semantic model - the refresh did not fail even if I have relationships and measure added to that table. So it works very nice, even more convenient than the description in the documentation 😀
This way, I am able to add and remove columns from the table, by using Dataflow Gen2. And the table still works fine in the direct lake semantic model 😀
I am curious, is the documentation not updated, or am I missing something?
Thank you 😀
I am also wondering if the word "flow" here should be removed. I don't understand why the word "flow" is there:
"Dynamic schema: When choosing dynamic schema, you allow for schema changes in the data destination when you republish the dataflow. Because you aren't using managed mapping, you still need to update the column mapping in the dataflow destination flow when you make any changes to your query. When the dataflow is refreshed, your table is dropped and recreated. Your dataflow refresh fails if you have any relationships or measures added to your table."
Hi frithjof_v
The internal team has received the feedback. They are going to make changes in the document soon. The document would be revised and made changes accordingly.
Thank you for the valuable feedback.
6 Replies
Hi frithjof_v
Thanks for using Fabric Community and bringing this to our notice.
At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
ThanksHi frithjof_v
The internal team has received the feedback. They are going to make changes in the document soon. The document would be revised and made changes accordingly.
Thank you for the valuable feedback.Anonymous
I did some further testing. It may seem that while the relationships on the tables still work afterwards, the foreign key and unique constraints on the tables are missing afterwards (after refreshing a Dataflow Gen2 with Dynamic schema):
I can confirm this behavior as well using a Custom Semantic Model built of the SQL Endpoint of my Lakehouse.
I wonder if we would get the error if the relationships are created in the Default Semantic Model instead of a Custom Semantic Model?
If I remember correctly, I tried both the default and the custom semantic model.
I also tried creating another lakehouse in another workspace, where all the tables were just shortcuts pointing to the original lakehouse.
These also didn't fail after I ran the Dataflows Gen2 and changed the schema of the table in the original lakehouse.
Which is great 😃 I just feel it's contradicting the information in the documentation, so I'm wondering if it's a supported feature or not.
I can confirm - As at June 25 - this is still showing. Unless it is now actually deleting measures
