topic Re: SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2 in Dataflow

SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2

frithjof_v — Sun, 21 Apr 2024 13:23:00 GMT

Hi,

I created a Lakehouse with 2 tables:

Fact table: RandomNumbers

Dimensions table: Dim_Date

I created a relationship between the tables in the SQL Analytics Endpoint

When I query the [INFORMATION_SCHEMA].[TABLE_CONSTRAINTS] view, I can see that these relationships have been registered as constraints on the tables in the SQL Analytics Endpoint.

When I connect to the RandomNumbers table of the SQL Analytics Endpoint from Power Query in Power BI Desktop, I can see that there is an expandable column so I can easily expand columns from Dim_Date into the RandomNumbers query.

I have a Dataflow Gen2 which, when it runs, does a Replace with Dynamic schema of the Dim_Date table.

After running the Dataflow Gen2, these are my observations:

The relationship still appears in the model pane of the SQL Analytics Endpoint ✔️
The relationship still works in the Default Direct Lake semantic model ✔️
The constraints are not found anymore when I query the [INFORMATION_SCHEMA].[TABLE_CONSTRAINTS] view ❌
The expandable column is not found anymore in Power Query in Power BI Desktop. An import mode semantic model I built which uses this expandable column, get failed refreshes because the column can't be found anymore. ❌

My questions:

- Is this the expected behavior?

- Shouldn't the table constraints and the model relationships in the SQL Analytics Endpoint be automatically synchronized?

Thank you! 😀

Re: SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2

AndyDDC — Sun, 21 Apr 2024 13:02:25 GMT

Just a quick one before I dive a little deeper, this is a Warehouse (and not a Lakehouse)?

Re: SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2

frithjof_v — Sun, 21 Apr 2024 13:06:10 GMT

It's a Lakehouse

Re: SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2

frithjof_v — Sun, 21 Apr 2024 13:24:03 GMT

I am suspecting that the reason is that the Dataflow Gen2 drops and recreates the Lakehouse table when using Replace mode with Dynamic schema.

So maybe that's why the constraints are removed (or rather: not re-applied) from the tables in the SQL Analytics Endpoint?

However, I think it's not consistent with the model relationships still being present and working afterwards.

These are the destination settings in my Dataflow Gen2:

Re: SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2

frithjof_v — Sun, 21 Apr 2024 14:20:06 GMT

Oh, I see... My bad. It's not regular Replace I am using. I am actually using Replace with Dynamic schema. I had forgotten that. (I now changed the info to Replace with Dynamic schema also in my previous comments)

I guess the Dynamic schema setting is causing the table to get dropped and recreated.

I still find it inconsistent that the constraints and model relationships are affected differently by this operation.

However, for my use, the model relationships are more important than the table constraints. So I like that the model relationships are not being dropped at least.

I just need to beware not to use the expand column option in Power Query for my import mode semantic models, when working with lakehouse tables which are subject to Dynamic schema.
As the expand column option seems to be depending on the table constraints in the SQL Analytics Endpoint, and the table constraints don't seem to get recreated when the table gets dropped and recreated.

I would prefer that the table constraints also keep working after running a Dataflow Gen2 with Dynamic schema, similar like the model relationships.

Re: SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2

Anonymous — Mon, 22 Apr 2024 18:04:49 GMT

Hi frithjof_v
Apologies for the issue you have been facing. Thanks for the feedback. I have forwarded the feedback to the internal team.
Hope this helps. Please let me know if you have any further queries. Glad to help.