DataFlow gen 2 not updating the data in my Warehouse
Hey Community, I have this automated pipeline: The DataFlow performs some transformations and dumps data into the Warehouse. And from the warehouse, I create my report. The pipeline ran successfully, however, the new columns I added in my dataflow are not reflected in my warehouse. Why is this happenning? And how can I make sure the data reflects? Cause I'm getting a lot of data and want it to see the warehouse and dashboard updated. Thanks !
SQL Analytics Endpoint - Table constraints disappear after running Dataflow Gen2
Hi, I created a Lakehouse with 2 tables: Fact table: RandomNumbers Dimensions table: Dim_Date I created a relationship between the tables in the SQL Analytics Endpoint When I query the [INFORMATION_SCHEMA].[TABLE_CONSTRAINTS] view, I can see that these relationships have been registered as constraints on the tables in the SQL Analytics Endpoint. When I connect to the RandomNumbers table of the SQL Analytics Endpoint from Power Query in Power BI Desktop, I can see that there is an expandable column so I can easily expand columns from Dim_Date into the RandomNumbers query. I have a Dataflow Gen2 which, when it runs, does a Replace with Dynamic schema of the Dim_Date table. After running the Dataflow Gen2, these are my observations: The relationship still appears in the model pane of the SQL Analytics Endpoint ✔️ The relationship still works in the Default Direct Lake semantic model ✔️ The constraints are not found anymore when I query the [INFORMATION_SCHEMA].[TABLE_CONSTRAINTS] view ❌ The expandable column is not found anymore in Power Query in Power BI Desktop. An import mode semantic model I built which uses this expandable column, get failed refreshes because the column can't be found anymore. ❌ My questions: - Is this the expected behavior? - Shouldn't the table constraints and the model relationships in the SQL Analytics Endpoint be automatically synchronized? Thank you! 😀
Need Help with Formatting DateTime in Azure Synapse Data Flow Expression Language
Hi All, I'm working with Azure Synapse Analytics and I'm facing an issue with the Data Flow expression builder. My goal is to format a column named "Start Date (BST)" which currently has the format yyyy-MM-ddTHH:mm:ss.SSSSSS to dd/MM/yyyy HH:mm:ss, effectively removing the milliseconds. Here's what the date looks like right now: 2022-07-07T23:43:13.722007 I tried using the following expression in the data flow expression builder: toTimestamp('Start Date (BST)', 'dd/MM/yyyy HH:mm:ss:SS') However, I am getting an error stating "Unable to parse the expression. Please make sure it is valid." Could someone please guide me on the correct syntax or the steps I should take to format the datetime values properly using the expression builder in Azure Synapse? I would appreciate any insights or examples of the correct expression to achieve this. Thank you in advance!
