Fix sticky values in Dataflow Data Destination UI
When I modify the data desination table name in a Git repository and pull the changes into Fabric, running the dataflow from Fabric results in the old table name being used. It's only possible to change the desination table name through the Fabric UI, not the Git code.
5 Comments
- frithjof_v
Community Champion
Did you try opening the dataflow and click Save after syncing it from Git? Please vote for this Idea: https://community.fabric.microsoft.com/t5/Fabric-Ideas/Automatically-validate-Dataflow-Gen2-after-deployment/idi-p/4868304 - surpriseturtleFrequent Visitorfrithjof_v, I'll give it a try next time. Interestingly, I've never had an issue updating the part of the data destination code that uses a workspace ID and lakehouse ID. I update these lines to refer to library variables using Variable.ValueOrDefault, and have confirmed they are working correctly in different workspaces without having to publish the dataflow after the sync.
- miguel
Community Admin
Status changed:NewtoNeed ClarificationWould you mind sharing how you're changing the table name through Git? What specific part of the mashup.pq document are you modifying?
Sharing any repro steps will help us get to the bottom of it.
- surpriseturtleFrequent VisitorFor example, like this: shared DimTABCRoster_DataDestination = let Pattern = Lakehouse.Contents([CreateNavigationProperties = false, EnableFolding = false]), Navigation_1 = Pattern{[workspaceId = workspace_id]}[Data], Navigation_2 = Navigation_1{[lakehouseId = renamed_lakehouse_id]}[Data], TableNavigation = Navigation_2{[Id = "DimTABCRoster", ItemKind = "Table"]}?[Data]? in TableNavigation; I was editing the Table Navigation step in the Git repository, where it filters to the table as "DimTABCRoster". Now that I take a closer look at it, it seems this step is referencing an existing table rather than creating a table. I would personally like for this step to create a new table if none exists with that name. It is more similar to the UI experience where a new table is created the first time you run the dataflow with a newly named destination.
- miguel
Community Admin
Status changed:Need ClarificationtoAbandonedIt is possible to change the table name in Git. However, we woudl recommend making the changes through the new Advanced Editor for the data destination query so you don't have to manipulate the whoel document manually or require Git:
Advanced edit for data destination queries in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
In the TableNavigation step we actually use the question marks (?) as a way to throw a null if the table isn't found. Depending on how the rest of your M Script is set, it may actually create a new table or not.
The article below goes a bit more in-depth about the Settings that the data destination query could have and how such settings are set as an attribute record to the query that originates the data:
Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
For more questions about the behavior of data destinations, we recommend posting the questions in the Data Factory forum.
Recent ideas
Add Automated Pre-Upgrade Compatibility Assessment for Fabric Spark Runtimes
Microsoft Fabric should provide an automated compatibility assessment before a workspace, Environment, notebook, or Spark Job Definition is upgraded to a newer Fabric Runtime. Spark runtime upgra...mourakshit57 minutes agoNew MemberNew2Views0likes0CommentsSupport Creator-Independent Managed Identity and Ownership Transfer for Fabric Operations Agents
Microsoft Fabric Operations Agents should support an enterprise identity and ownership model that is independent of the individual user who originally created the agent. Today, an Operations Agent r...mourakshit59 minutes agoNew MemberNew2Views0likes0CommentsAdd Centralized Health, Freshness Monitoring, and Alerts for OneLake Shortcuts
Microsoft Fabric should provide a centralized health-monitoring experience for OneLake Shortcuts across workspaces and data sources. As organizations adopt OneLake Shortcuts at enterprise scale, hun...mourakshit1 hour agoNew MemberNew3Views0likes0CommentsAdd Schema Compatibility Policies and Breaking-Change Detection to Fabric Event Schema Registry
Fabric Event Schema Registry should support configurable schema compatibility policies so teams can safely evolve event contracts without breaking downstream Eventstreams, Eventhouse ingestion, appli...mourakshit1 hour agoNew MemberNew2Views0likes0CommentsAdd Dry-Run Simulation and Historical Event Replay for Fabric Activator Rules
Fabric Activator should provide a dry-run simulation mode that allows developers to test rules against historical or sample event data without executing real downstream actions. Activator rules can ...mourakshit1 hour agoNew MemberNew2Views0likes0Comments