Is there any way of editing column mapping with advanced destination editor in Gen2 CICD dataflows?

[JoshT]

The below documentation refers to a preview feature where we can incorporate variable libraries into Gen2 dataflow destinations. It makes explicit the point that once the editor is used, the destinations GUI will no longer work:

https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-advanced-edit-data-destinations

Is there any means of updating the underlying column mapping once the editor has been used? The query does not appear to specify any columns, and adding columns into the destination does not appear to automatically map. I've tried switching the workspace and warehouse IDs to string literals (as they were before I edited), but the GUI still doesn't work. Has anyone thought about this workflow and the fact that output columns might change over time?

Only option I can see so far is to delete and recreate the query, which isn't ideal

[v-moharafi-msft]

Hi @JoshT ,

Thank you for raising this query, and sincere thanks to @LuitwielerMSFT  and @miguel  for the valuable insights provided. Their input helps clarify the current behavior and upcoming roadmap.

As mentioned, we recommend waiting for the next feature rollout shared by @miguel.

If you encounter any other issues, please feel free to reach out we’re happy to assist.

Thank you,
Abdul Rafi

[LuitwielerMSFT · Accepted solution]

Hi @JoshT !

Good question, and thanks for flagging this. I work on the Dataflows Gen2 product team.

You're right that once the advanced destination editor is used, the visual column mapping UI won't be available for that query anymore. The engine can't guarantee it can reconstruct the UI state from a customized M script, so this is an inherent limitation of the feature. And column mappings can't be edited through the advanced editor either, since that experience is focused on parameterizing the destination itself (e.g., swapping workspace or warehouse IDs via variable libraries).

So today, if your output columns change, deleting and recreating the destination is unfortunately the only path. We know that's not ideal, especially in CI/CD scenarios where schemas evolve over time. It's feedback the team is tracking.

On the PQT export question: as Miguel mentioned, support for including destination queries in .pqt exports has started rolling out and should be available for you this week. That should help with the source control side of things.

Hope that helps! Let us know if you run into anything else.

[JoshT]

thanks @LuitwielerMSFT, that helps clear up the question for me!

[JoshT]

It also appears that destination queries are not included in the .pqt export definitions. Is there any plan to get this included? It makes it harder for us to source control since the ability to export JSON definitions of CICD dataflows have been taken away, where our git repo is not DevOps or GitHub so we have to export definitions manually

[miguel]

This is actually something that we've started to roll out and you should be able to see it this week:

Nevertheless, you also have the option of using the Fabric REST API to get the full Dataflow definition and not just the mashup. Below is the REST API endpoint that you can see to download the full dataflow definition:

Items - Get Dataflow Definition - REST API (Dataflow) | Microsoft Learn

Hope this helps!

[JoshT]

thanks @miguel. might be interesting to hook up the API to source control as the process at the moment is fairly manual, with formatting and using VS code's PQ extension to extract query text

[v-moharafi-msft]

Hi @JoshT ,

Thank you for reaching out to the Microsoft Community.

This is a known limitation of the feature in its current preview state. Once Advanced Edit is used for a destination, the UI is no longer available, and all changes must be handled through M code. Reverting back to the UI isn’t supported, and recreating the destination is currently the only way to restore it.

Additionally, schema changes must be managed manually in the script, and destination queries are not included in  .pqt exports, which limits CI/CD scenarios outside of Git-integrated setups.

At this time, there are no announced updates addressing this behavior. You may also consider submitting a related request in the Fabric Ideas forum: Fabric Ideas - Microsoft Fabric Community, as community feedback helps prioritize future improvements.

Best Regards,

Abdul Rafi