Deactivating queries in Dataflow Gen2

[fbcideas_migusr]

In notebook you can freeze a cell, in pipeline you can deactivate an activity. Similar feature would be a great addition in Dataflow Gen2. I understand that I can copypaste queries between dataflows, but it can be cumbersome and would clutter the workspace. Sometimes just deactivating one of the queries for the publishment would do the trick.

[nishalit]

Hi! Could you please share more information about what it would mean to deactivate a query? today you can simply disable staging and the effect would be to not load the query at all. It would only be processed if required in the event that its part of an output destination or a subsequent query that does have staging enabled

[fbcideas_migusr]

Hi,

Apologies for delay. You are right, deactivating staging inactivates the query (it is shown in italics and greyed out). That was useful!

One thing I would consider here is that Staging in documentation is created to improve performance and manage large volumes of data. It is not explicitly stated anywhere that disabling Staging will stop the query from running.

Additionally, it seems that I needed to enable Staging and later disable it to stop the query from running (?). Here I'm not sure since it was strictly trial and error, but it would be beneficial if it was somehow clearly explained.

[Joe_Guidera]

Consider the following

Two of three query within the dataflow are valid and can be executed. The third is not valid (currently has one or more errors) and cannot be executed (perhaps due to a source API error that a vendor needs to resolve). That means you are unable to publish the dataflow, nor can it execute for the two valid queries since the third is not in a valid state.

Today I have only one option, delete the query and all of the transformation steps and then re-create after it is possible to do so. I would prefer the option to "disable" that query and its transformations which would allow me to successfully publish and execute the two valid queries within the flow. Then when corrections are possible I enable and edit (as required) the invalid query to make it valid again and publish/execute as normal.