topic Re: QUESTION::DATAFLOW GEN 2::Publish vs Save in Dataflow

QUESTION::DATAFLOW GEN 2::Publish vs Save

Element115 — Thu, 21 Mar 2024 16:55:08 GMT

Does anyone know what is the difference between Publish and close a DFg2 versus just closing the DFg2 and running it later from a pipeline?

In other words, if you close out from a DFg2 without publishing, and run it later (say, 1 day later) from a pipeline, does this mean that this DFg2 will automatically go through its first cylce of publish-->refresh, or not?

And if not, does this mean that when the pipeline runs the DFg2, the data will be stale, ie yesterday's data?

Re: QUESTION::DATAFLOW GEN 2::Publish vs Save

Anonymous — Thu, 21 Mar 2024 17:09:35 GMT

Hi Element115
Thanks for using Fabric Community.

Publish vs. Close in Dataflow Gen2:

Publish: When you publish a Dataflow Gen2, two things happen:
1. Validation: The dataflow definition undergoes a validation process to ensure it's error-free and can run successfully. This includes checking for missing connections, invalid transformations, etc.
2. Preparation for Refresh: The published dataflow is prepared for the refresh process. This involves generating an orchestration plan and allocating resources for data movement and transformation.
Close: Closing a Dataflow Gen2 simply saves your current work as a draft and closes the editing window. No validation or refresh preparation occurs.

Running a Closed Dataflow Later:

Yes, a closed Dataflow Gen2 can be run later from a pipeline. Pipeline does not publish the dataflow for you. So the run will use the last published version.

Hope this helps. Please let me know if you have any further questions.

Re: QUESTION::DATAFLOW GEN 2::Publish vs Save

Element115 — Thu, 21 Mar 2024 17:24:20 GMT

Thanks for the clarification. So just to confirm I understand the Running a Closed Dataflow Later properly:

Let's consider the following example:

0__create DF for the first time (1 M query only).

1__publish --> refresh.

2__open DF and change the M query.

3__close without publishing.

4__pipeline runs DF.

When step 4 occurs, the M query that gets executed is the one from step 1, not the one with the new changes from step 2. Is this correct?

Re: QUESTION::DATAFLOW GEN 2::Publish vs Save

Anonymous — Thu, 21 Mar 2024 17:33:29 GMT

Hi Element115
Yes you are right.

Re: QUESTION::DATAFLOW GEN 2::Publish vs Save

Anonymous — Thu, 21 Mar 2024 20:09:02 GMT

Hi Element115
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.

Re: QUESTION::DATAFLOW GEN 2::Publish vs Save

DaveGodri — Mon, 22 Jul 2024 12:09:18 GMT

There should be an option to publish without refresh.

One common issue is if you need to modify a setting that may have been missed such as setting a destination table to replace instead of append - now you have to re-publish the whole dataflow which triggers a refresh, whereas you just wanted to change that one setting and have it take affect on the next scheduled refresh. The current setup can you force you to consume CU's quickly especially for big dataflows. Please update this feature to allow publishing without refreshing.

Re: QUESTION::DATAFLOW GEN 2::Publish vs Save

frithjof_v — Tue, 23 Jul 2024 19:20:46 GMT

DaveGodri Please consider voting for this idea: https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=d7c2eb73-26c9-ee11-92bd-000d3adfe185