Forum Discussion

Element115's avatar
Element115
Memorable Member
2 years ago
Solved

QUESTION::DATAFLOW GEN 2::Publish vs Save

Does anyone know what is the difference between Publish and close a DFg2 versus just closing the DFg2 and running it later from a pipeline?   In other words, if you close out from a DFg2 without pu...
Dataflows
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi Element115 
    Thanks for using Fabric Community.

    Publish vs. Close in Dataflow Gen2:

    • Publish: When you publish a Dataflow Gen2, two things happen:
      1. Validation: The dataflow definition undergoes a validation process to ensure it's error-free and can run successfully. This includes checking for missing connections, invalid transformations, etc.
      2. Preparation for Refresh: The published dataflow is prepared for the refresh process. This involves generating an orchestration plan and allocating resources for data movement and transformation.
    • Close: Closing a Dataflow Gen2 simply saves your current work as a draft and closes the editing window. No validation or refresh preparation occurs.

    Running a Closed Dataflow Later:

    • Yes, a closed Dataflow Gen2 can be run later from a pipeline.  Pipeline does not publish the dataflow for you. So the run will use the last published version.

    Hope this helps. Please let me know if you have any further questions.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Element115 
Yes you are right.

DaveGodri's avatar
DaveGodri
Regular Visitor
2 years ago

There should be an option to publish without refresh.

 

One common issue is if you need to modify a setting that may have been missed such as setting a destination table to replace instead of append - now you have to re-publish the whole dataflow which triggers a refresh, whereas you just wanted to change that one setting and have it take affect on the next scheduled refresh. The current setup can you force you to consume CU's quickly especially for big dataflows. Please update this feature to allow publishing without refreshing.