Dataflow gen 2 publish is still a pretty painful process

[dbeavon3]

I was under the assumption that the "gen 2" version of dataflows would avoid some of the pain related to the "validation" and publishing.  However I'm seeing almost none of the improvements I hoped for .

First and foremost, I want to be able to avoid waiting on "validation".  For example, if I'm only making a minor change, like tweaking a parameter, I don't want to wait for 30 mins or an hour to lock in that change.  I just want to edit the PQ code of the DF without a rigorous review of my work.  I'm willing to "take the risk" myself, for making these changes behind the back of the PQ editor.  Yet there appears to be no approach for editing my PQ without waiting.  I can do no refreshes until the painful "publish" is completed.  

If PBI would give me direct access to my PQ code, that would resolve most of my complaints.  I do NOT need its validation or its data type detection.  Does anyone know a "back-door" way to edit minimal aspects of the PQ code so that it will take effect prior to the next normal refresh?  Even just being able to change parameters would be helpful.

The second issue is also related to the validation.  The issue is that validation can't be cancelled.  I'm not sure what I'm missing but the "gen 2" dataflows are often in a state of purgatory where they can't be edited or cancelled, and the refresh animation will spin indefinitely.  It feels like this must be a bug or design flaw.  It feels like this feature should never have been released out of "preview".

Last of all, I am having the same troubles with "Computed/Calculated" entities as I did in the first iteration of dataflows. The Power BI service is totally incapable of detecting dependencies of one table on another until it finishes the validation phase.  While it is doing that validation it will abuse the back-end resource (normally a WEB connector) by running the same data-retrieval request over and over and over.  The image (below) shows a dataflow with multiple tables.  These distinct tables will ultimately be available to clients by way of a single file stored in ADLS.  Yet as long as the "publish" operation is underway, I must wait for the WEB connector to be contacted numerous times.  It is pretty insane. 

The end result of all of this is that these back-end WEB api's must be adjusted to compensate for the crazy requirements of PQ.  The api's must accommodate a massive amount of redundant requests, sometimes in parallel.  The supposed benefits of using PQ for "low-code" ETL's are lost, when the team that manages the WEB api's must do a TON of additional work to accommodate the needs of PQ.

[Anonymous]

Hi @dbeavon3 ,

With Dataflow Gen2, we changed how saving a dataflow works. 

This powerful feature allows you to make changes to your dataflow without immediately publishing them to your workspace. Instead, all your changes are automatically saved as a draft, which you can review a later time, and then publish when you're ready. With this feature, you don't have to worry about losing your work if you want to resume it at a later time, if your dataflow fails validation, or if your editing session abruptly ends.

More details: How to save and publish your dataflow - Microsoft Fabric | Microsoft Learn

If it still does not help, please provide more details.

Best Regards
Community Support Team _ Rongtie

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[dbeavon3]

Hi @Anonymous 

Yes, the unpublished draft is useful.

What would be more useful is not having to wait an hour on a publish.   We need some direct access to the df-pq that would allow us to commit/publish the draft without waiting.  There is no point to waiting an hour for a parameter change or whatever.  In most cases users are just moving artifacts from one environment to the other, and the "validation" and "data type detection" is totally pointless.

As I mentioned earlier, the most painful aspect of "validation" is the fact that the, during validation, the service seems unable to make use of the "loaded" data that saves data to azure for the sake of "computed/calculated" tables.  As such, it runs lots of unnecessary queries at the original source, instead of using intermediate data.

See the message I posted earlier about troubles with "Computed/Calculated" entities. The Power BI service is totally incapable of detecting dependencies of one table on another until it finishes the validation phase.  

[dbeavon3]

UPDATE!  I have to update this question with a partial answer.

>> Even just being able to change parameters would be helpful.

I assumed that editing parameters required us to have to wait on the "publish" operation, but that appears to be incorrect.

I just tested the use of the the "manage parameters" option in a gen2 dataflow.  It appears that you can change these parameters and they will instantly take effect without saving/publishing.   It is not like any U/I experience I've ever encountered, but I'm not complaining if it helps me avoid "publish", and saves me hundreds of hours of time each year!

Now all I need is an equally magical way to make back-door changes in my PQ code itself without waiting on "publish".

[dbeavon3]

UPDATE on the UPDATE.  I was wrong.  Editing the parameters just looks like it may work, but it doesn't actually take effect.  There appear to be two different versions of a gen 2 dataflow....  the one you are editing, and the one that is published.  The one you are editing may or may not have been published yet.  Until it is published, the changes won't "stick". 

Does anyone know if there is a way to at least determine if the  version of the dataflow we are editing is equivalent to the published version?