'The dataflow has never been published.' error when using 'Save as Dataflow Gen2 (CI/CD)' feature

[b_vanderlaan]

The 'Save as Dataflow Gen2 (CI/CD)' from regular Dataflow Gen2 feature has been recently released in Fabric. However, when I try to refresh the newly created dataflow, I get an error that the dataflow has never been published. I understand that these dataflows are automatically published when being saved, but when I open the dataflow and press 'Save & run' nothing happens.

Hopefully you can support me with this. I'm looking forward to converting our dataflows to the CI/CD version.

[change007]

Following, similar error occuring and no idea why

[Mauro89]

Hey there, 

I had the same issue and just "figure out" how I can successfully run a refresh. In my case I had the "Enable staging" option for the dataflow tables disabled. After enabling this again the DF was recognized as "published" an the refresh worked.
Looks like this feature is currently a necessary always-on feature 😉

I hope this helped, if so feel free to mark it as solution.

BR

[belkov78]

The error kind of has gone away from me after connecting the workspace to DevOps.

I just wait for the popup to show up where it wants to commit, indicating the dataflow was saved succesfully.

After this I can either commit, wait for it to finish and then run the dataflow or refresh it manually.

But from the couple of times I ran into the issue, when I enable an incremental refresh from the get-go, it almost always fails to publish. So I just save it, commit it, reopen it and add the incremental refresh.

[v-csrikanth · Accepted solution]

Hi @b_vanderlaan 
Can you please raise a Microsoft support ticket. You can create a Microsoft support ticket using the link below.
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

If this helps, then please Accept it as a solution and dropping a "Kudos" so other members can find it more easily.

Hope this works for you!

Thanks.

[b_vanderlaan]

I have opened a support ticket and I was helped today. We were not able to reproduce the issue this time. It seems that the issue I was facing has been fixed in the meantime. 

[eranbenayun]

Hi @b_vanderlaan ,

The Save As feature was recently released as Public Preview, with some known limitations listed in Migrate to Dataflow Gen2 (CI/CD) using Save As (Preview) - Microsoft Fabric | Microsoft Learn. 

Will you be able to open a support case, so that we can further assist in analyzing and resolving the issue? 

[b_vanderlaan]

Thank you for reaching out. I think the issue I'm having is not part of the know limitations. I would be happy to open a support ticket, but I don't know how. Can you guide me how to do this?

[BryanRock]

@b_vanderlaan , I'm having the same issue and I think I have a workaround. Instead, of saving a copy, create a new Gen2 Dataflow with CI/CD. Then copy the queries from the old dataflow and paste them into the new one. For some reason, save & run works in this scenario. It's just less efficient because you have to reauthenticate, redo the destination steps, and, in my case, reconfigure incremental refresh.

[b_vanderlaan]

Thank you for thinking along. I agree that your approach will work, but this would be inefficient for the reasons you already mention. I have to recreate a lot of dataflows. The 'Save as Dataflow Gen2 (CI/CD)' functionality can prevent these additional steps, however, it does not work as expected. 

[belkov78]

Did you enable any incremental refresh options by any chance? I just ran into the same issue and disabling it made it work again.

Currently running the whole thing using incremental refresh on and it failed again. Disabling "require query to fully fold" worked again even though the whole thing is folding.

Mildly frustrating.

[v-csrikanth]

Hi @b_vanderlaan 
Yes this feature has been enabled recently into the Fabric.

After creating or modifying your dataflow, ensure you explicitly select the 'Save' or 'Save & run' options. This action should publish the dataflow. However, if 'Save & run' appears unresponsive, try the following steps:​

• Open the dataflow ->Make a minor edit (e.g., add and then remove a space in a query) ->Click 'Save' first, then attempt 'Save & run' again.

This approach can sometimes trigger the necessary publishing process.

It's important to note that Dataflow Gen2 with CI/CD and Git support is still in preview and has some limitations. For instance, orchestrating a refresh of such dataflows isn't currently possible in Fabric data pipelines. Additionally, the workspace view doesn't show if a refresh is ongoing for the dataflow. Being aware of these limitations can help set appropriate expectations and guide troubleshooting efforts.

Reference link: Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn

If the above information helps you, please give us a Kudos and marked the Accept as a solution.
Best Regards,
Community Support Team _ C Srikanth.

[b_vanderlaan]

Thank you for your response. Yesterday I tried both options ('Save' and 'Save & run') but both did not work. I just attempted your work around:
1. I added a blank query to the dataflow

2. Pressed 'Save', this closed the dataflow

3. Reopened the dataflow and pressed 'Save & run'

4. I received a notification: 'Refresh started', however, after checking 'Recent runs' on the artifact I did not see any changes.
5. I pressed 'Refesh now' on the artifact, I received again the notification 'Refresh started'. This time after checking 'Recent runs' I saw a new run but it indicated the 'DataflowNeverPublishedError' I saw before.

Unfortunately your workaround did not work. I hope you have other suggestions, or can help me troubleshoot.

PS: orchestrating a refresh of such dataflows in Fabric data pipelines has been released this week, I tested it in my workspace and it worked.