Dataflow refresh not running

[coverturfLSH]

Hi,

I have a Dataflow Gen2 that had been running fine for many months now. Today I had an issue where the Dataflow was taking much longer than usual (+2 hours vs normal 3 minute run). I canceled the Pipeline that refreshes the dataflow via an activity and it seemingly canceled the dataflow.

However, since that initial cancellation I have been unable to refresh the dataflow. I can open, edit, publish the data flow, and click refresh. The refresh icon spins next to the last "Refreshed" date/time, but no new data is captured or updated. I also confirmed the dataflow is not running when I check the Monitoring hub and the dataflow does not appear there.

Has anyone else encountered this issue or have suggestions of what I can try? I have tried adding a new Power Query with data destination and republished but didn't have any effect. I'd rather not recreate the dataflow since I have 15 PQs in there that I would have to set up again. Other dataflows in my workspace are working normally.

Thank you,

C

[v-veshwara-msft]

Hi @coverturfLSH ,

Thanks for reaching out to Microsoft Fabric Community.

Thanks for confirming the interim solution using the Power Query Template and setting the default data destination, that’s a practical and effective approach when a Dataflow becomes unresponsive or fails to refresh as expected.

Also, thank you @miguel  for outlining some of the key improvements available in the Dataflow Gen2 CI/CD experience.

Here are some notable benefits of Dataflow Gen2 with CI/CD and platform improvements that you might still find useful:

1. Multitasking support: Open multiple Dataflow Gen2 items alongside other Fabric items, improving your overall authoring experience.

2. Workspace folder support: Create dataflows directly in structured workspace folders for better organization.

3. Deployment pipelines: Automate movement of dataflows between environments (e.g Dev to Test to Prod) using deployment pipelines.

4. Fabric settings and scheduler: Manage refresh schedules and Dataflow Gen2 settings directly within Fabric.

5. Public APIs (preview): Automate and manage Dataflow Gen2 operations including create, update, delete, scheduling, and monitoring.

The Public APIs for Dataflow Gen2 enable advanced scenarios even without Git integration, such as:

1. Automated deployment across environments

2. Monitoring and alerts for tracking failures or performance issues

3. Integrating data from multiple sources into unified workflows

4. Custom error handling and retry mechanisms.

Here are some links which you might find useful:
Dataflow Gen2 CI/CD, GIT integration and Public APIs (Generally Available) | Microsoft Fabric Blog |...

Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn

Hope this helps. Please reach out for further assistance.

If you’ve already raised a support ticket, please consider sharing any insights or resolution steps provided by the support team here, so others in the community facing similar issues can benefit.

Please consider marking the helpful reply as Accepted Solution and giving kudos to assist others with similar issues.

Thnak you.

[miguel]

Would it be possible for you to raise a support ticket so we can look deeper into this and make sure that we can understand why it happened and make sure that it never happens again? You can use the link below to raise the support ticket:

http://support.fabric.microsoft.com/support

Using the Power Query template is possible, but you can also use the "Save as" feature as described in the article from the link below:

Migrate to Dataflow Gen2 (CI/CD) using Save As (Preview) - Microsoft Fabric | Microsoft Learn

[coverturfLSH]

Thanks, I will submit a ticket.

Generally, I like to follow the rule "if it's not broke, don't fix it". What would be the benefit of the DFG2 CI/CD? My team does not use GIT or Devops to maintain code nor do I see us starting any time soon.

[v-veshwara-msft]

Hi @coverturfLSH ,
As we haven't heard from you in a while, just following up to check if you had raised a support ticket and got any insights from the support team. If yes,could you share any key insights or the resolution here for the benefit of the wider community.

Additionally if any of the responses you found were helpful, please consider marking as Accepted solution to help others find easily. If there are no further updates, we may consider closing this thread.

Should you need further assistance in the future, please feel free to create a new post in the Microsoft Fabric Community Forum. We’ll be glad to support you.

Thank you for being a part of the Microsoft Fabric Community.

[coverturfLSH]

Hi Veshwara, I never got around to submitting the ticket since I got bogged down with other things. My interim solution became my permanent solution and thankfully that was easy to implement. 

[miguel]

Most of the benefits come from how tightly integrated this new version of Dataflows is to the rest of the Fabric ecosystem. No pressure to change everything to the Dataflow Gen2 with CI/CD, but definitely give it a try. 

Some examples of things that are pretty noticeable and are not focused on the developer experience:

- support for multitasking. You can finally have multiple Dataflows open and go between items that you have open

- Create a dataflow directly to a subfolder 

- Better experience when working with a draft of your dataflow and how you could discard changes without ever committing 

- New save button in case you only want to save and not run your Dataflow

These are just a handful of the benefits, but there are more waiting for you in the new Gen2 with CI/CD experience and more in the roadmap as well.

[coverturfLSH · Accepted solution]

Interim (final?) solution was to export the template (PQT) for the old not working, import the template in a new DFG2, and then set a default data destination. Made it fairly seemless.