Forum Discussion
Dataflow vs Dataset refresh
>So I assume the dataflow is much like a data storage component on its own that manages the updating from the data source, wherever that may be
That's correct, technically PBI's Dataflow uses Azure Data Lake Gen2 for storage.
One use case I plan to use this dual refresh structure for, is to handle sources (e.g. static files) that don't need to be refreshed in Dataflows where they'll be imported but not under scheduled refresh. I've found PBI's scheduled refreshes to fail easily, so cutting down the service's scheduled refreshes to sources that actually need to be refreshed should lower incidents (e.g. web API timeouts, credential issues etc.).
Hi,
After internal checking, it seems that the next refresh time update (without browser refresh) was fixed and is should be available in the following updates.
I will keep monitor it.
Thanks,
Assaf
Yes I don't understand this either. Hopefully someone can elaborate. Even if my report only connects to a single dataflow, it still creates a "dataset" when I publish it and thus I have to manage two refresh schedules (one for dataflow and one for dataset). I can see some use cases but I would think most of the time the preferred functionality is for a dataflow refresh to automatically kick off the associated dataset refresh.
The connection between Dataflow and Dataset refreshes could be automated using the API and MS Flow. Something like this:
1. Trigger your dataflow refresh via the API, for instance using PowerShell, which you could host/run from Azure Functions
- https://docs.microsoft.com/en-us/rest/api/power-bi/dataflows/refreshdataflow
- https://insightsquest.com/2019/03/03/refresh-power-bi-datasets-with-powershell/
- https://eriksvensen.wordpress.com/2017/08/16/updating-a-powerbi-dataset-using-microsoft-flow-azure-functions-and-a-flic-button/
Incidentally this approach opens you up more fine-grained control of your refreshes than using the static scheduling options in the Power BI service. For instance you could trigger your dataflow refreshes based on sources being updated, or integrate your Power BI refreshes in broader Azure Data Factory processes:
- https://blog.crossjoin.co.uk/2018/10/21/calling-the-power-bi-rest-api-from-microsoft-flow-part-2-refreshing-a-dataset-when-a-data-source-changes/
- https://altis.com.au/greater-control-of-your-power-bi-datasets-with-the-power-bi-rest-api/
2. Use NotifyOption in the API call above with MailOnCompletion sending an email to an inbox monitored by a Flow workflow. This email is your Flow trigger.
3. Trigger your "client" dataset refresh via the Power BI REST API as the follow-up action in the same Flow, as explained here:
- https://medium.com/@Konstantinos_Ioannou/refresh-powerbi-dataset-with-microsoft-flow-73836c727c33
- https://docs.microsoft.com/en-us/rest/api/power-bi/datasets/refreshdataset
4. Push a notification (email, RSS, Teams etc.) once the thing is complete, based on a similar flow triggered by the dataset refresh's success (or failure). A side benefit of handling your own notifications is that you get to choose the format, content, notification channel, and recipient(s), whereas the Power BI service hard-codes all of these.
Hi,
NotifyOption in Power Automate custom connector for either dataflow or dataset refresh is not working for me.
Anyone recived mail on completion?
Hi otravers
I have created custom connector in Power Automate to refresh dataflow but i am not getting an email on refresh completion. I have followed the links you have provided. Do you have any idea why i am not recieving the email?Hey
Make sure you set the "NotifyOption" parameter to "MailOnCompletion". You will receive email both on failure and on success of teh refresh
Refer to this Doc: https://docs.microsoft.com/en-us/rest/api/power-bi/dataflows/refreshdataflow#notifyoption
I ended up writing a blog post elaborating on my initial entry to tie everything together in Power Automate. However, even though I'm using the NotifyOption, I'm not getting the emails on success/failure dataflow refreshes - not even from the documentation Try It button. I'll update once I figure out why but this is puzzling as I followed the documented syntax to the letter.
https://www.oliviertravers.com/power-automate-powerbi-dataset-dataflow-refreshes/
Anonymous can you share screenshots of how you set it up?
