Forum Discussion
Dataflow vs Dataset refresh
Haven't found any documentation on how data refresh works with respect to a dataflow and then a dataset sourced from that dataflow.
So looking for feedback, based on what I've discovered:
- Both a dataflow and dataset need data to be refreshed
- So I assume the dataflow is much like a data storage component on it's own that manages the updating from the data source, wherever that may be
- And the dataset will refresh data from the dataflow 'storage'
- Thus a logical refresh sequence (such as setting a scheduled refresh) would see the dataflow update first then the dataset aftewards (maybe 30 mins later as I suspect doing both at the same time may not yield the right results)
I've come to this conclusion after seeing the behaviour of having one or the other set for scheduled refresh.
I'm also seeing inconsistency in the workspace contents view where it shows last and next refresh times.
For this dataflow, I've toggled off the scheduled refresh but it still shows a Next Refresh time (I would expect not to see any time stamp):
For these datasets, they have both had a refresh more recently than indicated here
Here's the first one:
And the second:
A bug?
>So I assume the dataflow is much like a data storage component on its own that manages the updating from the data source, wherever that may be
That's correct, technically PBI's Dataflow uses Azure Data Lake Gen2 for storage.
One use case I plan to use this dual refresh structure for, is to handle sources (e.g. static files) that don't need to be refreshed in Dataflows where they'll be imported but not under scheduled refresh. I've found PBI's scheduled refreshes to fail easily, so cutting down the service's scheduled refreshes to sources that actually need to be refreshed should lower incidents (e.g. web API timeouts, credential issues etc.).
Hi,
After internal checking, it seems that the next refresh time update (without browser refresh) was fixed and is should be available in the following updates.
I will keep monitor it.
Thanks,
Assaf
28 Replies
Hi ,
I dont understand... why do we have to Schedule the Dataset? From all the Microsoft articles that I read about Dataflow, it was made to look like dataset will be automatically connect to Dataflow and show users the latest data. Does this mean if I create 10 datasets using my Enterprise Dataflow, I need to configure all my 10 Datasets to refresh sepeartely? On top, we have to rely on API to get faster updates? This is totally unproductive and totally beats the purpose of having all my data on Cloud services already!
I hope this is all not true and I dont have to keep having separet schedules Dataflow and Datasets!Anonymous dataflows are not pushing data directly to datasets, they're just making the data available in a datalake. You might even have a dataflow that does not have a destination dataset. You do have to schedule refreshes in your datasets separately from your dataflows, just like you have to schedule refreshes against any other data source.
otraversThanks for the response. I still feel this defeats the purose of doing the ETL on PowerBI's own storage where my Datasets also reside. The purpose of Dataflow was to have a Centralized data source that all reports can consume. Now its basically telling me to take casre of indivudal report refreshes, inspite of having everything in a "centralized" place?! Ideally Dataset should work like a "Direct" Querying to Dataflow, not an Import
>So I assume the dataflow is much like a data storage component on its own that manages the updating from the data source, wherever that may be
That's correct, technically PBI's Dataflow uses Azure Data Lake Gen2 for storage.
One use case I plan to use this dual refresh structure for, is to handle sources (e.g. static files) that don't need to be refreshed in Dataflows where they'll be imported but not under scheduled refresh. I've found PBI's scheduled refreshes to fail easily, so cutting down the service's scheduled refreshes to sources that actually need to be refreshed should lower incidents (e.g. web API timeouts, credential issues etc.).
I do love dataflows and what they allow us to do (preparing entities only once and using them everywhere is a dream come true).
However, that's really disappointing... I will now suffer from more delay: the dataflow will have to refresh first (let's say at 9:00), and half an hour later the dataset (9:30). I wish datasets would automatically refresh when the dataflow does...
Yes I don't understand this either. Hopefully someone can elaborate. Even if my report only connects to a single dataflow, it still creates a "dataset" when I publish it and thus I have to manage two refresh schedules (one for dataflow and one for dataset). I can see some use cases but I would think most of the time the preferred functionality is for a dataflow refresh to automatically kick off the associated dataset refresh.
Hi!
Regarding the inconsistency in the next refresh time of the dataflow, after a browser refresh, do you still see the next refresh time?
Thanks,
Assaf
Assaf wrote:Regarding the inconsistency in the next refresh time of the dataflow, after a browser refresh, do you still see the next refresh time?
I do see the refresh time after a browser refresh. Also with the page remaining on screen, the refresh was scheduled to run and the date/time stamps updated without me having to refresh.
The issue I was experiencing included navigating away from that pgae, then returning to the page (so not refreshing the browser), so I'd have thought that would result in the updated time stamps too...I'll keep an eye on it.
Hi,
After internal checking, it seems that the next refresh time update (without browser refresh) was fixed and is should be available in the following updates.
I will keep monitor it.
Thanks,
Assaf
