Forum Discussion
Dataflow size limit
I'm considering breaking one of my larger models into a series of Dataflows to improve the overall refresh performance of the model. I was wondering whether an individual dataflow had a size limit? Also if several dataflows are combined into a single model, will refreshing the model force a refresh of the associated dataflows or will they refresh independently based on their individual refresh frequencies?
11 Replies
- Hi there
From what I understand there is a size limit in terms of storage depending on if you have Power BI Pro (10GB) or Power BI Premium (100TB)
Then if you had to break them up, they would all sit in their separate data stores in dataflows.
As it currently stands you would then manage the refresh of the dataflows to ensure that the data is up to date. If you are using Power BI Premium you could use Linked Entities or Computed Entities (Depending on requirements).
Then you would need to import the dataflows into your Power BI Desktop file (PBIX) and then upload that to the Power BI Service.
Once in the Power BI Service, you would then need to schedule the refresh for your PBIX file.- No. The schedules are separate. You can even have multiple dataflows with separate schedules feed the same report which can give you an extremely crude approximation of incremental refresh with out premium capacity.
The biggest limit I've run into is refresh time out (2 hours). I'm running a lot of Native Queries against AAS. These run slow as it is, but my feeling is that dataflows are slower than a regular service refresh. As a result, I've broken up a big model into entities in 5-6 different dataflows in the same workspace. Then I just staggered the refresh of each by two hours over the course of the night. Now everything runs fine.
- Hi there
Yes extracting data out of AAS could potentially be very slow if you are using it as a data extraction tool with row by row data.
That will work with the staggering of dataflows refreshing.
