Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
7 years ago

Dataflow +Dataset = NEED more Clarity!

Hi All, I have successfully built an Enterprise Dataflow that loads all my tables in PBI service. Its incremental, scheduled and all is good! Now I use PowerBI Desktop to consume my Dataflow: PS: I...
Dataflow
Need Help
Anonymous's avatar
Anonymous
Not applicable
7 years ago

No response?? :'(

I have a Dataflow Entity that has about 54Million record.s
I am trying to create a report using this Dataflow and everytime, Power BI Desktop trys to download 10s of GB of data from the Dataflow. This does not make any sense!

If I multiple this with the other Entities that have similar volume, I will never be able to create a report on my Desktop!

RBunting's avatar
RBunting
Icon for Advocate III rankAdvocate III
7 years ago

AnonymousI saw your reply in our thread about dataflow timeouts.

 

I don't think you should need to load your entire dataflow.  When I open PBI Desktop and click 'Get Data,' then pick 'Power BI Dataflows,' and go through the wizard to eventually select some specific dataflow that is deployed to the service, the next thing that happens is the Power Query Editor opens and creates a new query pointing to the selected dataflow as it's source.

 

You should be able to apply whatever filters and limiters you want at this point before loading the data to your model.  You could curtail that 54 million rows by whatever degree you need. 

 

  • Anonymous's avatar
    Anonymous
    Not applicable
    7 years ago

    RBunting Thanks for the response!

    My requriemnt is to show historical data/trends and I need to have all the data from this Dataflow. Of course, there will be filters for Year/Month/Week and Region to ensure not everything is loaded at once.
    This is the reson I decided to go with a Dataflow to pre-load my data into Cloud.

    Even if I do apply a filter, the data is still downloaded from Dataflow , its still going to be GBs worth of data everytime I need to fetch new data for my development. Do you think I am doing something wrong here?

    • parishdapbi's avatar
      parishdapbi
      Icon for Advocate II rankAdvocate II
      7 years ago

      Ideally, if you have Power BI Premium, you could use Incremental Refresh, both in loading data to the dataflow and to incrementally load new data into the dataset. Otherwise, you have to load all of the the data all the time. Only way around without Premium would be to create a table with history data that is not refreshed, then another table with currrent data. In a DAX measure you could combine the values, such as: Sales = SalesHistory[Units] + SalesCurrent[Units}.

      • Anonymous's avatar
        Anonymous
        Not applicable
        7 years ago

        Thanks parishdapbi 
        We do have Premium capacity and I am using Incremental Refresh on the Dataflow.

        I liked the idea of having Incremental refresh on the Dataset, however I always thought Dataset will have a live connection to Dataflows. This is really disappointing. 
        And I hope MS does something to only enable a preview Data during development using Dataflow and not force use to download complete data onto desktop.