Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Large dataset refresh error on powerbi service

Im currently subscribing to the pro license and I have a large databricks hive table that consists of 900,000,000 rows. When i tried to refresh this table on powerbi service to get the newest data, i...
Need Help
lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
3 years ago

The issue here doesn't seem to be the dataset size. Rather it takes too long to fetch the data from the data source.

 

Incremental refresh will help if you can identify a suitable datetime column for the partition management. You will have to experiment with the partition sizes to avoid running into the same token timeouts.

 

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Hi Ibendlin, thank you for the reply. However, I couldnt even get the data to be imported into powerbi desktop to configure the incremental refresh because of the size. 

Is it possible that I only use a subset of data (let's say 3 months) to configure the initial incremental refresh. But later on, I need a way to retrieve and refresh the whole dataset because my report needed it. 

  • lbendlin's avatar
    lbendlin
    Icon for Super User rankSuper User
    3 years ago

    yes, you can limit the developer data as you like by providing useful values for RangeStart and RangeEnd.  These values are just for you, they will be ignored by the Power BI service.

    • aj1973's avatar
      aj1973
      Icon for Community Champion rankCommunity Champion
      3 years ago

      I think Anonymous needs to be aware that a dataset cannot exceed 1G of size under a Pro license and in a shared capacity WS.