Forum Discussion

tomas12344's avatar
tomas12344
Helper III
3 years ago
Solved

Optimizing refresh of data

HI All

Have a customer, wich has data stored in SAP, and these data are loaded to PBI (using SAP BW connector) to PBI for some reports. The query in SAP is not very optimal, and probably the client dont want/will not change it, because it is still used for other reports in SAP. The refresh is quite long (2hours--5+hours), depending of the number of fields, hierarchies,... which is downloaded. The data could not be downloaded at onece, so the data is downloaded partialy by months, and downloaded is always the whole history of the data (or limited by the filter, but always all the history).

 

The question:

 

I'm looking for some solution, to not always download the full history, but only some part, therefore, I would like to ask you for any advice, tips,..., any other solutions :

1. Incremental download is currently not posible, because of back-valuation (accounting data). But is in PBI a way to delete/update some old data in already downloaded dataset? = When there will be some master data in PBI, and I could identify the changes which are back-valuated, then I need only to download the latest data, and delete and replace the changed old data.

 

2. In power query editor, eache table could be makred as "Include in report refresh" - what means, that this table is refreshed, when the dataset is refreshed. But found contradictory statements on some communities, if this works also in Service, and how. Have tried this, but not sure, if it works how I would need. = Have created dataset A, B and C which is append of C, added column with current time to see if it is refreshed. When table A was excluded from refresh, the time was from the initial load, that is OK, but in  table C (append) the time always refreshed when this C tables was loaded = for me it looks like, that table A and B was also refreshed. Any experience how this works, and if this could be used, that table A will contain some old data, table B will contain lates data, and table C will contian all data and will be source for the reports?

 

3. Is there a way how to have some datasets, which will contain historical data and actual data, these will be stored in Service and refreshed as needed (that mean only the dataset which contain actual data) and then appended and used in report? I thought it would posible to use more PBI Datasets and then append them in some other Dataset and this one should be the source for the report. But it looks like, that only one PBI Dataset (which is stored in Service) could be used as source in report. Is there any other solution?

 

Thank you for answer

  • Hi , tomas12344 

    [11:54 AM] Yueyun Zhang (Shanghai Wicresoft Co,.Ltd.)

     

    Here's my understanding of your question:

    Q1: We can reduce the size of your dataset :

    1. Manage Date Time Columns 
    2. Delete Unneeded Columns 
    3. Remove Unnecessary Rows 
    4. Amend Column Types 
    5. Create New Columns in Power Query instead of using Calculated Column in DAX

    For more information, you can refer to this document:

    How to Reduce Dataset Size in Microsoft Power BI? (alphabold.com)

     

    Q2:Actually this is something which works in Power BI Desktop but not in Power BI service. Hence, if you unheck Include in report refresh - The functionality works only in Power BI Desktop and it will not have any impact in Power BI Service i.e. if you refresh report in Power BI service, this will still refresh all the queries even through Include in report refresh is unchecked.

     

    Q3:In Power BI Desktop, you can try using the new feature DirectQuery for Power BI Dataset to connect multiple datasets.

    For more information, you can refer to :

    Using DirectQuery for datasets and Analysis Services (preview) - Power BI | Microsoft Learn

     

    Best Regards,

    Aniya Zhang

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

3 Replies

  • Hi , tomas12344 

    [11:54 AM] Yueyun Zhang (Shanghai Wicresoft Co,.Ltd.)

     

    Here's my understanding of your question:

    Q1: We can reduce the size of your dataset :

    1. Manage Date Time Columns 
    2. Delete Unneeded Columns 
    3. Remove Unnecessary Rows 
    4. Amend Column Types 
    5. Create New Columns in Power Query instead of using Calculated Column in DAX

    For more information, you can refer to this document:

    How to Reduce Dataset Size in Microsoft Power BI? (alphabold.com)

     

    Q2:Actually this is something which works in Power BI Desktop but not in Power BI service. Hence, if you unheck Include in report refresh - The functionality works only in Power BI Desktop and it will not have any impact in Power BI Service i.e. if you refresh report in Power BI service, this will still refresh all the queries even through Include in report refresh is unchecked.

     

    Q3:In Power BI Desktop, you can try using the new feature DirectQuery for Power BI Dataset to connect multiple datasets.

    For more information, you can refer to :

    Using DirectQuery for datasets and Analysis Services (preview) - Power BI | Microsoft Learn

     

    Best Regards,

    Aniya Zhang

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

    • tomas12344's avatar
      tomas12344
      Helper III

      Thank you for the answer. What about using the dataflows. I have read something about it yesterday, but not sure if it could be used as multiple tables/files (historical and actual) in one report? I dont like to use not-final features.