Forum Discussion

Marico's avatar
Marico
Icon for Helper IV rankHelper IV
1 year ago
Solved

Incremental data load in Power BI file

Hi,

 

I have a Power BI file with around 70-80M rows of data. The data is already summarized at the back-end and no further data compression at the database level can be done.

 

I was wondering if any is anyway to incrementally load the data in Power BI based on a date so as to not impact the performance of the report.

Need Help
  • v-tejrama's avatar
    v-tejrama
    1 year ago

    Hi Marico ,

     

    Yes, you can improve report performance even with datasets larger than 5 GB. Power BI offers several methods to help optimize speed and user experience. Here are some tips:

    1. Use Aggregations: Create summary tables to reduce data volume and improve load times. Power BI can switch between aggregated and detailed data as needed.
    2. Optimize the Data Model: Remove unused columns and tables, avoid high-cardinality columns, and use numeric keys for relationships.
    3. Use Hybrid or Composite Models: Store large fact tables in DirectQuery mode and import only smaller dimension tables to save memory.
    4. Apply Incremental Refresh: Use incremental refresh for large datasets with date/time fields to process only recent or changed data.
    5. Limit Visual and Filter Interactions: Reduce the number of slicers and filters, and use the "Apply" button for filters to control when visuals update.
    6. Split Reports: Divide complex reports into multiple pages or files so users only load what they need.
    7. Use Performance Analyzer: Identify slow visuals or queries with the Performance Analyzer and optimize them.
    8. Upgrade to Power BI Premium: Premium capacity provides more memory and processing power for large datasets.

    Thank you again.

     

11 Replies

  • v-tejrama's avatar
    v-tejrama
    Icon for Community Support rankCommunity Support
    1 year ago

    Hello Marico ,

    Thanks for reaching out to the Microsoft fabric community forum.

     

    Yes, you can achieve this in Power BI using the Incremental Refresh feature. This is particularly helpful for large datasets like yours with 70–80 million rows. Incremental Refresh lets you update only new or changed data based on a date column, rather than refreshing the entire dataset each time, which boosts performance and saves time. You set this up in Power BI Desktop, but keep in mind that incremental loading only works after publishing to the Power BI Service. In Desktop, it still does a full refresh for previewing.

    To set it up, create two DateTime parameters in Power Query: RangeStart and RangeEnd. Filter your date column using these parameters, then load the data. Right-click your table and choose “Incremental Refresh” to define how much data to keep and how often to refresh. After publishing to Power BI Service, the refreshes will run as configured.

    This feature is available in Power BI Pro, Premium Per User (PPU), or Premium capacity workspaces. Ensure your date column is in Date/Time format and properly indexed for best performance. If these requirements are met, Incremental Refresh should work effectively for your large dataset.

     

    If you need detailed setup instructions, feel free to ask.

     

    Thank you,

    Tejaswi.