Historical Data Load in Power BI

Hello PowerRanger03 ,

Splitting data into multiple batches is a good idea for loading large dataset. Have you tried running multiple queries to hanle this something like below :

1. Create a Query with Date Filter: 

   SELECT *
   FROM your_table
   WHERE date_column BETWEEN '2024-01-01' AND '2024-01-31'

2. Duplicate the Query for Each Month :

    duplicate above query for each month changing date range like 

    WHERE date_column BETWEEN '2024-01-01' AND '2024-01-31' -- for Jan

    WHERE date_column BETWEEN '2024-02-01' AND '2024-02-29'  -- for Feb etc

3. Load Each Batch Separately:

    Once the queries are set up, you can refresh each batch one by one. This reduces the memory load on the 

    gateway, as each batch will be smaller in size. You can disable parallel loading from setting to ensure a

    single batch is running.   

4. Combine Queries: 

    Once all the queries are loaded, you can combine them into a single table using Append Queries  in Power 

    BI to create a single dataset

You can change date range as per your need.

I hope this helps.

Did I answer your query , if yes please mark this as solution. You can say special thank you by giving Kudos

Cheers

Hi PowerRanger03 ,

Would you be concerned that the initial incremental refresh might time out due to the large volume of data?If so, I think this link will help you a lot:

Advanced incremental refresh - Prevent timeouts on initial full refresh

Best Regards,

Bof