Forum Discussion
Bulk Import and Incremental Refresh
Hi everyone,
I have a table with the results coming from a web service API call. I am looking to set this table up using increment refresh as I am able to send the RangeStart and RangeEnd as parameters to the API call. This all seems to work fine with the data I receive.
The issue that I have is that for this initial load in Incremental Refresh, the RangeStart and RangeEnd are too far apart and the API will fail to return the data. I cannot change RangeStart and RangeEnd as they are used elsewhere in the semantic model. I wish to now import historical data into that same table as a one-off import.
Is it possible to import the data from an Excel file one-time, and then setup Incremental Refresh from a daily API call? If so, how would this be achieved?
Jason
Hi JasonBurdetts ,
Based on the inputs, this is how this it will look
Partitions :
From August to January, data will be organized by month. In February, it will be organized by day. Once we reach March 8, the February daily data will be consolidated into monthly data, and March will be arranged by day.
Documentation : https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overviewPartition Date Range
2025-08 01-Aug-2025 → 31-Aug-2025
2025-09 01-Sep-2025 → 30-Sep-2025
2025-10 01-Oct-2025 → 31-Oct-2025
2025-11 01-Nov-2025 → 30-Nov-2025
2025-12 01-Dec-2025 → 31-Dec-2025
2026-01 01-Jan-2026 → 31-Jan-2026
2026-02-01 (Day wise) 01-Feb-2026
---- 02 Feb to 23 Feb --- Day wise partitions
2026-02-25 (Day wise) 25-Feb-2026
Thanks
If this response was helpful in any way, I’d gladly accept a kudo.
Please mark it as the correct solution. It helps other community members find their way faster
9 Replies
Hi JasonBurdetts ,
I have a workaround to overcome this issue.
define a parameter as LoadAllData
In Power Query stepLet .... SampleData = Table.FirstN(Source, 10), Check = if LoadAllData then Source else SampleData in Check
Keep the default value as False and publish the model to service and do the first refresh .
The first refresh will take care of creating all the partitions in the servivce , now set the LoadAllData to True and process the partitions one by one using ssms or fabric notebook.Thanks
If this response was helpful in any way, I’d gladly accept a kudo.
Please mark it as the correct solution. It helps other community members find their way fasterThis is really appreciated, but I could use some more clarification. Are you suggesting that I:
- Preload the historical data into a separate table
- Use the flag to first load the data from the separate table into the destination table
- Change the flag and then pull from the webservice instead with the incremental refresh in place.
Am I understanding this correctly?
Jason
Hi JasonBurdetts , I don't think wecan dynamically change the sources for old and new partition. However, if your ApI layer supports the range start and end, you can retrieve the old data from the API itself, right?
Thanks
This is the source of the original problem. I *can* pull historical data from the API, but initial range (several months) is too large for the API to return. This is why I need to import directly a bulk set, then incrementally update.
Hi JasonBurdetts , instead of loading the historical data as a single partition (Say for the entire year) you can break it by month by adjusting the incremental refresh policies and load the data by months
ThanksThank you again for the response. I am new to the incremental refreshes, so how they work is not entirely clear. Let's say I set the archive data to 6 months, and then I set the refresh to be 7 days. Will the initial data loading:
- Start at 6 months ago and ask the API for data between 6 months and 6 months + 7 days
- Then do the next 7 days with another API call
- Keep doing 7 day increments until it has "caught up"?
If this is the case, will it keep retrying a period if the API becomes rate limited?
Jason
Hi JasonBurdetts ,
Based on the inputs, this is how this it will look
Partitions :
From August to January, data will be organized by month. In February, it will be organized by day. Once we reach March 8, the February daily data will be consolidated into monthly data, and March will be arranged by day.
Documentation : https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overviewPartition Date Range
2025-08 01-Aug-2025 → 31-Aug-2025
2025-09 01-Sep-2025 → 30-Sep-2025
2025-10 01-Oct-2025 → 31-Oct-2025
2025-11 01-Nov-2025 → 30-Nov-2025
2025-12 01-Dec-2025 → 31-Dec-2025
2026-01 01-Jan-2026 → 31-Jan-2026
2026-02-01 (Day wise) 01-Feb-2026
---- 02 Feb to 23 Feb --- Day wise partitions
2026-02-25 (Day wise) 25-Feb-2026
Thanks
If this response was helpful in any way, I’d gladly accept a kudo.
Please mark it as the correct solution. It helps other community members find their way fasterRight, I think I understand it better.
- By partitioning the table into months, it creates an area reserved for August, September, October, etc where each area is defined by its own min and max date ranges. i.e. August 01 to August 31 for example
- The service then makes a call to the API passing those two dates for that month and only getting the August table, filling up that August partition
- It continues til Feb, where it starts filling up February from 01 to the current date. On the next refresh, it throws out the data from Feb 01 and refills it from the refresh
- Once it reaches 7 days into March (based on my setting), it will no longer request Feb data and start from March 01
Is this understanding correct?
Hi JasonBurdetts
Thanks for reaching out to the Microsoft fabric community forum.I would also take a moment to thank Natarajan_M , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.
You’re very close, and here’s the simplest way to understand it. When you set Archive = 6 months and Refresh = 7 days, Power BI automatically splits the older six months into monthly partitions, such as August, September, and October, and splits the most recent seven days into daily partitions. During the first refresh, it loads one full month at a time for the historical data and one day at a time for the most recent seven days. It does not move forward in 7-day chunks across the entire history. For example, August would be loaded with one API call covering August 1 to August 31, September would be another single call for that full month, and the current month (like February) would be loaded day by day.
On future refreshes, only the last seven days are refreshed again, while older months are left untouched. Once a month is completely outside the seven-day refresh window, it becomes fixed and will no longer be refreshed. If the API fails due to rate limits or any other issue, the refresh will fail, and it will not automatically keep retrying that partition. So overall, Power BI makes separate API calls based on each partition’s date range, but it does not step forward in repeated 7-day blocks across all historical data.
Best Regards,
Community Support Team
