Forum Discussion

JasonBurdetts's avatar
JasonBurdetts
Frequent Visitor
6 months ago
Solved

Bulk Import and Incremental Refresh

Hi everyone,   I have a table with the results coming from a web service API call. I am looking to set this table up using increment refresh as I am able to send the RangeStart and RangeEnd as para...
Need Help
  • Natarajan_M's avatar
    Natarajan_M
    6 months ago

    Hi JasonBurdetts  , 

    Based on the inputs, this is how this it will look  


    Partitions :

    From August to January, data will be organized by month. In February, it will be organized by day. Once we reach March 8, the February daily data will be consolidated into monthly data, and March will be arranged by day.



    Documentation : https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview

    Partition                                Date Range

    2025-08                                01-Aug-2025 → 31-Aug-2025
    2025-09                                01-Sep-2025 → 30-Sep-2025
    2025-10                                01-Oct-2025 → 31-Oct-2025
    2025-11                                01-Nov-2025 → 30-Nov-2025
    2025-12                                01-Dec-2025 → 31-Dec-2025
    2026-01                                01-Jan-2026 → 31-Jan-2026
    2026-02-01 (Day wise)         01-Feb-2026
    ---- 02 Feb to 23 Feb  ---      Day wise partitions
    2026-02-25 (Day wise)         25-Feb-2026




    Thanks 

    If this response was helpful in any way, I’d gladly accept a kudo.
    Please mark it as the correct solution. It helps other community members find their way faster

Natarajan_M's avatar
Natarajan_M
Super User
6 months ago

Hi JasonBurdetts , 

I have a workaround to overcome this issue.

define a parameter as LoadAllData 

 


In Power Query step 

 

Let 

....

SampleData = Table.FirstN(Source, 10),
Check = if LoadAllData then Source else SampleData 



in 

Check


Keep the default value as False and publish the model to service and do the first refresh .

The first refresh will take care of creating all the partitions in the servivce , now set the LoadAllData to True and process the partitions one by one using ssms or fabric notebook.

 

 

Thanks

 

If this response was helpful in any way, I’d gladly accept a kudo.
Please mark it as the correct solution. It helps other community members find their way faster

JasonBurdetts's avatar
JasonBurdetts
Frequent Visitor
6 months ago

This is really appreciated, but I could use some more clarification. Are you suggesting that I:

  • Preload the historical data into a separate table
  • Use the flag to first load the data from the separate table into the destination table
  • Change the flag and then pull from the webservice instead with the incremental refresh in place.

Am I understanding this correctly?

 

Jason