Forum Discussion
Power BI Incremental Refresh Behavior with API Question
Remember that Incremental Refresh has two benefits. First is the speed-up when you only have process the new data, and second is that your data can grow beyond 10 GB thanks to the partition management.
The first benefit is ONLY realized when your data source query is foldable. That is not the case with an API data source.
The second benefit is only realized when your partition boundaries are set by an immutable data source field (ie "Date created" rather than "Date modified".
You mentioned the archiving - neither Power BI Semantic Models nor Dataflows Gen1 support that. You would only get that in Fabric with Dataflows Gen2 ("Append mode") and even then there is no safety net. You need to consider a scenario when your Power BI data is corrupted and you have to re-load everything from source. If your source does not support that then you will need to create your own archiving solution , independently of Power BI.
Remember that Incremental Refresh has two benefits. First is the speed-up when you only have process the new data, and second is that your data can grow beyond 10 GB thanks to the partition management.
The first benefit is ONLY realized when your data source query is foldable. That is not the case with an API data source.
The second benefit is only realized when your partition boundaries are set by an immutable data source field (ie "Date created" rather than "Date modified".
You mentioned the archiving - neither Power BI Semantic Models nor Dataflows Gen1 support that. You would only get that in Fabric with Dataflows Gen2 ("Append mode") and even then there is no safety net. You need to consider a scenario when your Power BI data is corrupted and you have to re-load everything from source. If your source does not support that then you will need to create your own archiving solution , independently of Power BI.