Forum Discussion
Datamarts Preview Appending data
You need a separate ELT solution. No Power BI solutions are a proper datawarehouse - not meant be. Azure SQL, or other tools are meant for that. All Power Bi tools (dataflows/datasets/datamarts) wipe all data on refresh (or keep some historical data in an incremental refresh scenario, but that is still not meant to be a datawarehouse for storing snapshots, and those historical partitions can be purged on a schema change, losing all data).
Yes, you'll have to manage these snapshot files eventually. Cloud storage cost is really the least of your concerns these days if "100GB is plenty" for you.
You can set up incremental refresh against ADLS Storage Gen 2:
That's really neat! I was concerned over PowerBI having to read from every file from the growing data set every time and the associated network costs. If it only does this incrementally based on what has changed that does change things.
Microsoft won't charge you egress costs as long as the Azure data lake is in the same region as your Power BI tenant ("The Power BI workspace tenant region should be the same as the storage account region"):
The main concern here is ever increasing refresh times that might eventually lead to time outs if you were to always fully refresh from your growing collection of snapshots.