Forum Discussion
Creating historical data with/without Incremental refresh
- 1 year ago
Hi JakubV
to keep historical data, you will need to have your data to be stored somewhere.
Within the Microsoft Fabric environment you could achieve this using a data warehouse combined with Gen2 dataflows.
First you will want to create a warehouse. Here's how: Create a Warehouse - Microsoft Fabric | Microsoft Learn
After that you will set up your Gen2 dataflow within a workespace of your choice just like a standard dataflow, but additionally you will add a data destination in the bottom right corner of the dataflow interface. Just choose your warehouse as destination and every time you refresh the dataflow, the result of the table will be written inside the warehouse. While setting up the destination, you will want to use the "append" option.
Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
Hope this helps.
Cheers
Tim
Thanks. With the several possible setups I ran this would be the only solution. I will either be ruuning a solution creating external data files with the help of Power Automate (and appending the result files) or will have a SQL database created with a set of historical and temporary (current data run) tables.
Sounds good. If you need any further advice, don't hesitate to ask.