topic Re: Saving a daily copy of inventory table to lakehouse in Dataflow

Saving a daily copy of inventory table to lakehouse

mds123 — Tue, 16 Apr 2024 13:24:16 GMT

I'm brand new to fabric and the data factory tools. I have an inventory table that shows the current invenotry for all my products. I'm looking to build a table that has a daily snapshot of the inventory. I'm assuming I could do this with incremental refresh but how do I make it so I don't delete historic information?

I'm brand new to all the Fabric / Data Factory tools so any help would be amazing.

Re: Saving a daily copy of inventory table to lakehouse

Anonymous — Wed, 17 Apr 2024 07:21:38 GMT

Hi mds123 ,

Thanks for using Fabric Community.
If you looking got incremental load then you can refer these documents -
Incremental Load in Fabric
Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

FYI: In future release we provide the incremental refresh.

Inorder to know future release - What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Hope this is helpful. Please let me know incase of further queries.

Re: Saving a daily copy of inventory table to lakehouse

Anonymous — Thu, 18 Apr 2024 10:49:49 GMT

Hi mds123 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Saving a daily copy of inventory table to lakehouse

Anonymous — Fri, 19 Apr 2024 10:32:25 GMT

Re: Saving a daily copy of inventory table to lakehouse

frithjof_v — Sat, 20 Apr 2024 10:56:25 GMT

If you don't want to delete historic data, then you can use the Append mode. In this way, you just add new rows, without deleting old rows. I would add a snapshot timestamp column so you can keep track of which rows belong to which snapshot.

Re: Saving a daily copy of inventory table to lakehouse

miguel — Tue, 30 Apr 2024 00:09:07 GMT

you don't really need incremental refresh for this. If your goal is to take a snapshot, then just run the Dataflow Gen2 perhaps every day and then append the data to a "snapshots" table. Each run would be the snapshot at a particular date. You can also add another column where you can set a timestamp as to when the process was executed.

Again, it doesn't appear that you need incremental refresh, but that feature will come soon as well.

Re: Saving a daily copy of inventory table to lakehouse

miguel — Tue, 30 Apr 2024 00:10:16 GMT

It doesn't seem like you'd need incremental refresh.

You can just run the dataflow every day, add a column to have the timestamp and just append the result of every day execution to a "snapshots" table. Again, it doesn't seem like you need incremental refresh at all and just doing an Append operation would suffice.