topic Re: Selective update of lakehouse table in Fabric in Dataflow

Selective update of lakehouse table in Fabric

ebjim — Thu, 08 Feb 2024 02:58:43 GMT

I use a dataflow gen2 to check for changes between a lakehouse table and an onprem table. I can identify and isolate the rows that have been changed but I am having difficulty determining if there is a way in Fabric to selectively update the lakehouse table. The dataflow doesn't offer such an option. I have read articles about replaceWhere in Databricks but it does not look like Fabric can handle it. In any case, is there a way to perform selective updates of a lakehouse table based on records of another lakehouse table?

Re: Selective update of lakehouse table in Fabric

Anonymous — Thu, 08 Feb 2024 06:55:52 GMT

Hi ebjim
Thanks for using Fabric Community.
I think we can use Incremental load in your case. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.
Please refer to this document for more information: Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further queries.

Re: Selective update of lakehouse table in Fabric

Anonymous — Fri, 09 Feb 2024 15:36:48 GMT

Hi ebjim
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
Thanks