topic How to implement Change Data Capture in Dataflow

How to implement Change Data Capture

Mohida — Tue, 26 Mar 2024 12:56:56 GMT

Hi Team,

I have several big query tables, from which I need to bring the data to my fabric warehouse. I have created a control table in my warehouse where i maintain the last update dates pertaining to a table. I use a dataflow to bring data from bigquery table to warehouse table. Now i want to include a filter here which filters the data for - last_update-date >= (value from control table).

How should we implement this in dataflow ? Or how should one implement this using data pipelines ? Please suggest.

Also , Currently i do not see a way to pass a parameter from datapipeline to dataflow. Please let me know the correct approach for the above scenario ?

Regards

Mohida

Re: How to implement Change Data Capture

miguel — Tue, 26 Mar 2024 16:53:25 GMT

Hi!

Can you create a query which yields the value for (value from control table)? if yes, you can pass that as a filter of the query that you want to use it with as part of a "Filter rows" or Table.SelectRows function

There is currently no way to pass a parameter from a pipeline to a Dataflow. You can vote for this idea to be implemented using the link below:

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=2f761603-a400-ee11-a81c-6045bdb98602

Re: How to implement Change Data Capture

Anonymous — Thu, 28 Mar 2024 18:14:39 GMT

Hi Mohida
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks