topic Change Data Capture in Dataflow

Change Data Capture

PSJohnG — Thu, 25 May 2023 16:29:21 GMT

I'm hoping that we will see an increase in connectors that support CDC, we have SQL Server and SAP I believe now, but would love to see this expanded out over Oracle, DB2 (LUW & AS400), MySQL, Postrgre etc. Working with larger source systems across Wans; it's just not efficent to move millions let alone billions of rows repeatedly, not everything has a reliable timestamp, nor the control to modify the source table structures to add.

Re: Change Data Capture

miguel — Thu, 25 May 2023 23:01:33 GMT

Hey!

Could you please share more information? What are you referring to Change Data Capture in the context of Microsoft Data Factory Dataflows?

Re: Change Data Capture

ScottPowellECU — Thu, 01 Jun 2023 21:33:31 GMT

I'm sure he's in exactly the same case I am. We have an on-prem SQL server database that will most likely never move to the cloud - is supports our EPIC medical system. I'd love to move the data from it up to Fabric. But there is no way that nightly I can move billions of rows of data.

We'd like CDC so we can do one gigantic "initial load", and after that run nightly incremental processes to only get records that changed. That "should" be doable from a volume standpoint - I don't think copying the entire database is.

Note this is on-prem SQL server, not Azure SQL.

Hope this helps,

Scott

Re: Change Data Capture

Anonymous — Tue, 06 Jun 2023 10:06:03 GMT

This is Azure Data Factory feature missing in Fabric Data Factory (Data Pipeline), right? So the question is, when it will be also available in Fabric.

Re: Change Data Capture

miguel — Tue, 06 Jun 2023 14:30:06 GMT

help us prioritize this feature. Please post your idea on https://ideas.fabric.microsoft.com/ and share the link to the idea here

Re: Change Data Capture

ScottPowellECU — Thu, 17 Apr 2025 21:37:31 GMT

And just as an update, it sounds like on-prem SQL server and Oracle mirroring are in private preview. Hopefully we can start kicking the tires on this soon.

Scott