topic Re: data replication from source upon change in Data Engineering

data replication from source upon change

arkiboys2 — Thu, 14 Nov 2024 11:38:02 GMT

Hello,

I have created data factory pipelines in fabric which ingests data from sql server and lands into fabric warehouse once a day.

How is it possible to ingest the sql server table data only when they change at source?

thank you

Re: data replication from source upon change

AndyDDC — Thu, 14 Nov 2024 14:25:53 GMT

Hi @arkiboys2 there is nothing yet that can detect source data changes (e.g. CDC) so you'll need to implement some form of scheduled polling with incremental loading.

We're still waiting to see if any CDC functionality will be added, and also when we can mirror SQL Server data into Fabric itself.

Re: data replication from source upon change

arkiboys2 — Thu, 14 Nov 2024 14:31:32 GMT

mirroring is available, right?

Re: data replication from source upon change

AndyDDC — Thu, 14 Nov 2024 15:29:39 GMT

It is but only for Azure SQL Database, not SQL Server. Others supported is CosmosDB, Databricks, Snowflake,

Re: data replication from source upon change

frithjof_v — Fri, 15 Nov 2024 06:20:26 GMT

https://blog.fabric.microsoft.com/en-US/blog/mirroring-sql-server-database-to-fabric/

Re: data replication from source upon change

arkiboys2 — Sun, 17 Nov 2024 06:27:21 GMT

Does this mean in azure sql server, that if a row in a table is updated or inserted then the same table gets an upsert?