Forum Discussion
SQL CDC to Fabric
Scenario:
On-prem SQL db with CDC logging enabled
Fabric is all setup
Connections to the SQL db is all good but when I try to use the LSN values __$Start_LSN for referance for the CDC ingestion, they are in binary format and cant be read by any tool I have tried in fabric.
It dont need real time, just CDC.
Hello J-Schreck,
The challenge with __$Start_LSN values appearing in binary format is expected behavior within SQL Server CDC. Since Fabric doesn’t handle this format directly, a practical approach is to convert the binary LSN values into a readable format (such as hexadecimal or timestamps) using SQL Server’s built-in functions, like CONVERT() or sys.fn_cdc_map_lsn_to_time()
sys.fn_cdc_map_lsn_to_time (Transact-SQL) - SQL Server | Microsoft Learn.For periodic CDC ingestion — given that real-time processing isn’t a requirement — Dataflows Gen2 or Data Factory pipelines are effective solutions within Fabric. These methods support scheduled data pulls and transformations.
To ensure you capture only new changes, I recommend tracking the last processed LSN in a Fabric table and using it as a reference point to fetch only incremental data on each load.
This approach should resolve the binary LSN issue and ensure smooth, repeatable CDC ingestion into Fabric.
I hope this could resolve you issue, if you need any further assistance, feel free to reach out.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thank you for using Microsoft Community Forum.
9 Replies
Hello J-Schreck,
The challenge with __$Start_LSN values appearing in binary format is expected behavior within SQL Server CDC. Since Fabric doesn’t handle this format directly, a practical approach is to convert the binary LSN values into a readable format (such as hexadecimal or timestamps) using SQL Server’s built-in functions, like CONVERT() or sys.fn_cdc_map_lsn_to_time()
sys.fn_cdc_map_lsn_to_time (Transact-SQL) - SQL Server | Microsoft Learn.For periodic CDC ingestion — given that real-time processing isn’t a requirement — Dataflows Gen2 or Data Factory pipelines are effective solutions within Fabric. These methods support scheduled data pulls and transformations.
To ensure you capture only new changes, I recommend tracking the last processed LSN in a Fabric table and using it as a reference point to fetch only incremental data on each load.
This approach should resolve the binary LSN issue and ensure smooth, repeatable CDC ingestion into Fabric.
I hope this could resolve you issue, if you need any further assistance, feel free to reach out.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thank you for using Microsoft Community Forum.
Hi J-Schreck,
I wanted to follow up on our previous suggestions regarding the issue. We would love to hear back from you to ensure we can assist you further.
If our response has addressed your query, please Accept it as a solution and give a ‘Kudos’ so other members can easily find it. Please let us know if there’s anything else we can do to help.
Thank you.
Hi J-Schreck,
May I ask if you have gotten this issue resolved?
If it is solved, please mark the helpful reply or share your solution and accept it as solution, it will be helpful for other members of the community who have similar problems as yours to solve it faster.
Thank you.