Forum Discussion
Advocate IAzure Synapse Link for Dataverse to Fabric - CSV management
Hi Luigia-Costabil , Thank you for reaching out to the Microsoft Community Forum.
1. Do I need Synapse SQL pool to get delta?
You do NOT need Synapse SQL Pool to get delta (CDC) from Synapse Link (Append Only). You already have the delta in CSV form.
2. How do I consume delta without Synapse?
You read files incrementally from ADLS. Use columns like IsDelete and timestamps (SinkCreatedOn, modifiedon). Reconstruct CDC yourself in Fabric.
Please refer below sample code.df = spark.read.csv("adls_path", header=True)
# Deduplicate by key + timestamp
windowed_df = (
df.withColumn("rn",
row_number().over(
Window.partitionBy("primaryKey")
.orderBy(col("SinkCreatedOn").desc())
)
)
.filter("rn = 1")
)Note: This is exactly what Synapse SQL pool would have done for you but you are doing it in Fabric (cheaper + more control).
Please try below Architecture.
1. Synapse Link (Append Only) --> Cheap ingestion and Full CDC
2. ADLS Gen2 (CSV, cheap storage)
3. Fabric Shortcut --> No data duplication and Direct access
4. Bronze Delta tables --> Reliability, Performance and Fabric-native
5. Silver (SCD2)Note: Do NOT use Synapse SQL pool.
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
Advocate IHi @arabalca , thanks for your reply.
I consulted the following Microsoft documentation:
Create an Azure Synapse Link for Dataverse with Azure Data Lake in Power Apps - Power Apps | Microsoft Learn
My goal is to avoid using Link to Fabric because the costs of Dataverse storage are high. So, I'd like to transfer the data to ADSL Gen2 because the data storage cost is low. The documentation I attached states that the format used is CSV. To obtain the delta, I'd need to connect an Azure Synapse Pool, which I'd like to avoid because I'm already paying for Fabric capacity.
I hope I have explained my intentions more clearly.
Community SupportHi Luigia-Costabil , Thank you for reaching out to the Microsoft Community Forum.
1. Do I need Synapse SQL pool to get delta?
You do NOT need Synapse SQL Pool to get delta (CDC) from Synapse Link (Append Only). You already have the delta in CSV form.
2. How do I consume delta without Synapse?
You read files incrementally from ADLS. Use columns like IsDelete and timestamps (SinkCreatedOn, modifiedon). Reconstruct CDC yourself in Fabric.
Please refer below sample code.
df = spark.read.csv("adls_path", header=True)
# Deduplicate by key + timestamp
windowed_df = (
df.withColumn("rn",
row_number().over(
Window.partitionBy("primaryKey")
.orderBy(col("SinkCreatedOn").desc())
)
)
.filter("rn = 1")
)
Note: This is exactly what Synapse SQL pool would have done for you but you are doing it in Fabric (cheaper + more control).
Please try below Architecture.
1. Synapse Link (Append Only) --> Cheap ingestion and Full CDC
2. ADLS Gen2 (CSV, cheap storage)
3. Fabric Shortcut --> No data duplication and Direct access
4. Bronze Delta tables --> Reliability, Performance and Fabric-native
5. Silver (SCD2)
Note: Do NOT use Synapse SQL pool.
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
- v-dineshya
Hi Luigia-Costabil ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.
Regards,
Dinesh
- v-dineshya
Hi @Luigia-Costabil ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.
Regards,
Dinesh