Forum Discussion
Azure Synapse Link for Dataverse in a Fabric architecture
Hi,
I'm evaluating Azure Synapse Link for Dataverse as a Bronze layer for a Microsoft Fabric Medallion Architecture and I have some questions regarding the CSV vs Delta Lake options.
From the Microsoft documentation, my understanding is that:
- Azure Synapse Link exports data to ADLS Gen2 in CSV format.
- To obtain Delta Lake tables, a Synapse Spark Pool is required to periodically convert CSV files into Delta format.
- Append Only and IsDelete support are documented in this Synapse Link scenario.
My questions are:
- Has anyone successfully used Azure Synapse Link with CSV only (without Delta Lake and without a Synapse Spark Pool) as the Bronze layer of a Medallion Architecture?
- What are the practical disadvantages of keeping the data in CSV format?
- Performance
- Schema evolution
- CDC/SCD2 processing
- Query complexity
- What are the typical costs associated with:
- ADLS Gen2 storage
- Spark Pool execution
- Has anyone implemented Azure Synapse Link exporting CSV files to ADLS and then used Fabric Spark (instead of Synapse Spark) to build the Bronze Delta layer?
I'm interested in real-world experiences, especially from teams using Dataverse as a source for a Fabric Medallion Architecture.
Thanks!
Hi Luigia-Costabil ,
Thanks for reaching out to Microsoft Fabric Community and Thanks to Vinodh247 for providing meaningful insights.Converting the Append Only CSV exports into Delta tables using Fabric notebooks does not inherently reduce the benefits of Synapse Link change tracking. The main consideration is data freshness, as the Delta Bronze layer will only reflect changes after the Fabric notebook or pipeline has processed the latest exported files.
Regarding Append Only mode, Microsoft documentation states that create, update, and delete changes are written as additional records in the destination files, and delete operations can be identified through the IsDelete column. This supports incremental processing scenarios.
However, the documentation does not explicitly state that the exported files should be treated as a complete audit trail containing every intermediate change event. Therefore, while Append Only preserves the changes exported by Synapse Link, any requirements for full historical auditing should be validated against your specific business needs.
A common Fabric approach is to retain the exported CSV files as the raw layer and load them into Delta Bronze tables using Fabric Spark notebooks or pipelines, allowing downstream workloads to benefit from Delta Lake capabilities while preserving the original exports.
For Reference:
Advanced configuration options in Azure Synapse Link Link your Dataverse environment to Microsoft Fabric and unlock deep insights
Export Dataverse data in Delta Lake formatBest Regards,
Abdul Rafi
6 Replies
Hi Luigia-Costabil ,
Thankyou for reaching out to Microsoft Fabric Community.Yes, Azure Synapse Link's CSV export can be used as a raw landing or Bronze layer in a Medallion Architecture. This approach preserves the source data in its original form and can be suitable for organizations that want to minimize transformations during ingestion. However, many Fabric implementations subsequently convert the data to Delta format, as Fabric Lakehouse workloads are optimized for Delta tables.
When data remains in CSV format, there are a few considerations. Query performance is generally lower compared to Delta tables, storage usage tends to be higher, and schema changes often require additional handling. While Synapse Link provides change tracking metadata that can support incremental processing, implementing CDC or SCD Type 2 patterns typically requires more custom logic than a Delta-based approach. Data consumption can also become more complex since schema management and data consistency must be handled outside the file format itself.
Regarding costs, ADLS Gen2 charges are primarily based on storage consumption and transaction volume. Compute costs depend on the processing engine used, data volume, and refresh frequency. Incremental processing is commonly used to help optimize compute consumption.
Using Fabric Spark instead of a Synapse Spark Pool to create the Bronze Delta layer is a valid architecture pattern and aligns well with a Fabric-first strategy.
One supported architecture is to use Azure Synapse Link to export Dataverse data into ADLS Gen2 and then use Fabric Spark notebooks or pipelines to ingest those files into Delta tables within a Fabric Lakehouse for downstream Bronze, Silver, and Gold processing.This allows the raw exported files to be retained while enabling downstream layers to benefit from Delta Lake features such as improved performance, schema evolution, and efficient merge operations.
For Reference :
Create an Azure Synapse Link for Dataverse with Azure Data Lake
Export Dataverse data in Delta Lake format
Azure Synapse FAQ'sFor a Fabric-centric Medallion Architecture, one supported approach is to use the CSV export as the raw landing zone in ADLS Gen2 and then use Fabric Spark to create Delta-based Bronze tables within Fabric Lakehouse for downstream Silver and Gold processing
Best Regards,
Abdul Rafiyes, teams have used azure synapse link exporting CSV only as a Bronze layer, but in practice it is usually a temp or transitional setup rather than a long-term design.
Keeping CSV as Bronze works, but it comes with clear trade-offs:
-
Performance: CSV is row-based, no indexing, no metadata pruning. Queries are slow and expensive at scale compared to Delta (no predicate pushdown, no file skipping).
-
Schema evolution: Weak. Any column change requires manual handling. No built-in versioning like Delta.
-
CDC / SCD2: You must implement everything yourself using
IsDelete, timestamps, and joins. This becomes complex and error-prone quickly. -
Query complexity: High. You need to repeatedly define schema, handle nulls, types, and deduplication logic in every downstream process.
In short: CSV is fine for raw landing, but not for anything beyond ingestion.
On cost:
-
ADLS Gen2: Cheap. You pay for storage (roughly $0.018 – $0.03 per GB/month depending on tier) plus transaction costs. CSV vs Delta does not change storage much, but Delta reduces compute cost later.
-
Spark (Synapse or Fabric): This is where cost matters. Synapse Spark pools are billed per vCore/hour even when idle (unless paused). Fabric Spark is capacity-based (FSKU), typically more cost-efficient if you are already invested in fabric.
Using Fabric Spark instead of Synapse Spark to convert CSV -> Delta is increasingly the preferred approach:
-
It removes the need for synapse workspace and dedicated Spark pools
-
Keeps everything inside Fabric(Lakehouse, OneLake, governance)
-
Lets you directly build bronze delta tables aligned with Medallion architecture
Real-world pattern that works well:
Dataverse -> Synapse Link (CSV in ADLS) -> Fabric Notebook/Dataflow Gen2 -> Bronze Delta in Lakehouse -> Silver/Gold
Thank you! I have a clarification to ask.
Suppose I have the following setup:
- Dataverse
- Azure Synapse Link (Append Only)
- ADLS Gen2 (CSV)
- Fabric
My question is about the Bronze layer.
If I create Fabric notebooks that periodically read the exported CSV files and materialize them into Delta Lake Bronze tables, do I lose part of the benefit of Synapse Link's near real-time change tracking?
When using Azure Synapse Link in Append Only mode, are the exported CSV snapshots guaranteed to preserve all historical change events (including intermediate updates and IsDelete records), making them suitable as a Bronze layer without materializing Delta tables?Hi Luigia-Costabil ,
Thanks for reaching out to Microsoft Fabric Community and Thanks to Vinodh247 for providing meaningful insights.Converting the Append Only CSV exports into Delta tables using Fabric notebooks does not inherently reduce the benefits of Synapse Link change tracking. The main consideration is data freshness, as the Delta Bronze layer will only reflect changes after the Fabric notebook or pipeline has processed the latest exported files.
Regarding Append Only mode, Microsoft documentation states that create, update, and delete changes are written as additional records in the destination files, and delete operations can be identified through the IsDelete column. This supports incremental processing scenarios.
However, the documentation does not explicitly state that the exported files should be treated as a complete audit trail containing every intermediate change event. Therefore, while Append Only preserves the changes exported by Synapse Link, any requirements for full historical auditing should be validated against your specific business needs.
A common Fabric approach is to retain the exported CSV files as the raw layer and load them into Delta Bronze tables using Fabric Spark notebooks or pipelines, allowing downstream workloads to benefit from Delta Lake capabilities while preserving the original exports.
For Reference:
Advanced configuration options in Azure Synapse Link Link your Dataverse environment to Microsoft Fabric and unlock deep insights
Export Dataverse data in Delta Lake formatBest Regards,
Abdul Rafi
-
Hi @Luigia-Costabil ,
Thank you for reaching out to Microsoft Fabric Community and Thanks to Vinodh247 for Sharing valuable insights.
Just wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.Best Regards,
Abdul Rafi.
Could you please confirm if the issue has been resolved? If not, feel free to reach out if you have any further questions.
Your update would be helpful for other members who may face a similar issue.
Best Regards,
Abdul Rafi
