ADF vs Fabric

[MatrixMan112]

Hey fabricators

I am new to Fabric and trying to understand the benefits of switching from ADF to Fabric. Currently I am using the trial version. I just want to understand is it beneficial to switch to Fabric despite having so many bugs and items in preview. 

I have tried dataflows , really slow performance for high volume. Please give your suggestions.

Thanks,

Edward

[paul_hasell]

I have yet to find any workload for which Fabric is the better choice over ADF/SQL is large scale data processing, either on performance or cost. Indeed, so far my experience has been that it has a long way to go before it becomes performant and reliable enough for business critical processes. The platform is a good idea but does not have the maturity required, this will change no doubt.

I'd give it a break for now and review it again in, say, a years time.

[Anonymous · Accepted solution]

Hi @MatrixMan112 
Thanks for using Fabric Community.

Switching from Azure Data Factory (ADF) to Fabric can have several benefits, but it also depends on your specific use case and requirements. Here are some points to consider:

1. Performance: Fabric is designed to handle large-scale data processing and analytics workloads, and it can provide better performance for high-volume dataflows compared to ADF.

2. Flexibility: Fabric provides a more flexible and interactive environment for data exploration and transformation. It supports both batch and streaming data processing, and it allows you to use a variety of languages and libraries, including Python, Scala, and SQL.

3. Integration: Fabric integrates well with other Azure services, making it easier to build end-to-end data pipelines. It also supports Delta Lake, which provides ACID transactions, scalable metadata handling, and unified batch and streaming data processing.

4. Cost: Depending on your workload, Fabric could potentially be more cost-effective than ADF, especially for large-scale data processing tasks.

However, as you mentioned, Fabric is still in preview and there might be some bugs or features that are not fully developed yet. Our engineering team is actively working on improving dataflow performance. We're constantly optimizing the platform and addressing bugs.

In terms of the performance issues you’re experiencing with dataflows, there could be several factors at play, including the size and complexity of your data, the configuration of your Spark cluster, and the specific transformations you’re applying. It might be worth exploring ways to optimize your dataflows, such as partitioning your data, optimizing your transformations, or scaling up your Spark cluster.

You can refer to these links for more information:
https://learn.microsoft.com/en-us/fabric/data-factory/compare-fabric-data-factory-and-azure-data-fac...
https://learn.microsoft.com/en-us/fabric/data-factory/frequently-asked-questions
https://www.linkedin.com/pulse/microsoft-azure-data-factory-adfms-fabric-vs-synapse-analytics-vgogf/
https://www.casewhen.co/blog/data-factory-showdown-fabric-vs-azure

I hope this helps! If you have any more questions, feel free to ask.

[MatrixMan112]

That was an indeed a descriptive answer. 

Thanks @Anonymous 

[Anonymous]

Hi @MatrixMan112 
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.