Forum Discussion

iBusinessBI's avatar
iBusinessBI
Kudo Collector
2 years ago

Dataflow vs. Dataflow Gen2 vs. Warehouse

I have the following scenario:

1. Step 1 - I want to connect to Salesforce and get an Opportinities table AS-IS

2. I want to connect to the table from Step 1 and do all kinds of complex transformations with Power Query.

 

My task is to make an API request JUST ONCE to Salesforce. All the rest load should be done inside Microsoft capacity. So I want to separate the two steps most efficiently.

(When I do everything inside ONE Dataflow - the second step is sending API requests again and again and again to Salesforce so it is not an option)

 

I have created 2 dataflows and the second one refers to the first (it created a "linked" entity with enhances mode, watever it means). And I've created a Power Automate flow so the second dataflow will refresh after the first is finished.

But it still takes a long time for the second one to run. And I wonder whether it still propagates the API calls to Salesforce because of "linked entity".

 

I am thinking to test another alternative:

Create a Gen2 Dataflow, save the initial "AS-IS" table in a Warehouse/or Lakehouse.
Then, connect the second Dataflow to it. And schedule everything with Pipeline.

 

Should there be differences in this approach?

Or is it more or less the same as just separating two Gen1 dataflows?

 

What does the theory tell?

Need Help

2 Replies

  • aj1973's avatar
    aj1973
    Community Champion
    2 years ago

    Hi iBusinessBI 

    Dataflow and Dataflow Gen2 are two features of Microsoft Fabric Data Factory that allow you to transform data using a low-code interface and Power Query. Dataflow Gen2 is the newer and improved version of Dataflow, with some key differences:

    • Dataflow Gen2 has a shorter and simpler authoring experience, with auto-save and background publishing features.
    • Dataflow Gen2 supports more data destinations, such as Fabric Lakehouse, Azure Data Explorer, Azure Synapse Analytics, and Azure SQL Database.
    • Dataflow Gen2 has more powerful compute engines and data movement capabilities, enabling high-scale and high-performance data transformations.
    • Dataflow Gen2 supports Direct Query and Incremental Refresh via the Dataflow connector.

    Differences between Dataflow Gen1 and Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

    • iBusinessBI's avatar
      iBusinessBI
      Kudo Collector
      2 years ago

      Thanks, Amine. 
      I wanted a solution to my specific scenario. 
      I am pretty much convinced that I am good with Gen1 dataflows, maybe except for "improved compute engine" of Gen2