Forum Discussion
Solved
Dataflow vs. Dataflow Gen2 vs. Warehouse
I have the following scenario: 1. Step 1 - I want to connect to Salesforce and get an Opportinities table AS-IS 2. I want to connect to the table from Step 1 and do all kinds of complex transfo...
Thanks for the follow up!
- The output can become anything you want! I believe you needed to trigger an API in salesforce before the refresh starts, if that is not the case you can ignore this activity in the pipeline and straight away run the first dataflow to consume from salesforce and load it into your lakehouse
- Some advantages compared to gen1 are:
- You can decide where to save the data with data destinations feature in gen2, which can easily be accessed by other means. So you are not restricted to the dataflow connector anylonger as you were with dataflows gen1.
- Bringing in the data raw into your lakehouse allows you to build different dataflows on top of it for experimenting or other business needs without putting load on your salesforce instance multiple times.
- Gen2 compared to Gen1 the performance for complex transformations will be better as we try to leverage the compute endpoints of the staging lakehouse and warehouse which in many cases can be way faster than the native PowerQuery engine when working with large datasets.
Hope this clarifies 😁
The theory tells you that the 2nd way of working is good. It is common for data to be loaded into a DWH as is, and only to be transformed afterwards. The goal of this approach is often to not disturb the connection with the source for too long and having an easier time debugging.
But isn't it exactly what two separate Dataflows Gen1 can achieve? Without need for DWH?
