Forum Discussion
Dataflow vs. Dataflow Gen2 vs. Warehouse
Thanks for the follow up!
- The output can become anything you want! I believe you needed to trigger an API in salesforce before the refresh starts, if that is not the case you can ignore this activity in the pipeline and straight away run the first dataflow to consume from salesforce and load it into your lakehouse
- Some advantages compared to gen1 are:
- You can decide where to save the data with data destinations feature in gen2, which can easily be accessed by other means. So you are not restricted to the dataflow connector anylonger as you were with dataflows gen1.
- Bringing in the data raw into your lakehouse allows you to build different dataflows on top of it for experimenting or other business needs without putting load on your salesforce instance multiple times.
- Gen2 compared to Gen1 the performance for complex transformations will be better as we try to leverage the compute endpoints of the staging lakehouse and warehouse which in many cases can be way faster than the native PowerQuery engine when working with large datasets.
Hope this clarifies 😁
Hi BusinessBI!
My name is Jeroen Luitwieler and a Senior Product Manager on the Dataflow/dataintegration team.
Looking at your case you may want to consider the following:
Steps:
- Create a lakehouse in your workspace
- Create a pipeline in your fabric workspace
- Add a Web activity to sent a http request to salesforce
- On success, run the first Dataflow Gen2 that:
- Raw ingest all the tables you are interested in
- Use data destination to load the tables into the lakehouse
- Disable staging for all tables to improve performance
- On success of that Dataflow, run another dataflow that:
- Consumes the ingested data from the lakehouse
- Apply the transformations to the data
- Write the results of your transformations into your data destination of your liking
- Apply the schedule to the data pipeline that fits your business needs
Would this cover your usecase? do you have any questions or concerns with this solution?
Thanks, LuitwielerMSFT Jeroen,
That's interesting.... Some thoughts:
1. Where does the Web activity saves its output to? (What is the source for the "Ingest to lakehouse" DF?)
2. What is the advantage of saving the intermediate results to a Lakehouse?
I could just build 2 regular Gen1 Dataflows:- the first connects to the Salesforce and brings all the table in the raw format as-is.
- the second DF connects to the first and does all the necessary transformation.
So what advantage I get working with the Lakehouse and Gen2 DF, instead of just 2 regular Gen1 DF?
ThanksThanks for the follow up!
- The output can become anything you want! I believe you needed to trigger an API in salesforce before the refresh starts, if that is not the case you can ignore this activity in the pipeline and straight away run the first dataflow to consume from salesforce and load it into your lakehouse
- Some advantages compared to gen1 are:
- You can decide where to save the data with data destinations feature in gen2, which can easily be accessed by other means. So you are not restricted to the dataflow connector anylonger as you were with dataflows gen1.
- Bringing in the data raw into your lakehouse allows you to build different dataflows on top of it for experimenting or other business needs without putting load on your salesforce instance multiple times.
- Gen2 compared to Gen1 the performance for complex transformations will be better as we try to leverage the compute endpoints of the staging lakehouse and warehouse which in many cases can be way faster than the native PowerQuery engine when working with large datasets.
Hope this clarifies 😁
