Forum Discussion
generic question about how dataflow store the data
If i have two dataflow Gen1
dataflow 1 : ingestion dataflow. Reads the data from data source (SQL for example)
dataflow 2 : transformation dataflow. Uses ingestion dataflow as data source.
my questions.
1. When i reload the dataflow 1 (ingestion) .. Where is the data stored? is it part of the dataflow? is it stored at a technical blob storage that us mortals can't see.
2. When i reload dataflow 2 (transformation) ... is the data fetch from actual data source (SQL in this case) .. OR is the data taken directly from dataflow 1?
3. is the Staging Enabled feature have any role in this?
4. is DF Gen2 any different here? Except that you have the option the store the data into a lakehouse for example
Hi amien
1. It is stored within the data flow
2. It will get the data from your data flow 1.
3. Wednesday staging is enabled it will then get this using a SQL endpoint.
4. Yet, the data flow gin 2 is totally rewritten under the hoods. So is it a lot faster and will be able to store the output in a lake house or sql server, as an example.
1 Reply
Hi amien
1. It is stored within the data flow
2. It will get the data from your data flow 1.
3. Wednesday staging is enabled it will then get this using a SQL endpoint.
4. Yet, the data flow gin 2 is totally rewritten under the hoods. So is it a lot faster and will be able to store the output in a lake house or sql server, as an example.