Forum Discussion
Data Pipeline - DataFlow Gen 2, Variables assignment
Hi divyamalhotra12 ,
Inorder to understand what is Data Pipeline -
List of Data Transformation Options present in Data Pipeline - Data transformation activities - Microsoft Fabric | Microsoft Learn
Basically if you have Dataflow you can publish it inorder to do transformation.
You can schedule the published Dataflow directly without any dependency.
If you want to do prior steps before its execution you can use dataflow activity in data pipeline along with some series of activities.
It completely depends on how do you want to design your pipeline or consume your data flow gen 2.
Hope this is helpful. Please let me know if you have any further queries."If you want to do prior steps before its execution you can use dataflow activity in data pipeline along with some series of activities."
So, this means, I choose the dataflow activity in my pipeline, select the dataflow and then click on the arrow in the dataflow. That will then take me to the dataflow and I can edit, publish it and come back to my pipeline and run it?
I want to understand that,
1. Can I update the data in lakehouse? For example, can I add new rows to the existing columns? How do I update the same table/file? Or do I need to import a new file in lakehouse?
2. If the data gets updated then do the same transformations apply to the newly added data by running the same pipeline again?
3. If importing a new file is the only option then I would need to create a new dataflow gen 2 and apply transfomrations manually?
Ideally this is my use case, I have a dataset to which I want to apply some transformations. I want the data to be updated periodically and the transformations to be automatically applied to the updated data through data pipelines.
Hi divyamalhotra12 ,
1. Can I update the data in lakehouse? For example, can I add new rows to the existing columns? How do I update the same table/file? Or do I need to import a new file in lakehouse?
Yes, you can update the same table/file present in lakehouse. You can use option called append while configuring data flow gen 2.
2. If the data gets updated then do the same transformations apply to the newly added data by running the same pipeline again?
Yes, all transformations in data flow gen 2 will be applied to the data for every execution/run.
You can schdule option inorde to avoid manual trigger -
3. If importing a new file is the only option then I would need to create a new dataflow gen 2 and apply transfomrations manually?
I didn't get this point.
Hope this is helpful. Please let me know incase of further queries.
Thank you for the response.
I have two more questions.
1. I create a data pipeline. I choose Dataflow activity and select a dataflow that I created. Can I specify some transformations to be performed in the dataflow through data pipeline? Can I see those steps in data pipeline?
2. If I create a pipeline and select 'dataflow' activity and then I select an already published dataflow from my workspace, then what is the output expected once I run the pipeline. I do not understand the purpose of the same.
Hi divyamalhotra12 ,
Inorder to understand what is Data Pipeline -
List of Data Transformation Options present in Data Pipeline - Data transformation activities - Microsoft Fabric | Microsoft Learn
Basically if you have Dataflow you can publish it inorder to do transformation.
You can schedule the published Dataflow directly without any dependency.
If you want to do prior steps before its execution you can use dataflow activity in data pipeline along with some series of activities.
It completely depends on how do you want to design your pipeline or consume your data flow gen 2.
Hope this is helpful. Please let me know if you have any further queries.Hi divyamalhotra12 ,
We haven’t heard from you on the last response and was just checking back to see if we answered your query.
Otherwise, will respond back with the more details and we will try to help .