Forum Discussion
Data Pipeline - DataFlow Gen 2, Variables assignment
1. I have a dataflow in Data Factory to which I applied some transformations such as 'Unpivot Columns' and 'Group By'. I want to create a pipeline which ensures that new data gets added in my dataflow automatically and any transformation such as 'unpiviot' or 'group by' that I applied to my dataflow earlier gets applied to the new data that is added.
2. I have a question regarding the 'Variable' feature in Data Pipeline. I created a pipeline wherein I have dataflow gen 2 as first activity and on success of that activity, I want to assign my dataflow a variable. The pipeline runs successfully but I do not see anything in the output.'
Hi divyamalhotra12 ,
Inorder to understand what is Data Pipeline -
List of Data Transformation Options present in Data Pipeline - Data transformation activities - Microsoft Fabric | Microsoft Learn
Basically if you have Dataflow you can publish it inorder to do transformation.
You can schedule the published Dataflow directly without any dependency.
If you want to do prior steps before its execution you can use dataflow activity in data pipeline along with some series of activities.
It completely depends on how do you want to design your pipeline or consume your data flow gen 2.
Hope this is helpful. Please let me know if you have any further queries."If you want to do prior steps before its execution you can use dataflow activity in data pipeline along with some series of activities."
So, this means, I choose the dataflow activity in my pipeline, select the dataflow and then click on the arrow in the dataflow. That will then take me to the dataflow and I can edit, publish it and come back to my pipeline and run it?
9 Replies
Hi divyamalhotra12 ,
Thanks for using Fabric Community.
If you are taking about Incremental Load using Data Flow Gen 2, then i recommend this - Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
I think your understanding towards Set Variable is different.Can you please check the output there you can find the value.
Docs to refer -
Set Variable Activity - Azure Data Factory & Azure Synapse | Microsoft Learn
Set Pipeline Return Value - Azure Data Factory & Azure Synapse | Microsoft Learn
Hope this is helpful. Please let me know in case of further queries.I want to understand that,
1. Can I update the data in lakehouse? For example, can I add new rows to the existing columns? How do I update the same table/file? Or do I need to import a new file in lakehouse?
2. If the data gets updated then do the same transformations apply to the newly added data by running the same pipeline again?
3. If importing a new file is the only option then I would need to create a new dataflow gen 2 and apply transfomrations manually?
Ideally this is my use case, I have a dataset to which I want to apply some transformations. I want the data to be updated periodically and the transformations to be automatically applied to the updated data through data pipelines.
Hi divyamalhotra12 ,
1. Can I update the data in lakehouse? For example, can I add new rows to the existing columns? How do I update the same table/file? Or do I need to import a new file in lakehouse?
Yes, you can update the same table/file present in lakehouse. You can use option called append while configuring data flow gen 2.
2. If the data gets updated then do the same transformations apply to the newly added data by running the same pipeline again?
Yes, all transformations in data flow gen 2 will be applied to the data for every execution/run.
You can schdule option inorde to avoid manual trigger -
3. If importing a new file is the only option then I would need to create a new dataflow gen 2 and apply transfomrations manually?I didn't get this point.
Hope this is helpful. Please let me know incase of further queries.