Forum Discussion

WishAskedSooner's avatar
WishAskedSooner
Continued Contributor
1 year ago
Solved

Dataflow Gen2 Connections

Do I understand this correctly? Dataflow Gen2 (PowerQuery) has the ability to connect to over 300 different sources. Yet, despite this massive list, Data Pipeline activity output/parameters/variables...
  • mllopis's avatar
    1 year ago

    Thanks for the feedback, and sorry that you're struggling to realize the value or make specific use cases work within Data Factory in Fabric.

     

    I am not entirely sure about which one of these two use cases you are aiming for, so let me cover both:

    1. Having parameterized dataflows, which you can orchestrate in a Data Pipeline and pass different Dataflow parameter values at runtime - First-class support for this common usage pattern is something that we're working towards (and tracked as a very popular feature request here). In the meantime, there are ways for you to emulate this behavior with a more complex solution that is based on passing those "runtime parameters" as data values stored in a table that Pipeline can write to (e.g. to set the value) and Dataflow can read from (as dynamic input to your dataflow queries). A similar pattern is covered in this blog: CHANGE (IN THE HOUSE OF LAKES) - It's Not About The Cell (itsnotaboutthecell.com)
    2. Being able to transform data ingested via a Data Pipeline using a Dataflow: This should be possible today as long as you pick a data sink in your Data Pipeline that can be leveraged as a data source in the dataflow. Furthermore, you can within the same Data Pipeline orchestrate the execution of your dataflow when the Data Pipeline activity that ingests data into that location completes. This article provides further details on the latter: Use a dataflow in a pipeline - Microsoft Fabric | Microsoft Learn

    Thank you again for your feedback. Please let us know if this information helped or you have further questions.

     

    Regards,
    M.