Forum Discussion

ToddChitt's avatar
ToddChitt
Super User
1 year ago
Solved

Pipeline's Dataflow task cannot 'see' my Dataflow

I am Administrator on a Workspace with a Dataflow Gen2 and am trying to incorporate that inside a Pipeline. I add the Dataflow task to the pipeline canvas and set the properties: The workspace is thi...
Need Help
  • TBNPieter's avatar
    TBNPieter
    1 year ago

    Hi,

    Just had the exact same issue with one of my colleagues. The dataflow gen2 with CI/CD support does not work in Data pipelines. This is listed in the Limitations and Known Issues of that new Dataflow with CI/CD support. 
    We just had to remake the dataflow in a normal Dataflow Gen2.
    Be advised, you have to create that Dataflow in the main folder of the Workspace otherwise it will start a Dataflow with CI/CD support and will not give you the option to disable the preview checkbox. 
    It's really disappointing to see that even basic Dataflow-features are not even working in the Preview-version.
    I am advising our entire team to stay away from the CI/CD-dataflow untill it is completely operational. This is holding back our entire CI/CD-implementation on Fabric.

    So before tinkering with the new Dataflow Preview, i would suggest to read those Restrictions & Known Issues very carefully.

    Best of luck, I hope this answers your question!

     

Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi ToddChitt

 

Thanks for your feedback.

 

In my test, CI/CD option is optional.

 

 

CI/CD preview features are still under development and may have some incomplete features or compatibility issues. These issues will be resolved before the official release, thank you for your understanding!

 

If this issue is important to you, you can create a support ticket and a dedicated Microsoft engineer will respond to you with more details.

 

It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.

 

The link of Power BI Support:  https://powerbi.microsoft.com/en-us/support/

 

For how to create a support ticket, please refer to How to create a support ticket in Power BI - Microsoft Fabric Community

 

Regards, 

Nono Chen 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly. 

  • ToddChitt's avatar
    ToddChitt
    Super User
    1 year ago

    Dataflows in Fabric, while they are very powerful, are still extremely limited. Coming soon and long overdue is the ability to set a pipeline parameter from OUTSIDE the Dataflow, like in a Pipeline task that calls said Dataflow.

     

    Currently, I must open the DF, set the Parameter value, then PUBLISH it in order for that version of the DF to be used in a Pipeline. And when I PUBLISH it, upon closing it RUNS the dataflow. So if it gets RUN, what is the sense in running it again in the Pipeline.

    If, instead, I simply open the DF, set the Parameter value, then SAVE and CLOSE, when the Pipeline runs the DF, the VERSION with the LAST PUBLISHED Parameter value gets run by the Pipeline.

    Third, when I do NOT Publish the DF, but edit the Parameter and save and close, if I manually click the "Refresh now" option in the elipsis context menu, The DF runs with the latest, UN-Published parameter value.

     

    Let's assume the Parameter value is the name of a file to be processed and loaded into a Lakehouse Destination. We have a situation where you can get different results (different file processed) FROM THE SAME SAVED DATAFLOW depending on how you run it. 

     

    This is, IMHO, absolutely HORRIBLE. It should never have been released like this.