Pipeline's Dataflow task cannot 'see' my Dataflow

[ToddChitt]

I am Administrator on a Workspace with a Dataflow Gen2 and am trying to incorporate that inside a Pipeline. I add the Dataflow task to the pipeline canvas and set the properties: The workspace is this workspace, but it does not 'see' the Dataflow I just created. 

The Datflow runs just fine by istelf and has several successful runs in history. It is not broken.

There are three Dataflows in the Workspace and the pipeline task only sees ONE of them. 

Any ides?

Could it be because the Dataflow is in a different Folder than the root? Both the Dataflow and Pipeline are contained in the same folder. 

Thanks in advance.

Addendum: The one Dataflow that the pipeline CAN see is simply listed as type: "Dataflow Gen2". But the other two are listed as "Dataflow Gen2 (CI/CD, preview)". Is THAT it? 

I created a new Dataflow and it automatically adds the CI/CD option and does NOT give me the option to disable it. WTF?!?

[TBNPieter · Accepted solution]

Hi,

Just had the exact same issue with one of my colleagues. The dataflow gen2 with CI/CD support does not work in Data pipelines. This is listed in the Limitations and Known Issues of that new Dataflow with CI/CD support. 
We just had to remake the dataflow in a normal Dataflow Gen2.
Be advised, you have to create that Dataflow in the main folder of the Workspace otherwise it will start a Dataflow with CI/CD support and will not give you the option to disable the preview checkbox. 
It's really disappointing to see that even basic Dataflow-features are not even working in the Preview-version.
I am advising our entire team to stay away from the CI/CD-dataflow untill it is completely operational. This is holding back our entire CI/CD-implementation on Fabric.

So before tinkering with the new Dataflow Preview, i would suggest to read those Restrictions & Known Issues very carefully.

Best of luck, I hope this answers your question!

[ToddChitt]

@TBNPieter Thanks for your reply.

>>Be advised, you have to create that Dataflow in the main folder of the Workspace otherwise it will start a Dataflow with CI/CD support and will not give you the option to disable the preview checkbox. <<

I was switching to another Workspace and back again to get rid of the "CD/CD (preview)" forced option. But that was, in effect, simply putting me back a the root folder of the Workspace. 

Also sucks that for Dataflows, you could never create them from inside a folder. If you did, they ended up in the root folder anyway. 

Opinion: I wish Microsoft would take a thrid of the developers they have currently working on AI and CoPilot in Fabric and go back and attend to the technical debt they have amassed over the past 10 years with Power BI and stuff like this. 🙂 

[v-nuoc-msft]

Hi @ToddChitt

Thanks for your feedback.

In my test, CI/CD option is optional.

CI/CD preview features are still under development and may have some incomplete features or compatibility issues. These issues will be resolved before the official release, thank you for your understanding!

If this issue is important to you, you can create a support ticket and a dedicated Microsoft engineer will respond to you with more details.

It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.

The link of Power BI Support:  https://powerbi.microsoft.com/en-us/support/

For how to create a support ticket, please refer to How to create a support ticket in Power BI - Microsoft Fabric Community

Regards, 

Nono Chen 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly. 

[ToddChitt]

Dataflows in Fabric, while they are very powerful, are still extremely limited. Coming soon and long overdue is the ability to set a pipeline parameter from OUTSIDE the Dataflow, like in a Pipeline task that calls said Dataflow.

Currently, I must open the DF, set the Parameter value, then PUBLISH it in order for that version of the DF to be used in a Pipeline. And when I PUBLISH it, upon closing it RUNS the dataflow. So if it gets RUN, what is the sense in running it again in the Pipeline.

If, instead, I simply open the DF, set the Parameter value, then SAVE and CLOSE, when the Pipeline runs the DF, the VERSION with the LAST PUBLISHED Parameter value gets run by the Pipeline.

Third, when I do NOT Publish the DF, but edit the Parameter and save and close, if I manually click the "Refresh now" option in the elipsis context menu, The DF runs with the latest, UN-Published parameter value.

Let's assume the Parameter value is the name of a file to be processed and loaded into a Lakehouse Destination. We have a situation where you can get different results (different file processed) FROM THE SAME SAVED DATAFLOW depending on how you run it. 

This is, IMHO, absolutely HORRIBLE. It should never have been released like this. 

[ToddChitt]

So it turns out Dataflow Gen2 (CI/CD preview) do NOT have "Publish" button in the lower right corner of the design area. And it needs to be published in order for it to show up as available to a Pipeline.

Thanks Microsoft.