Forum Discussion
Re: 'The dataflow has never been published.' error when using 'Save as Dataflow Gen2 (CI/CD)' featur
I, too, am having issues with Dataflow Gen2 (CI/CD).
I created one with a Parameter, and referenced that paramter in one of the PowerQuery steps. It works. There is a Data Destination to a Warehouse table. (Source is a Fabric SQL Database).
The refresh from the Workspace browser, while it still does not indicate any run or refreshed status, did in fact populate the table as expected.
The issue comes when I try to run the Dataflow from a Pipeline and specify the value of the Dataflow parameters in the section that now becomes available because the Dataflow is CI/CD.
But I get an error with little information:
"User configuration issue". What am I doing wrong? The user documentation Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn does NOT have mention that you cannot call it from a Pipeline.
I can call it from a Pipeline and it runs, but if I try to set the Parameter in the Pipeline, it fails.
Frustrated that Microsoft would release something to production so half-baked.
Hi ToddChitt !
I have taken the liberty of moving the message that you've created to it's own topic so we can address your inquiry which is quite specific to a new feature that we recently introduced. This feature is Dataflow parameters and this feature is currently in public preview. To enable your Dataflow to accept parameters, you need to enable this capability within your Dataflow. You can read more about this setting and the feature from the documentation below:https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-parameters
Could you please confirm if the setting was enabled in the parameters section of the options dialog?
once you enable it, you should be able to pass parameter values to your Dataflow.
Edit: Forgot to mention that we're currently working on improving this experience on the pipelines side as well as the error messages that may come from it.
6 Replies
Hi ToddChitt !
I have taken the liberty of moving the message that you've created to it's own topic so we can address your inquiry which is quite specific to a new feature that we recently introduced. This feature is Dataflow parameters and this feature is currently in public preview. To enable your Dataflow to accept parameters, you need to enable this capability within your Dataflow. You can read more about this setting and the feature from the documentation below:https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-parameters
Could you please confirm if the setting was enabled in the parameters section of the options dialog?
once you enable it, you should be able to pass parameter values to your Dataflow.
Edit: Forgot to mention that we're currently working on improving this experience on the pipelines side as well as the error messages that may come from it.
Thanks. I did find the solution last night after some research, and it is exactly in line with what you suggested, so it is marked at a Solution.
From this site: Use public parameters in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn
There is this comment in the Considerations and Limitations:
- Parameters that affect the resource path of a data source or a destination are not supported. Connections are linked to the exact data source path defined in the authored dataflow and can't be currently override to use other connections or resource paths.
Any idea if that will ever change? Is this a temporary limitation?
Also, when you create a new Dataflow Gen2 (CI/CD), it does NOT have the "(preview)" qualifying text anymore. Is it considered GA?
And finally, there is a known bug where by if you want to create a original Dataflow Gen2 (i.e.: NOT CI/CD) then you need to do it from the root folder of the Fabric Workspace. If you do it from a Folder, it defaults to CI/CD and the check-box is disabled:
(Commentary: Still a ways to go before this is of any use.)
Q: Any idea if that will ever change? Is this a temporary limitation?
A: This is inherintly a limitation of Dataflows today when trying to deal with a concept of dynamic connections. We're working on a plan to enable a Dynamic connection in Dataflows, but it all boils down to how in the querymetadata.json there's a binding happening between the resourcePath for your data source and the connectionId that needs to be used for it.
Q: Also, when you create a new Dataflow Gen2 (CI/CD), it does NOT have the "(preview)" qualifying text anymore. Is it considered GA?
A: This is correct. Dataflow Gen2 with CI/CD capabilities are now generally available. You can read more about this from the blog post below:
Dataflow Gen2 CI/CD, GIT integration and Public APIs (Generally Available) | Microsoft Fabric Blog | Microsoft FabricQ: And finally, there is a known bug where by if you want to create a original Dataflow Gen2 (i.e.: NOT CI/CD) then you need to do it from the root folder of the Fabric Workspace. If you do it from a Folder, it defaults to CI/CD and the check-box is disabled
A: This is currently by design. The new Dataflow Gen2 with CI/CD support provide this missing functionality as well as many other such as REST API support, CI/CD and ALM support as well as many other benefits. We're working towards a plan to consolidate all versions of Dataflow Gen2 into a single one that has all the capabilities that Fabric supports. No ETA on when this will happen just yet, but we're working on it and we recommend using the CI/CD version of Dataflow Gen2 - specially now that its GA.
Q: And finally, there is a known bug where by if you want to create a original Dataflow Gen2 (i.e.: NOT CI/CD) then you need to do it from the root folder of the Fabric Workspace. If you do it from a Folder, it defaults to CI/CD and the check-box is disabled
A: This is currently by design.
Really? (and pardon me for having an opinion about this) This this is a VERY POOR DESIGN. I have seen this stock answer all over Microsoft forums and I just have to shake my head at it.
When it was in preview, I didn't want to use it, and it took HOURS for me to figure out how to NOT use it.
Maybe default the check-box to TRUE, but for goodness sake, ENABLE it for the users!
Is there a migration option planned, whereby I can migrate all of my Dataflow Gen2 artifacts to CI/CD version? If NOT, then I certainly don't want to have a MIX of CI/CD enabled and not in my workspaces.
Make it easy for the users.
- Parameters that affect the resource path of a data source or a destination are not supported. Connections are linked to the exact data source path defined in the authored dataflow and can't be currently override to use other connections or resource paths.
