Unable to run Dataflow Gen 2 (CI/CD) by giving Workspace ID and Dataflow ID in the Dataflow Activity

[DreamNattapat]

Does anyone has a solution to this yet?

Problem: Unable to run Dataflow Gen 2 (CI/CD) by giving Workspace ID and Dataflow ID in the Dataflow Activity

To Reproduce:

1. I have a Dataflow Gen 2 (CI/CD) in a workspace with the IDs as shown below

2. I specified the Workspace ID and Dataflow ID in the settings of the Dataflow Activity

3. After running, it showed me the Error Code 20302

However, if I select an option from the dropdown instead of specifying the IDs, it is able to run. However, this does not achieve the purpose of CI/CD, where we need to utilize dynamic content.

[bhavya5903]

Hi @DreamNattapat  ,

This is currently a limitation in Microsoft Fabric, we are unable to trigger or parameterize a Dataflow Gen2 (CI/CD) using Workspace ID and Dataflow ID directly in the Dataflow activity within a pipeline.

As of now, dynamic connections for Dataflow Gen2 in pipeline activities are not supported.

We hope to see improvements in future updates.

[DreamNattapat]

Thank you for the clarification. Noted.

[v-kpoloju-msft]

Hi @DreamNattapat,

Thank you for reaching out to the Microsoft fabric community forum. Also, thanks to  @NandanHegde, for his inputs on this thread. Here I provided workaround below that might be helpful to you to resolve the issue. Please go through to it.

Based on the error you are encountering (Error Code: 20002, InvalidRequest, "Unexpected dataflow error") and the setup you have described (manually entering Workspace ID and Dataflow ID for a Gen2 Dataflow in the pipeline activity), this appears to be related to a known limitation with Dataflow Gen2 and CI/CD integration in Fabric.

Currently, the Dataflow Gen2 activity in Fabric pipelines expects a metadata-bound configuration. When you manually provide the Workspace ID and Dataflow ID (instead of selecting from the dropdown), the pipeline may not be able to resolve the necessary metadata behind the scenes, which leads to a failure at runtime.

This is a documented limitation in the official Microsoft documentation here:
Dataflow Gen2 CI/CD and Git integration – Known Issues

Workaround is: For CI/CD scenarios where dynamic execution is required, we recommend using a Web activity to trigger the Dataflow Gen2 refresh via REST API, which gives you full control over Workspace/Dataflow IDs using parameters.

Also, please refer to the below mentioned links for better understanding:
Dataflow activity - Microsoft Fabric | Microsoft Learn
Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn

Hope this helps clarify things and let me know what you find after giving these steps a try happy to help you investigate this further.

Thank you for using the Microsoft Community Forum.

[DreamNattapat]

Thank you for the workaround. Can you please determine what is the exact REST API do we need to trigger the Dataflow Gen2 ? because I found only this link but it does not work for my case.

(Dataflows - Refresh Dataflow: https://learn.microsoft.com/en-us/rest/api/power-bi/dataflows/refresh-dataflow)

[v-kpoloju-msft]

Hi @DreamNattapat,

Thanks for the follow-up. Also, mentioned by the point @bhavya5903, is valid for this thread. You are right that the link you shared refers to the legacy Dataflow (Gen1) API. For Dataflow Gen2, the REST API is different and currently not documented under the standard Power BI REST API.

To programmatically trigger a refresh for Dataflow Gen2, you will typically need to interact with the Fabric API surface, and as of now, public endpoints for Gen2 refresh via REST API might not be fully available or supported like Gen1.

Workaround is: If your scenario involves scheduling or automation, consider using Power Automate with Microsoft Fabric Dataflow Gen2 connector, or if you are using a pipeline, you can trigger the dataflow as an activity step in a Fabric pipeline.

Optional Additions: If the OP is using Microsoft Fabric Pipelines, add: Within Fabric, you can add the Dataflow Gen2 as a pipeline activity and configure triggers, accordingly, avoiding the need for a separate REST call.

If REST API is a must: Now, a direct REST API endpoint for Gen2 refresh is not officially available in the Power BI REST API reference. It's possible that this functionality is still rolling out or may require additional Microsoft Fabric workspace configuration.

Hope this helps clarify things and let me know what you find after giving these steps a try happy to help you investigate this further.

Thank you for using the Microsoft Fabric Community Forum.

[v-kpoloju-msft]

Hi @DreamNattapat,

Just checking in to see if the issue has been resolved on your end. If the earlier suggestions helped, that’s great to hear! And if you’re still facing challenges, feel free to share more details happy to assist further.

Thank you.

[v-kpoloju-msft]

Hi @DreamNattapat,

Hope you had a chance to try out the solution shared earlier. Let us know if anything needs further clarification or if there's an update from your side always here to help.

Thank you.

[v-kpoloju-msft]

Hi @DreamNattapat,

Just wanted to follow up one last time. If the shared guidance worked for you, that’s wonderful hopefully it also helps others looking for similar answers. If there’s anything else you'd like to explore or clarify, don’t hesitate to reach out.

Thank you.

[NandanHegde]

Try creating 2 variables and add those values in the variables and try calling those variables via dynamic content in the workpsace and dataflow expressions and confirm whether you are able to execute it now.

Also before that, check the input of the activity via the harcoded version and another via the one which you are trying with ID.(something might be diff)

[DreamNattapat]

Thank you for the suggestion. Unfortunately, it does not work as it produced the same error.