Question about using data flow gen2 (ci/cd) in pipelines as parameters

[Filipbukvic96]

I want to use dataflow gen2 (ci/cd) as a dynamic value parameter in pipelines but there is no way to forward any kind of unique identifier of a dataflow gen2 (ci/cd). With standard dataflows gen2 I would usually put GUID values, but there are no GUID values for dataflows gen2 (ci/cd), not even in json files when they are synced to git. What would be the alternative to get their GUIDs and use them as dynamic values in pipelines if there is one.

[v-agajavelly]

Hi @Filipbukvic96 ,

Sorry for the delay in getting back to you. I went through the issue in detail, and the main reason you’re hitting those errors is that Dataflows Gen2 (CI/CD) behave differently compared to standard Gen2 dataflows. When they are Git-linked, the artifact in the workspace is basically read-only, so Fabric APIs can’t refresh them directly that’s why you keep getting InvalidRequest or UnknownException errors.

Here’s the solution that works for you.

1.Deploy the CI/CD Dataflow into Live Mode

Use Deployment Pipelines in Fabric to push the dataflow from the Git branch into the target workspace.

Once deployed, the dataflow becomes a workspace-level artifact, and refresh works as expected.

2.Get the GUID dynamically

Continue using the Fabric REST API:

GET https://api.fabric.microsoft.com/v1/dataflows

Filter by name, and grab the id. Trigger the refresh successfully

Use the refresh endpoint.

POST https://api.fabric.microsoft.com/v1/dataflows/{dataflowId}/refreshes

This works for the deployed (live) dataflow, not the Git-linked one.

Why this is the only stable option right now.
Git-linked dataflows can’t be refreshed via API or pipeline because the workspace version is just a mirror from Git. Deploying it creates a real artifact that the API can work with.

Regards,
Akhil.

[v-agajavelly]

Hi @Filipbukvic96 ,

Just checking were you able to get the pipeline running end-to-end after pulling the Dataflow Gen2 ID dynamically? If it’s working smoothly now, that’s great news. If you’re hitting any snags in passing the parameter or triggering the run, we can troubleshoot the exact step together.

Regards,
Akhil.

[Filipbukvic96]

UPDATE 2: I also tried with FABRIC API request, and when I trigger the refresh they always fail with the same message:  "errorCode": "UnknownException", "message": "Something went wrong, please try again later. If the error persists, please contact support."

[Filipbukvic96]

UPDATE: I have now managed to get the dataflow GUIDs, but still, passing this GUID values to pipeline and refreshing the pipeline does not refresh the Dataflows automatically, I get the error: Refresh Dataflow failed with status: BadRequest, Failure reason: {"error":{"code":"InvalidRequest","message":"Unexpected dataflow error: "}}

[Filipbukvic96]

@v-agajavelly Not really, I have tried everything but I keep on getting the following: 

Invoke-RestMethod : {"requestId":"8cdaf62f-ab9a-412d-9c88-a5d931987e26","errorCode":"Unauthorized","message":"The calle
r is not authenticated to access this resource"}
At line:28 char:14
+ ... Dataflows = Invoke-RestMethod -Method Get -Uri $ListDataflowsUri -Hea ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc
   eption
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

How come there is not any other faster way to retrieve Dataflow GUIDs for Dataflows Gen2 (CI/CD), with standard Gen2 it works fine, I would just like to avoid redoing my Dataflows from CI/CD ones into standard Gen2 

[v-agajavelly]

Hi @Filipbukvic96 ,

Thanks  @ibarrau for your inputs. Yes, using the Fabric REST API to list the dataflows and filter by name works perfectly for getting the GUID of a Dataflow Gen2 (CI/CD). I was able to retrieve the ID and pass it as a dynamic parameter in my pipeline. Appreciate the clear pointer to the docs.

Thanks,
Akhil.

[ibarrau]

Hi. I'm not quite sure what is it that you want to do, but if you want to get the GUID for a dataflow gen2 at a workspace, you could use Fabric Rest API. This is the request for getting them (then filter by name):
https://learn.microsoft.com/en-us/rest/api/fabric/dataflow/items/list-dataflows?tabs=HTTP

I hope that helps,