How to update data destination lakehouse of a dataflow gen2 CI/CD?

[JKL]

When I deploy a dataflow gen2 CI/CD dataflow from development to test in the deployment pipeline it keeps the lakehouse from development as destination. 

It is not possible to set up a deploymentrule to dataflow gen 2's in general.

My solution for now is to go into each dataflow gen2 in test and change data destination manual.

Another issue based on this is that in the deployment pipeline this dataflow gen2 CI/CD will differs from development to test - since the data destination has been changed manually..

How do I get arround this issue? Anyone figuring out a way to easily change all data destination for a workspace instead of each dataflow?

Maybe with CLI or variable library? These are just completely new features and I'm not sure where to start.

[v-sathmakuri]

Hi @JKL ,

I hope this information provided is helpful. Feel free to reach out if you have any further questions or would like to discuss this in more detail. If responses provided answers your question, please accept it as a solution so other community members with similar problems can find a solution faster.

Thank you!!

[v-sathmakuri]

Hi @JKL ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If the responses has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you!!

[v-sathmakuri]

Hi @JKL ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-sathmakuri]

Hi @JKL ,

Thank you for reaching out to Microsoft Fabric Community.

After deployment, automate updating the sink (destination) using the Fabric REST API. Use PowerShell or Python Script to List all Dataflow Gen2s in the Test workspace and PATCH each one to point to the correct sink Lakehouse. Below is the sample powershell script.

# Variables
$workspaceId = "<your-test-workspace-id>"
$targetLakehouseId = "<your-test-lakehouse-id>"
$token = "<your-access-token>"  # Get this via OAuth2 or Service Principal
$apiUrl = "https://api.fabric.microsoft.com"

# List all Dataflows Gen2
$dataflows = Invoke-RestMethod -Method Get -Uri "$apiUrl/v1/workspaces/$workspaceId/dataflows" -Headers @{ Authorization = "Bearer $token" }

foreach ($df in $dataflows.value) {
    $dataflowId = $df.id

    # Prepare the sink update payload
    $payload = @{
        sink = @{
            lakehouse = @{
                id = $targetLakehouseId
            }
        }
    } | ConvertTo-Json -Depth 10

    # Update sink
    Invoke-RestMethod -Method Patch -Uri "$apiUrl/v1/workspaces/$workspaceId/dataflows/$dataflowId" `
        -Headers @{ Authorization = "Bearer $token"; "Content-Type" = "application/json" } `
        -Body $payload

    Write-Host "Updated sink for Dataflow $($df.name)"
}

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

Thank you!!

[Adriansm]

Hi

Thanks for the proposed solution. I have tried to do this with DFG2 CI/CD, but I'm getting the error below. Is that because this option is not available for CI/CD Dataflows?

Is there any resource that explains in full detail the available options in the API? Thank you

    "errorCode": "InvalidInput",

    "moreDetails": [

        {

            "errorCode": "InvalidParameter",

            "message": "UpdateArtifactRequest should have at least one valid field to update"

        }

    ],

    "message": "The request has an invalid input",

[JKL]

Thanks for the input. 

But I'm looking into how it can be done with CLI.
Did some testing with Library Variable and figured out, that will not help us solve our issue. But still hopefull for CLI.

[v-sathmakuri · Accepted solution]

Hi @JKL ,

Microsoft Fabric's CLI does not currently support direct updates to Dataflow Gen2 destinations using CLI.The CLI is primarily designed for workspace management, dataset operations, and pipeline orchestration, but it lacks specific commands for modifying Dataflow Gen2 configurations, such as changing sink destinations.

As previously suggested, a viable approach is to use the Fabric REST API to automate the process of updating the sink (destination) for Dataflow Gen2 instances.

Microsoft is actively working on enhancing the capabilities of the Fabric CLI.While there is no official timeline for when support for updating Dataflow Gen2 configurations via the CLI will be available.

I recommend submitting your detailed feedback and ideas through Microsoft's official feedback channels, such as the Microsoft Fabric Ideas. 

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

Thank you!!