Forum Discussion

CloudVasu's avatar
CloudVasu
Icon for Helper I rankHelper I
10 months ago

Need help: Dynamically updating parameter values during Fabric CI/CD deployment

Hi everyone,

I have multiple environments in Microsoft Fabric, for example Dev, Test, Pre-Prod, and Prod. I’m automating deployments using Azure DevOps YAML pipelines and PowerShell scripts.

Right now, I’m deploying Data Pipelines and Notebooks from one environment to another (for example, Dev to Test to Pre-Prod to Prod) using the Fabric REST APIs. To handle workspace-to-workspace deployments, I’m using the sample script from Microsoft’s GitHub:
github.com/microsoft/fabric-samples/blob/main/features-samples/fabric-apis/DeploymentPipelines-SelectiveDeploy.ps1

I can successfully deploy notebooks and pipelines between environments, but I’m not able to update or replace parameter values dynamically. For example, I want to automatically change connection IDs, database names, or other environment-specific settings during deployment.

Is there a way to update or replace parameter values dynamically when promoting Data Pipelines between Fabric workspaces through the REST API or CI/CD?

If anyone has implemented this or knows the right approach, please help. Your guidance would be really appreciated. I'm missing my deadline and need some direction.

11 Replies

  • You need to update parameter values when deploying Data Pipelines and Notebooks across Fabric environments using Azure DevOps or REST APIs. This is a common CI/CD challenge.

    The Fabric REST API doesn't natively support parameter overrides during deployment. You need a post-deployment step to update parameters.

    Here's your approach:

    Option 1: Use Fabric REST API for Parameter Updates

    After deploying your pipeline, call the Update Item Definition API to modify parameters:

    $workspaceId = "your-workspace-id"
    $pipelineId = "your-pipeline-id"
    $bearerToken = "your-token"
    
    $body = @{
        definition = @{
            parts = @(
                @{
                    path = "pipeline-content.json"
                    payload = @{
                        activities = @(...)
                        parameters = @{
                            connectionId = @{
                                type = "String"
                                defaultValue = "prod-connection-id"
                            }
                            databaseName = @{
                                type = "String"
                                defaultValue = "ProductionDB"
                            }
                        }
                    }
                }
            )
        }
    } | ConvertTo-Json -Depth 10
    
    Invoke-RestMethod -Uri "https://api.fabric.microsoft.com/v1/workspaces/$workspaceId/items/$pipelineId/updateDefinition" -Method Post -Headers @{Authorization = "Bearer $bearerToken"} -Body $body -ContentType "application/json"
    ```
    
    **Option 2: Environment-Specific Definition Files**
    
    Store separate pipeline definition JSON files for each environment in your repo:
    ```
    /pipelines
      /dev
        pipeline-config.json
      /test
        pipeline-config.json
      /prod
        pipeline-config.json

    During deployment, reference the correct file based on your target environment variable.

    Option 3: Notebook Parameters Workaround

    For Notebooks specifically, use Azure DevOps variable groups and pass parameters at runtime:

    # In your notebook
    dbutils.widgets.text("environment", "dev")
    dbutils.widgets.text("connectionString", "default")
    
    env = dbutils.widgets.get("environment")
    connString = dbutils.widgets.get("connectionString")

    Then schedule or trigger the notebook with environment-specific parameter values through the Fabric API.

    Option 4: Post-Deployment Script

    Create a PowerShell script in your pipeline that reads environment variables and updates Fabric items:

    - task: PowerShell@2  inputs:
        targetType: 'inline'
        script: |
          $env = "$(Environment)"      $configFile = Get-Content "./config/$env.json" | ConvertFrom-Json
          
          # Update pipeline parameters via REST API
          # Call Update Definition API with environment-specific values

    The GitHub sample you referenced focuses on deployment but doesn't handle parameter substitution. You need to add a custom step after the deployment task.

    • CloudVasu's avatar
      CloudVasu
      Icon for Helper I rankHelper I

      Thanks a lot for the detailed explanation, this is really helpful.

      That makes sense now. The Fabric REST API doesn’t currently support direct parameter overrides during deployment, so adding a post-deployment step to update pipeline definitions using the updateDefinition API looks like the right approach.

      I’ll try implementing Option 1 (calling the Update Definition API after deployment) in my Azure DevOps YAML pipeline to dynamically replace connection IDs and other environment-specific parameters.

      Also, good point about maintaining separate config JSON files or using environment variables for notebooks. That gives us flexibility for future scaling.

      I really appreciate the clarification, this clears up a blocker we were facing. I’ll implement this approach and share my updates here once tested

    • CloudVasu's avatar
      CloudVasu
      Icon for Helper I rankHelper I

      Thank you, Kevin. When I deploy using a Fabric token, I can successfully deploy both pipelines and notebooks. However, when I use a service principal, I’m only able to deploy notebooks, and pipeline deployment fails with the message: Deployment status: Failed. I’ll focus on resolving this issue first, and then I’ll use the Fabric-CICD Python library as an additional layer, as you suggested.

  • v-prasare's avatar
    v-prasare
    Icon for Community Support rankCommunity Support

    Hi CloudVasu,

    We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.


    KevinChant & MJParikh ,Thanks for your prompt response

     

     

     

    Thank you for your patience and look forward to hearing from you.
    Best Regards,
    Prashanth Are
    MS Fabric community support

    • CloudVasu's avatar
      CloudVasu
      Icon for Helper I rankHelper I

      Thank you for the prompt response from the community. I appreciate the guidance provided by Kevinchant and mjparikh. Due to other issues, I have not yet implemented their suggestions, but I plan to do so shortly and will share an update here once completed.

  • v-prasare's avatar
    v-prasare
    Icon for Community Support rankCommunity Support

    Hi CloudVasu,

    We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

     

     

    Thank you for your patience and look forward to hearing from you.
    Best Regards,
    Prashanth Are
    MS Fabric community support

    • CloudVasu's avatar
      CloudVasu
      Icon for Helper I rankHelper I

      Hi Prashanth,

      I just wanted to let you know that I don’t have any questions on this topic at the moment. If I need any support in the future, I’ll reach out to the community.

      I really appreciate the help and guidance provided by the Fabric community.it’s been very valuable.

    • CloudVasu's avatar
      CloudVasu
      Icon for Helper I rankHelper I

      Thank you for the prompt response from the community. I appreciate the guidance provided by Kevinchant and mjparikh. Due to other issues, I have not yet implemented their suggestions, but I plan to do so shortly and will share an update here once completed.

      • v-prasare's avatar
        v-prasare
        Icon for Community Support rankCommunity Support

        CloudVasu Thanks for the response. please do keep posted updates here

         

        thanks,

        prashanth