Forum Discussion
Need help: Dynamically updating parameter values during Fabric CI/CD deployment
Hi everyone,
I have multiple environments in Microsoft Fabric, for example Dev, Test, Pre-Prod, and Prod. I’m automating deployments using Azure DevOps YAML pipelines and PowerShell scripts.
Right now, I’m deploying Data Pipelines and Notebooks from one environment to another (for example, Dev to Test to Pre-Prod to Prod) using the Fabric REST APIs. To handle workspace-to-workspace deployments, I’m using the sample script from Microsoft’s GitHub:
github.com/microsoft/fabric-samples/blob/main/features-samples/fabric-apis/DeploymentPipelines-SelectiveDeploy.ps1
I can successfully deploy notebooks and pipelines between environments, but I’m not able to update or replace parameter values dynamically. For example, I want to automatically change connection IDs, database names, or other environment-specific settings during deployment.
Is there a way to update or replace parameter values dynamically when promoting Data Pipelines between Fabric workspaces through the REST API or CI/CD?
If anyone has implemented this or knows the right approach, please help. Your guidance would be really appreciated. I'm missing my deadline and need some direction.
11 Replies
- MJParikh
Super User
You need to update parameter values when deploying Data Pipelines and Notebooks across Fabric environments using Azure DevOps or REST APIs. This is a common CI/CD challenge.
The Fabric REST API doesn't natively support parameter overrides during deployment. You need a post-deployment step to update parameters.
Here's your approach:
Option 1: Use Fabric REST API for Parameter Updates
After deploying your pipeline, call the Update Item Definition API to modify parameters:
$workspaceId = "your-workspace-id" $pipelineId = "your-pipeline-id" $bearerToken = "your-token" $body = @{ definition = @{ parts = @( @{ path = "pipeline-content.json" payload = @{ activities = @(...) parameters = @{ connectionId = @{ type = "String" defaultValue = "prod-connection-id" } databaseName = @{ type = "String" defaultValue = "ProductionDB" } } } } ) } } | ConvertTo-Json -Depth 10 Invoke-RestMethod -Uri "https://api.fabric.microsoft.com/v1/workspaces/$workspaceId/items/$pipelineId/updateDefinition" -Method Post -Headers @{Authorization = "Bearer $bearerToken"} -Body $body -ContentType "application/json" ``` **Option 2: Environment-Specific Definition Files** Store separate pipeline definition JSON files for each environment in your repo: ``` /pipelines /dev pipeline-config.json /test pipeline-config.json /prod pipeline-config.jsonDuring deployment, reference the correct file based on your target environment variable.
Option 3: Notebook Parameters Workaround
For Notebooks specifically, use Azure DevOps variable groups and pass parameters at runtime:
# In your notebook dbutils.widgets.text("environment", "dev") dbutils.widgets.text("connectionString", "default") env = dbutils.widgets.get("environment") connString = dbutils.widgets.get("connectionString")Then schedule or trigger the notebook with environment-specific parameter values through the Fabric API.
Option 4: Post-Deployment Script
Create a PowerShell script in your pipeline that reads environment variables and updates Fabric items:
- task: PowerShell@2 inputs: targetType: 'inline' script: | $env = "$(Environment)" $configFile = Get-Content "./config/$env.json" | ConvertFrom-Json # Update pipeline parameters via REST API # Call Update Definition API with environment-specific values
The GitHub sample you referenced focuses on deployment but doesn't handle parameter substitution. You need to add a custom step after the deployment task.
- CloudVasu
Helper I
Thanks a lot for the detailed explanation, this is really helpful.
That makes sense now. The Fabric REST API doesn’t currently support direct parameter overrides during deployment, so adding a post-deployment step to update pipeline definitions using the updateDefinition API looks like the right approach.
I’ll try implementing Option 1 (calling the Update Definition API after deployment) in my Azure DevOps YAML pipeline to dynamically replace connection IDs and other environment-specific parameters.
Also, good point about maintaining separate config JSON files or using environment variables for notebooks. That gives us flexibility for future scaling.
I really appreciate the clarification, this clears up a blocker we were facing. I’ll implement this approach and share my updates here once tested
- KevinChant
Super User
If you need to do a lot of parameter replacement I strongly recommennd looking to use the fabric-cicd Pythin library as a layer of abstraction instead of working with the API's directly.
https://microsoft.github.io/fabric-cicd/0.1.30/- CloudVasu
Helper I
Thank you, Kevin. When I deploy using a Fabric token, I can successfully deploy both pipelines and notebooks. However, when I use a service principal, I’m only able to deploy notebooks, and pipeline deployment fails with the message: Deployment status: Failed. I’ll focus on resolving this issue first, and then I’ll use the Fabric-CICD Python library as an additional layer, as you suggested.
- v-prasare
Community Support
Hi CloudVasu,
We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.
KevinChant & MJParikh ,Thanks for your prompt responseThank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support- CloudVasu
Helper I
Thank you for the prompt response from the community. I appreciate the guidance provided by Kevinchant and mjparikh. Due to other issues, I have not yet implemented their suggestions, but I plan to do so shortly and will share an update here once completed.
- v-prasare
Community Support
Hi CloudVasu,
We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.
Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support- CloudVasu
Helper I
Hi Prashanth,
I just wanted to let you know that I don’t have any questions on this topic at the moment. If I need any support in the future, I’ll reach out to the community.
I really appreciate the help and guidance provided by the Fabric community.it’s been very valuable.
- v-prasare
Community Support
Thanks for your prompt response here.
- CloudVasu
Helper I
Thank you for the prompt response from the community. I appreciate the guidance provided by Kevinchant and mjparikh. Due to other issues, I have not yet implemented their suggestions, but I plan to do so shortly and will share an update here once completed.
- v-prasare
Community Support