Forum Discussion
How to automatically switch database connections when deploying Data Pipeline from Dev to QA
Body:
Hello Community,
I am working with Microsoft Fabric workspaces and facing an issue during deployment.
Scenario:
I deployed a workspace from Dev to QA.
The workspace includes a Data Pipeline.
After deployment, the pipeline’s Lookup activity and the pipeline database connection are still pointing to the Dev database connection.
Problem:
Currently, I have to manually update the database connection in the Lookup activity and Data Pipeline after every deployment.
Expectation:
I would like to configure Deployment Rules (or any other supported mechanism) so that during deployment, the pipeline and lookup automatically point to the correct QA database connection instead of Dev.
Question:
Is there a supported way to parameterize database connections in Data Pipelines during workspace deployment?
Or is this a current limitation of Fabric Deployment Rules?
Any guidance or workaround will be very helpful.
Labels: Data Factory, Administration, General Discussion
Hi DarpanP ,
Thank you for reaching out to Microsoft Community.Microsoft Fabric's current CI/CD deployment process presents a challenge when it comes to environment-specific connections in data pipelines and Lookup activities. Unlike semantic models, dataflows (Gen1), and reports, which are supported by Fabric’s deployment rules, pipelines and Lookup activities retain their original connection GUIDs after deployment. This means that even if a resource with the same name exists in the QA or Prod environment, the pipeline will still point to the Dev resource unless manually updated.
You can try this workaround to use the new Variable Libraries feature, which is currently in preview. This allows you to define configuration variables such as connection object IDs and database names with default values for Dev and alternate values for QA or Prod. Within your pipeline, you can reference these variables using dynamic expressions under “Dynamic Content.” When the pipeline runs in the target workspace, the active value set from the Variable Library ensures the correct GUID is used. However, keep in mind that this method only supports GUID-based parameterization, and data preview is disabled for parameterized activities.
Reference: Variable library integration with Data pipelines (Preview) - Microsoft Fabric | Microsoft Learn
Parameterization - fabric-cicdAnother option is to use the open-source Fabric-cicd deployment tool. This tool enables automated substitution of connection GUIDs by defining mappings in a parameter.yml file. You specify the find_key path in the pipeline JSON that points to the connection GUID, and then map it to environment-specific GUIDs for QA or Prod. During deployment, Fabric-cicd updates the pipeline JSON accordingly before pushing it to the target workspace. This approach is reliable and automates the GUID swapping process, though it does require managing pipeline files outside the Fabric UI.
Both methods offer viable solutions depending on your workflow preferences and tolerance for preview limitations or external tooling.
For more details on deployment rules, refer to Microsoft’s official documentation Create deployment rules for Fabric's ALM - Microsoft Fabric | Microsoft Learn.Hope this helps.
Warm Regards,
Chaithra E.Hello DarpanP,
It is absolutely possible to parameterize connection strings to automatically switch between your development (Dev) and testing (QA/Test) environments. I use Variable Libraries within Fabric. You have first to enable in the admin portal on Microsoft Fabric because it's in Preview mode.
The official Microsoft documentation confirms this approach. By using a variable library, you can define variables for each environment (for example, Dev_Database_Connection_String and QA_Database_Connection_String) and link them to the parameters of your data pipeline activities.
Important point: There is one manual step you must not forget. After deploying to the test environment, you need to make sure to set the test environment's variable library as the active library.
The documentation detailing this process is available here: https://learn.microsoft.com/en-us/fabric/cicd/variable-library/get-started-variable-libraries?tabs=home-page
In short, you no longer need to manually update the connection after each deployment. This is a key feature of Fabric for automating and securing your deployment processes.
Hope it can help you !Best regards,Antoine
8 Replies
Hello DarpanP,
have you already looked at the Microsoft documentation on deployment rules?
https://learn.microsoft.com/en-us/fabric/cicd/deployment-pipelines/create-rules?tabs=new-ui
This is also explained again in a video by Guy in a Cube.
https://youtu.be/mXngl2ChBxM?si=txpVnv6fjPiUXT69
I hope that helps you for now.
Best regards
Hi DarpanP ,
Thank you for reaching out to Microsoft Community.Microsoft Fabric's current CI/CD deployment process presents a challenge when it comes to environment-specific connections in data pipelines and Lookup activities. Unlike semantic models, dataflows (Gen1), and reports, which are supported by Fabric’s deployment rules, pipelines and Lookup activities retain their original connection GUIDs after deployment. This means that even if a resource with the same name exists in the QA or Prod environment, the pipeline will still point to the Dev resource unless manually updated.
You can try this workaround to use the new Variable Libraries feature, which is currently in preview. This allows you to define configuration variables such as connection object IDs and database names with default values for Dev and alternate values for QA or Prod. Within your pipeline, you can reference these variables using dynamic expressions under “Dynamic Content.” When the pipeline runs in the target workspace, the active value set from the Variable Library ensures the correct GUID is used. However, keep in mind that this method only supports GUID-based parameterization, and data preview is disabled for parameterized activities.
Reference: Variable library integration with Data pipelines (Preview) - Microsoft Fabric | Microsoft Learn
Parameterization - fabric-cicdAnother option is to use the open-source Fabric-cicd deployment tool. This tool enables automated substitution of connection GUIDs by defining mappings in a parameter.yml file. You specify the find_key path in the pipeline JSON that points to the connection GUID, and then map it to environment-specific GUIDs for QA or Prod. During deployment, Fabric-cicd updates the pipeline JSON accordingly before pushing it to the target workspace. This approach is reliable and automates the GUID swapping process, though it does require managing pipeline files outside the Fabric UI.
Both methods offer viable solutions depending on your workflow preferences and tolerance for preview limitations or external tooling.
For more details on deployment rules, refer to Microsoft’s official documentation Create deployment rules for Fabric's ALM - Microsoft Fabric | Microsoft Learn.Hope this helps.
Warm Regards,
Chaithra E.Hi DarpanP ,
I hope the information provided is helpful.I wanted to check whether you were able to resolve the issue with the provided solutions.Please let us know if you need any further assistance.
Thank you.
Solution:
Parameterize database connections in Linked Services (e.g., ServerName, DatabaseName).Pass these parameters from the pipeline.
Use Deployment Rules to set environment-specific values for QA, Dev, Prod.
Alternative: Create separate Linked Services for each environment and make the pipeline pick the correct one via a parameter.
Key point: Without parameterization, Fabric requires manual updates—automatic switching isn’t supported.
Hi DarpanP ,
We’d like to follow up regarding the recent concern. Kindly confirm whether the issue has been resolved, or if further assistance is still required. We are available to support you and are committed to helping you reach a resolution.
Best Regards,
Chaithra E.Hello DarpanP,
It is absolutely possible to parameterize connection strings to automatically switch between your development (Dev) and testing (QA/Test) environments. I use Variable Libraries within Fabric. You have first to enable in the admin portal on Microsoft Fabric because it's in Preview mode.
The official Microsoft documentation confirms this approach. By using a variable library, you can define variables for each environment (for example, Dev_Database_Connection_String and QA_Database_Connection_String) and link them to the parameters of your data pipeline activities.
Important point: There is one manual step you must not forget. After deploying to the test environment, you need to make sure to set the test environment's variable library as the active library.
The documentation detailing this process is available here: https://learn.microsoft.com/en-us/fabric/cicd/variable-library/get-started-variable-libraries?tabs=home-page
In short, you no longer need to manually update the connection after each deployment. This is a key feature of Fabric for automating and securing your deployment processes.
Hope it can help you !Best regards,AntoineHi DarpanP ,
May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.
Thank you.
While deploying code from Dev to Prod, I encountered the same issue — the connection was being deployed and still pointing to the Dev environment from the Prod workspace.
Below is a simple workaround I implemented for the stored procedure and lookup activity.
First Create - Library Variable
warehouse_id - Library Variablewarehouse_sql_end - Library Variable
System Variableworkspace_id : System Variable
Lookup:
Connection - (none)
Connection Type: Warehouse/Lakehouse
Warehouse: warehouse_id - Library Variable
Wokrspace ID : Workspace_id : System Variable
SQL Connection String : warehouse_sql_end - Library Variable
Choose object : Table/Query/Store ProcedureStore Procedure:
Connection (Warehouse ID): GUID Here I am Using Warehouse ID from Library Variable
Connection Type: WarehouseSQL connection name : warehouse_sql_end - Library Variable
Wokrspace ID : Workspace_id : System Variable
This approach worked well for me, and the code has been successfully deployed to production.
Thank You,
Jagdish
