Forum Discussion
How to automatically switch database connections when deploying Data Pipeline from Dev to QA
Hi DarpanP ,
Thank you for reaching out to Microsoft Community.Microsoft Fabric's current CI/CD deployment process presents a challenge when it comes to environment-specific connections in data pipelines and Lookup activities. Unlike semantic models, dataflows (Gen1), and reports, which are supported by Fabric’s deployment rules, pipelines and Lookup activities retain their original connection GUIDs after deployment. This means that even if a resource with the same name exists in the QA or Prod environment, the pipeline will still point to the Dev resource unless manually updated.
You can try this workaround to use the new Variable Libraries feature, which is currently in preview. This allows you to define configuration variables such as connection object IDs and database names with default values for Dev and alternate values for QA or Prod. Within your pipeline, you can reference these variables using dynamic expressions under “Dynamic Content.” When the pipeline runs in the target workspace, the active value set from the Variable Library ensures the correct GUID is used. However, keep in mind that this method only supports GUID-based parameterization, and data preview is disabled for parameterized activities.
Reference: Variable library integration with Data pipelines (Preview) - Microsoft Fabric | Microsoft Learn
Parameterization - fabric-cicdAnother option is to use the open-source Fabric-cicd deployment tool. This tool enables automated substitution of connection GUIDs by defining mappings in a parameter.yml file. You specify the find_key path in the pipeline JSON that points to the connection GUID, and then map it to environment-specific GUIDs for QA or Prod. During deployment, Fabric-cicd updates the pipeline JSON accordingly before pushing it to the target workspace. This approach is reliable and automates the GUID swapping process, though it does require managing pipeline files outside the Fabric UI.
Both methods offer viable solutions depending on your workflow preferences and tolerance for preview limitations or external tooling.
For more details on deployment rules, refer to Microsoft’s official documentation Create deployment rules for Fabric's ALM - Microsoft Fabric | Microsoft Learn.Hope this helps.
Warm Regards,
Chaithra E.Hello DarpanP,
It is absolutely possible to parameterize connection strings to automatically switch between your development (Dev) and testing (QA/Test) environments. I use Variable Libraries within Fabric. You have first to enable in the admin portal on Microsoft Fabric because it's in Preview mode.
The official Microsoft documentation confirms this approach. By using a variable library, you can define variables for each environment (for example, Dev_Database_Connection_String and QA_Database_Connection_String) and link them to the parameters of your data pipeline activities.
Important point: There is one manual step you must not forget. After deploying to the test environment, you need to make sure to set the test environment's variable library as the active library.
The documentation detailing this process is available here: https://learn.microsoft.com/en-us/fabric/cicd/variable-library/get-started-variable-libraries?tabs=home-page
In short, you no longer need to manually update the connection after each deployment. This is a key feature of Fabric for automating and securing your deployment processes.
Hope it can help you !Best regards,Antoine
While deploying code from Dev to Prod, I encountered the same issue — the connection was being deployed and still pointing to the Dev environment from the Prod workspace.
Below is a simple workaround I implemented for the stored procedure and lookup activity.
First Create - Library Variable
warehouse_id - Library Variable
warehouse_sql_end - Library Variable
System Variable
workspace_id : System Variable
Lookup:
Connection - (none)
Connection Type: Warehouse/Lakehouse
Warehouse: warehouse_id - Library Variable
Wokrspace ID : Workspace_id : System Variable
SQL Connection String : warehouse_sql_end - Library Variable
Choose object : Table/Query/Store Procedure
Store Procedure:
Connection (Warehouse ID): GUID Here I am Using Warehouse ID from Library Variable
Connection Type: Warehouse
SQL connection name : warehouse_sql_end - Library Variable
Wokrspace ID : Workspace_id : System Variable
This approach worked well for me, and the code has been successfully deployed to production.
Thank You,
Jagdish