Forum Discussion

SørenBrandt's avatar
SørenBrandt
Frequent Visitor
1 year ago
Solved

How to deploy Mirrored Azure Databricks Catalog to another workspace

Dear all, In a development workspace, I have a Mirrored Azure Databricks Catalog. I would like to roll out the contents of the workspace to many customer specific workspaces, each of which needs to ...
data engineering
  • SørenBrandt's avatar
    SørenBrandt
    1 year ago

    Thanks to input from Vinodh247 and Microsoft support, I quickly realized that what I was trying to achieve is not currently supported. According to MS Support, it is on the roadmap, but without an ETA.

    Instead, I decided to downscale my ambitions: In each customer specific workspace, I now manually create the Mirrored Azure Databricks Catalog, along with a Lakehouse and shortcuts pointing to the mirror. With that in place, I can deploy remaining workspace items using deployment pipelines that update the data source for semantic models. It works, and I am ok with this solution for now.

    That said, I do look forward to being able to use Terraform to create the Azure Databricks Catalog and Storage connections required for the mirror, and be able to roll out changes to these connections as part of my deployment pipelines. The current solution is a bit too manual for my taste :).

Vinodh247's avatar
Vinodh247
Super User
1 year ago

This is an expected error. Deployment pipelines currently don’t let you parameterize or override the catalogName, databricksWorkspaceConnectionId, or storageConnectionId when deploying a Mirrored Azure Databricks Catalog.  The REST based Update Mirrored Azure Databricks Catalog API supports modifying only a limited set of properties like autoSync, mirroringMode, and storageConnectionId, but not catalogName or databricksWorkspaceConnectionId. That’s why attempting to change those fields triggers the "CatalogName or DatabricksWorkspaceConnectionId mismatch" error.

 

Fix that you can try:

  1. Per Workspace Setup via Script/ARM Template: Automate the creation of Mirrored Catalogs in each target customer Fabric workspace by scripting against the APIs or using ARM templates. This gives you full control over the catalog name and connections per customer.
  • Use the REST APIs for creating a new Mirrored Azure Databricks Catalog, specifying per workspace details.
  • You can automate this via PowerShell, Azure CLI, or custom orchestration in your CI/CD pipeline.
  1. Template + Post Creation Patch: If you still want a somewhat template driven model...
  • Deploy a base mirrored catalog to each workspace.
  • Immediately call the Update API to adjust properties that are supported (like autoSync, mirroringMode, or storageConnectionId).
  • For properties that you cannot update, you’ll need to recreate (or delete and re-create) the mirrored catalog with correct parameters.

To sum it up, you cannot override the catalog’s connection details through Fabric Git or deployment pipelines as of today. The best option is programmatic provisioning of each mirrored catalog with the proper conn strings per customer. I would recommend building that as a script or ARM based automation to keep it clean and repeatable.

 

Please 'Kudos' and 'Accept as Solution' if this answered your query.