Lakehouse and Warehouse Deployment with Azure DevOps: A Practical Guide

Building an analytical platform often requires promoting Lakehouse schemas, files, and Warehouse objects across multiple environments. While Microsoft Fabric currently does not provide native support for Lakehouse schema or file deployment, we can still achieve this using Azure DevOps. In this blog, I’ll walk you through an approach that works for both Lakehouse and Warehouse deployments.

Why This is Important

When working with Microsoft Fabric, managing CI/CD for Lakehouse and Warehouse components is critical for maintaining consistency across development, staging, and production environments. Automating this process ensures reliability and reduces manual effort.

Lakehouse Deployment Strategy

A Lakehouse typically contains files (such as metadata used for processing) and tables. Here’s how you can deploy both:

1. Deploy Lakehouse Files Using AZCopy

• Create an Azure Git repository and upload metadata files and table schemas.
• Set up an Azure DevOps pipeline with PowerShell tasks:
• Use scripts to log in to Azure.
• Execute azcopy commands to copy files from the Git repo to the target Fabric workspace.

2. Deploy Lakehouse Tables Using Notebooks

• Upload all table schema files to the Git repository.
• Create a notebook that:
• Iterates through all SQL files.
• Uses Spark SQL to create tables in the Lakehouse.
• Configure an Azure DevOps pipeline with these steps:
• AzLogin for authentication.
• AzCopy to copy SQL scripts from the repo to the Lakehouse files section.
• A PowerShell script to call the Notebook REST API with jobType=RunNotebook.

📖 Reference: https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/run-on-demand-item-job?tabs=HTTP

Warehouse Deployment Strategy

Deploying a Warehouse is more straightforward:

• Download the DB project for the Warehouse from the Fabric portal and upload it to the Git repository.
• Create an Azure DevOps pipeline with these steps:
• Install .NET SDK.
• Install SQLPackage.
• Authenticate using AzLogin.
• Use sqlpackage with DACPAC commands to deploy from the Git repo to the target workspace.

📖 Reference: https://learn.microsoft.com/en-us/azure/devops/pipelines/targets/azure-sqldb?view=azure-devops&tabs=yaml%2Carm%2Cextract

Conclusion:

• Microsoft Fabric does not yet support native Lakehouse schema/file deployment, but Azure DevOps pipelines can bridge the gap.
• Use AZCopy for files deployment and Notebooks for table creation in Lakehouse.
• For Warehouse, leverage SQLPackage and DACPAC deployment.