1. Lakehouse with shortcut connections Problem Statement: Lakehouse deployment fails when shortcut connections are present. Additionally, the deployment pipeline is unable to create a Lakehouse with the default schema in the target workspace. Even if a Lakehouse with the same name is manually created and shortcut connections are added, the deployment pipeline treats it as a different item and continues to fail. As a result, complete manual intervention is required for each deployment, which is not scalable or feasible for ongoing development and release cycles. 2. Lakehouse with custom schema and tables Problem Statement: Deployment of a Lakehouse without shortcuts succeeds; however, the deployed Lakehouse does not include any custom schemas or table definitions that were created in the development workspace. In our implementation, we work with multiple datasets and rely on custom schemas and structured tables for better organization and tracking of data. Currently, the deployment process results in a newly created Lakehouse without any of the existing schema or table metadata, which significantly limits the usefulness of Lakehouse deployment. Similarly, Git integration does not capture custom schemas or table definitions in the repository, leading to increased manual intervention during deployment. img: Metadata of Lakehouse in GitHub Repo As a result, deploying a Lakehouse create only a new Lakehouse in the target workspace. Recreating schemas and tables manually or through scripts is more complex and time-consuming than creating a new Lakehouse itself. Lakehouse deployment should not be limited to merely creating an empty Lakehouse. 3. Warehouse or SQL Database deployment as complete unit Problem Statement: Currently, Warehouses and SQL Databases are deployed as a complete unit, without the ability to selectively deploy individual schemas or tables. In real-world development scenarios, multiple schemas and tables are often developed in parallel. While some objects may be ready for deployment, others may still be under active development. The existing deployment approach forces teams to deploy all schemas and tables together, even when only a subset of objects is production ready. Deployment pipeline (No selective options for SQL Database modified objects) This increases the risk of deploying incomplete or unstable objects and limits flexibility in release planning. As a result, teams are either forced to delay deployments or rely on manual scripts and workarounds, which reduces the effectiveness of automated deployment pipelines.
... View more