Forum Discussion

seppl-se's avatar
seppl-se
New Member
27 days ago
Solved

Warehouse CICD

Hi,

I’m working on a Microsoft Fabric project where I’ve implemented a CI/CD process using Azure DevOps. I’m using the fabric_cicd library for deploying everything except the Warehouse. For the Warehouse, I’m using a SQL Database Project deployment.

This worked well until we started using direct queries to the Lakehouse in our Warehouse views and stored procedures, for example:

SELECT * FROM lakehouse.stage.customer;

 

This approach is very convenient, but it introduces a problem during deployment. The SQL Database Project now requires a reference to the Lakehouse database project, and the deployment fails if that reference is missing.

It is possible to create a Lakehouse database project and keep it alongside the Warehouse database project in the repo, and that does work. However, maintaining that project and keeping it in sync with the actual Lakehouse schema is a significant amount of extra work.

So my question is: what is the recommended way to deploy Fabric Warehouses from Git when they contain references to Lakehouse objects?

We’ve also tested Fabric Deployment Pipelines, but our goal is to have everything version-controlled and reviewed in Git before anything is deployed to Production.

  • Hi seppl-se ,

     

    This is a really good question, and I think it's something many teams moving to Git-based CI/CD in Fabric are starting to run into.

    Once Warehouse objects begin referencing Lakehouse tables (for example, lakehouse.stage.customer), the SQL Database Project understandably tries to validate those dependencies during the build, which makes deployments more challenging.

    A few approaches our teams consider are:

    • Maintain a Lakehouse database project as a reference, which provides build-time validation but does require keeping it synchronized with the Lakehouse schema.
    • Deploy the Warehouse first, then deploy the views and stored procedures that reference the Lakehouse as a separate post-deployment step. This avoids the build dependency while still keeping everything automated and version-controlled.
    • Generate the reference project automatically as part of the CI pipeline, rather than maintaining it manually. It takes a bit more automation upfront but can reduce the ongoing maintenance effort.

    At the moment, I'm not aware of a native Fabric capability that automatically resolves these cross-artifact dependencies in a Git-based deployment, so I'd be interested to hear how others are handling this as well.

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!

     

  • Hi seppl-se,

     

    I have run into a similar design concern with Warehouse projects that contain three-part references to Lakehouse tables.

     

    The important distinction I would make is that the Warehouse SQL project can version and deploy the Warehouse objects themselves, such as views and stored procedures, but the referenced Lakehouse SQL analytics endpoint is a separate Fabric item and dependency.

     

    Microsoft currently documents database-project references for dependencies between Fabric Warehouses. However, the same guidance explicitly notes that it does not cover cross-database development against a Lakehouse SQL analytics endpoint, because Lakehouse tables and SQL analytics endpoint objects are not tracked in the same way as Warehouse project objects. Develop and deploy cross-Warehouse dependencies.

     

    Because of that, I would not try to solve this by adding the Lakehouse itself as a conventional SQL database project purely to satisfy the Warehouse build, unless the current Lakehouse source-control representation gives you the exact objects and validation behaviour you need.

     

    The approach I would use is to separate build-time validation from environment deployment.

     

    For build-time validation, I would either:

    • maintain a lightweight reference project or stub containing only the Lakehouse schemas and table definitions required by the Warehouse project or
    • exclude unresolved cross-database references from local validation where appropriate and perform the definitive validation against a Fabric test workspace after deployment.

    For deployment, I would use this order:

    1. Deploy or update the Lakehouse in the target workspace.
    2. Confirm that the expected Lakehouse tables are registered and visible through its SQL analytics endpoint.
    3. Deploy the Warehouse project containing the views and stored procedures that reference those tables.
    4. Run post-deployment validation queries against the target environment.

    This keeps the Warehouse schema fully version-controlled, while treating the Lakehouse as an external deployment dependency rather than pretending both items form one SQL project.

     

    I would also keep the Lakehouse name consistent across development, test and production where possible. Your three-part references use the Fabric item name, so consistent naming reduces the need for environment-specific script changes.

     

    Where different names are unavoidable, I would generate or replace the environment-specific reference during the Azure DevOps release rather than hard-coding the production Lakehouse name in the main project. A pre- or post-deployment script may help for some deployment tasks, although I would still keep the main object definitions deterministic and reviewable in Git. Microsoft now documents pre- and post-deployment scripts for Fabric Warehouse SQL projects here: Pre-deployment and post-deployment scripts.

     

    Fabric Git integration and deployment pipelines can now version and promote both Warehouse and Lakehouse items, but that does not necessarily mean that a Warehouse SQL project will automatically resolve every cross-item T-SQL dependency during its database-project build. Source control with Fabric Data Warehouse. Lakehouse Git integration and deployment pipelines.

     

    So my preferred pattern would be:

    Git repository
    ├── Lakehouse item definition
    ├── Warehouse SQL project
    ├── Optional Lakehouse reference/stub project
    └── Azure DevOps pipeline
          1. Deploy Lakehouse
          2. Validate SQL analytics endpoint objects
          3. Build and deploy Warehouse
          4. Run integration tests

    That gives you pull-request review and version control for the whole solution without requiring the Lakehouse SQL endpoint to behave like a fully deployable Warehouse database project.

4 Replies

  • Hi seppl-se ,

     

    This is a really good question, and I think it's something many teams moving to Git-based CI/CD in Fabric are starting to run into.

    Once Warehouse objects begin referencing Lakehouse tables (for example, lakehouse.stage.customer), the SQL Database Project understandably tries to validate those dependencies during the build, which makes deployments more challenging.

    A few approaches our teams consider are:

    • Maintain a Lakehouse database project as a reference, which provides build-time validation but does require keeping it synchronized with the Lakehouse schema.
    • Deploy the Warehouse first, then deploy the views and stored procedures that reference the Lakehouse as a separate post-deployment step. This avoids the build dependency while still keeping everything automated and version-controlled.
    • Generate the reference project automatically as part of the CI pipeline, rather than maintaining it manually. It takes a bit more automation upfront but can reduce the ongoing maintenance effort.

    At the moment, I'm not aware of a native Fabric capability that automatically resolves these cross-artifact dependencies in a Git-based deployment, so I'd be interested to hear how others are handling this as well.

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!

     

    • ShivekMaharaj's avatar
      ShivekMaharaj
      Impactful Individual

      Hi seppl-se,

       

      I have run into a similar design concern with Warehouse projects that contain three-part references to Lakehouse tables.

       

      The important distinction I would make is that the Warehouse SQL project can version and deploy the Warehouse objects themselves, such as views and stored procedures, but the referenced Lakehouse SQL analytics endpoint is a separate Fabric item and dependency.

       

      Microsoft currently documents database-project references for dependencies between Fabric Warehouses. However, the same guidance explicitly notes that it does not cover cross-database development against a Lakehouse SQL analytics endpoint, because Lakehouse tables and SQL analytics endpoint objects are not tracked in the same way as Warehouse project objects. Develop and deploy cross-Warehouse dependencies.

       

      Because of that, I would not try to solve this by adding the Lakehouse itself as a conventional SQL database project purely to satisfy the Warehouse build, unless the current Lakehouse source-control representation gives you the exact objects and validation behaviour you need.

       

      The approach I would use is to separate build-time validation from environment deployment.

       

      For build-time validation, I would either:

      • maintain a lightweight reference project or stub containing only the Lakehouse schemas and table definitions required by the Warehouse project or
      • exclude unresolved cross-database references from local validation where appropriate and perform the definitive validation against a Fabric test workspace after deployment.

      For deployment, I would use this order:

      1. Deploy or update the Lakehouse in the target workspace.
      2. Confirm that the expected Lakehouse tables are registered and visible through its SQL analytics endpoint.
      3. Deploy the Warehouse project containing the views and stored procedures that reference those tables.
      4. Run post-deployment validation queries against the target environment.

      This keeps the Warehouse schema fully version-controlled, while treating the Lakehouse as an external deployment dependency rather than pretending both items form one SQL project.

       

      I would also keep the Lakehouse name consistent across development, test and production where possible. Your three-part references use the Fabric item name, so consistent naming reduces the need for environment-specific script changes.

       

      Where different names are unavoidable, I would generate or replace the environment-specific reference during the Azure DevOps release rather than hard-coding the production Lakehouse name in the main project. A pre- or post-deployment script may help for some deployment tasks, although I would still keep the main object definitions deterministic and reviewable in Git. Microsoft now documents pre- and post-deployment scripts for Fabric Warehouse SQL projects here: Pre-deployment and post-deployment scripts.

       

      Fabric Git integration and deployment pipelines can now version and promote both Warehouse and Lakehouse items, but that does not necessarily mean that a Warehouse SQL project will automatically resolve every cross-item T-SQL dependency during its database-project build. Source control with Fabric Data Warehouse. Lakehouse Git integration and deployment pipelines.

       

      So my preferred pattern would be:

      Git repository
      ├── Lakehouse item definition
      ├── Warehouse SQL project
      ├── Optional Lakehouse reference/stub project
      └── Azure DevOps pipeline
            1. Deploy Lakehouse
            2. Validate SQL analytics endpoint objects
            3. Build and deploy Warehouse
            4. Run integration tests

      That gives you pull-request review and version control for the whole solution without requiring the Lakehouse SQL endpoint to behave like a fully deployable Warehouse database project.

  • v-kathullac's avatar
    v-kathullac
    Community Support

    Thankyou ssrithar  for Addressing the issue.

     

    Hi seppl-se ,

     

    As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?

     

    Thanks,

    Chaithanya.

  • v-kathullac's avatar
    v-kathullac
    Community Support

    Thankyou @ssrithar  for Addressing the issue.

     

    Hi @seppl-se ,

     

    As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?

     

    Thanks,

    Chaithanya.