Forum Discussion

ObungiNiels's avatar
ObungiNiels
Resolver III
1 year ago
Solved

Deployment Process for Warehouse Projects

Hello fellow warehouse users, 

 

I have a quite plain question: How are you realizing deployments processes of warehouses which are loaded with stored procedure definitions etc.? 

 

In the past, we had major problems including workspaces with warehouses in our deployment pipelines. Reason for that is, that changes to a table which are implemented in a stored procedure and then deployed to a new environment where the table still is in it's "old" structure, the stored procedure cannot be saved in the new environment since it references columns that are not part of the table. At some point we switched from using deployment pipelines to using a schema compare in a database project, for example in Visual Studio or Azure Data Studio. Although this mostly works, we are growing dissatisfied with this option as it often times involes lots of copying and pasting from the external tool to the warehouse SQL pane (because some updates don't work when sent from Visual Studio). Due to these circumstances I am reaching out to collect valuable insights from you. Happy about all kinds of contributions! 🙂 

 

data warehouse
Warehouse

8 Replies

  • v-prasare's avatar
    v-prasare
    Community Support
    1 year ago

    Hi   ObungiNiels , We understand the limitations of current deployment pipelines in handling tightly coupled schema changes and procedural logic. For long-term scalability, Microsoft Fabric's deployment tooling is expected to evolve, but for now, automation using DevOps or scripting offers more control

     

    I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas. Feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.

     

     

    Thanks,

    Prashanth

    MS Fabric community support

     

  • v-prasare's avatar
    v-prasare
    Community Support
    1 year ago

    Hi ObungiNiels,

    We are following up once again regarding your query. Could you please confirm whether you have raised ideas/feedback with Microsoft? we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

    Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

     

    Thank you for your understanding and participation.

    • stuckerj's avatar
      stuckerj
      Helper I
      7 months ago

      Hi Srisakthi and others,

       

      Thanks for the link to DacFx info.  I see this is a simple way to recreate the metadata in a new Warehouse. Any thoughts on taking it to the next step?

      1. Scripting the export and import, and especially:
      2. Getting the data from old to new Warehouse.

       

      Thanks!

      Jeff

      • Srisakthi's avatar
        Srisakthi
        Super User
        7 months ago

        Hi stuckerj ,

         

        DacFx can be used only to deploy schemas. If we need to migrate data then we can do by the following way,

        1. Data flow gen2

        2. Data pipeline copy activity

        3. Cross database queries

         

        I have tried copying the parquet files from exisitng warehouse to new warehouse using one lake explorer but it is not working as expected.

        Thanks,

        Srisakthi

  • MaciejUrbaniak's avatar
    MaciejUrbaniak
    New Member
    1 year ago

    Hi ObungiNiels,

     

    In VS, which project type and target database did you chose for MS Fabric DWH project? I use standard SQL database project template but no matter which Target Platform I chose, I get the same error message:

     

    The source and target platform must be the same to use Schema Compare for Synapse Data Warehouse in Microsoft Fabric

    My VS2022 version is 17.13.5.

     

    Thanks,

    Maciej