Forum Discussion

AdarshChekodu's avatar
AdarshChekodu
Icon for Advocate IV rankAdvocate IV
10 months ago
Solved

Deployment Overwrites Production Pipeline Schedule with Development Schedule – Expected Behavior?

Hi all,

 

Yesterday, while deploying my data pipeline from Development to Production, I noticed that the schedule configured in the Production pipeline was overwritten by the schedule from the Development pipeline.

 

This is a problem for us because the cadence of data refresh in Dev and Prod environments is intentionally different.

Is this expected behavior (possibly a new feature or change)?


If so, what's the recommended workaround or best practice to preserve environment-specific configurations like scheduling?

Any insights or suggestions would be appreciated. Thanks!

pipelines
  • V-yubandi-msft's avatar
    V-yubandi-msft
    10 months ago

    Hi AdarshChekodu ,

    Thanks for your response. You are correct in Microsoft Fabric, deploying a Data Pipeline using Deployment Pipelines replaces the entire pipeline definition, including schedules and triggers. This is the expected behavior, and currently, there isn’t a built in option to keep environment specific settings like schedule during deployment.

     

    1. This is the current workaround, though it can be tedious. The ARM template model in ADF does provide more flexibility, and this limitation in Fabric has been noted by many users.

    2. While these help with dynamic logic within a pipeline, such as changing data sources or filters, they cannot be used to set or manage the schedule. There is no documentation supporting this use at the moment, but I will share any updates if that changes.

    3. At present, it is not possible to isolate or exclude the schedule when deploying a pipeline. Deployments are all or nothing, which can be challenging when development and production require different refresh schedules.

     

    I’d recommend raising it on the Fabric Ideas - Microsoft Fabric Community many users have requested more control over environment specific settings like schedules.

     

    Regards,
    Yugandhar.

4 Replies

  • V-yubandi-msft's avatar
    V-yubandi-msft
    Icon for Community Support rankCommunity Support
    10 months ago

    Hi AdarshChekodu ,

    Thank you for engaging with the Microsoft Fabric Community. when you deploy a pipeline from Development to Production, the entire definition is replaced, including schedules, parameters, and triggers. Because refresh timings often differ between Dev and Prod, it’s best to keep environment specific settings separate from the deployment process.

    To manage this,
    1. Manually set the schedule in Production after deployment to maintain the correct refresh timing.
    2. Use parameters or environment variables for values that vary between environments.
    3. If you use CI/CD, consider excluding schedule configurations from deployment or setting up separate release definitions for Dev and Prod.

     

    Regards,
    Yugandhar.

    • AdarshChekodu's avatar
      AdarshChekodu
      Icon for Advocate IV rankAdvocate IV
      10 months ago

      V-yubandi-msft 

      Thanks alot for the response,

      1. I belive this is what I do now but it can get tedious! I belive in ADF the deployment (+ ARM Template) this is can be done the correct way without any manual effort

       

      2. I dont think we can use Parameters or Env Variable to schdule pipeline; Could you please send some documentation on the same?

      3. I use the Fabric Deployment Piplines for the Deployment process and as far as I understand using this I cant Isolate the data pipeline form scheduling (please correct me if I am wrong)

      Thanks,

      Adarsh

      • V-yubandi-msft's avatar
        V-yubandi-msft
        Icon for Community Support rankCommunity Support
        10 months ago

        Hi AdarshChekodu ,

        Thanks for your response. You are correct in Microsoft Fabric, deploying a Data Pipeline using Deployment Pipelines replaces the entire pipeline definition, including schedules and triggers. This is the expected behavior, and currently, there isn’t a built in option to keep environment specific settings like schedule during deployment.

         

        1. This is the current workaround, though it can be tedious. The ARM template model in ADF does provide more flexibility, and this limitation in Fabric has been noted by many users.

        2. While these help with dynamic logic within a pipeline, such as changing data sources or filters, they cannot be used to set or manage the schedule. There is no documentation supporting this use at the moment, but I will share any updates if that changes.

        3. At present, it is not possible to isolate or exclude the schedule when deploying a pipeline. Deployments are all or nothing, which can be challenging when development and production require different refresh schedules.

         

        I’d recommend raising it on the Fabric Ideas - Microsoft Fabric Community many users have requested more control over environment specific settings like schedules.

         

        Regards,
        Yugandhar.