Git Branching with active Schedules

[caseybks]

Hi,

We are using git integration with our development environment workspace. Develop is our collaboration branch from which the data engineers create a feature branch from. Their feature branch is linked to their private branched workspace. 

The issue is, when an engineer creates a new feature branch from the develop branch, all the active schedules are also active in the branch, therefore, every time they create a new feature branch, they have to go through any pipeline with a schedule to turn the schedule off. This doesn't seem like a very scalable way of working. What if someone forgets to turn the schedules off in their branched workspace? We could have multiple scheduled pipelines running in parallel in many branched workspaces.

I've added .schedules to .gitignore so any changes, e.g. turning the schedule off, is not merged into the develop branch. The issue is when new branches are created. I've created master orchestration pipelines with sch_ prefixed to signify this pipeline is scheduled and grouped them in one folder together.

Would be interested to learn about how other teams are dealing with this?

Thanks.

[JonathanFlint]

scheduling in dev or test purely comes down to the SDLC process and naming conventions, I am having a similar issue though

I have scheduled a pipeline in my production environment, but because the schedule itself is tracked by git, it wants to enable that schedule on my dev workspace which I obviously don't want to do

I can disable it in the dev workspace but this is entirely manual (without adding extra steps to our devops pipelines to do this for us when dev branch is updated via PR) AND if I do this, my dev workspace now reports a diff between the workspace and git which is very annoying because future features added to dev may cause issues when calling the gitSync API endpoint in the DevOps pipeline if the workspace believes there to be a conflict

It fundamentally doesn't make sense to track schedules in git this way when you are not going to want every item to be scheduled in every feature, dev or test/UAT workspace, I can't think of any git based workflow that tracking if the schedule is enabled would work that doesn't cause the above issues, even the simplest trunk based workflow with feature PRs directly into main/prod would still cause either: 

schedules for everything enabled in every feature workspace OR

diffs reported in main/production workspace by adding schedules outside of git

just feels like someone has gone git-happy by adding this without actually considering the impact on different workflows

Also, I have added *.schedules and **.schedules to my git ignore and this doesn't prevent the workspace from reporting a diff

[caseybks · Accepted solution]

Hi @JonathanFlint,

Haha, yes it does feel like someone has gone git happy 😄 I wish they'd focus on Lakehouse git version integration instead.

Yes, we've gone for a similar solution. In the develop branch, which is the branch we all pull from, we have the committed versions of the schedules turned off, but then turn them back on in the workspace. So unfortuantly we have to live with the uncommited icon noficiations in the workspace, which is annoying, but we can live with.

Only our Dev workspace is backed by git, the develop branch. The code is pushed into other workspaces by our DevOps deployment pipelines. Or you can use the Fabric deployment pipelines too. So turning on/off schedules in Dev workspace does not affect the test and production workspaces.

[v-achippa]

Hi @JonathanFlint,

Thank you for sharing your experience. Yes tracking schedules in git can cause unexpected diffs in Dev/feature workspaces. The best way to avoid this is to keep schedules disabled in Dev and enable them only in Test/Prod via deployment pipelines or automation. This keeps git clean and makes sure schedules only run where they should run.

Thanks and regards,

Anjan Kumar Chippa

[JonathanFlint]

How can disabling the schedule in dev/feature workspaces avoid a diff when this is exactly what causes the diff in the first place?

Or are you suggesting that you keep all schedules disabled in the source code, and enable them in Test/Prod as part of the deployment process? This would also cause a diff but now its on Test/Prod instead of Dev/feature because git thinks the schedule is disabled but the workspace has it enabled?

[v-achippa]

Yes enabling or disabling schedules directly in the workspace can cause diffs. The best way is that keep all schedules disabled in git across all branches. Then after deploying to Test/Prod via a deployment pipeline, enable the schedules at runtime in the workspace without committing that change back to git.
This way Dev/feature branches remain clean and only the Test/Prod runs the schedules, avoiding unwanted diffs.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @caseybks,

We wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[v-achippa]

Hi @caseybks,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @tayloramy for the prompt response. 

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[tayloramy]

Hi @caseybks, 

This is a valid problem, and I don't think there is an immediate solution other than to adjust your process. 

I do question why you have things schedueld in DEV however. In my enviornment, we don't schedule anything in DEV as it's used purely for development. Once we are ready to start testing it with data on a scheduled basis and not ad-hoc, we promote it to TST which is where things are scheduled. This prevents the issue you are encountering, but is a pretty big process shift from what you're currently doing it seems. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[caseybks]

Hi @tayloramy,

We have all environments running daily becasue we need the data across all environments to be the same. Otherwise we would be back filling data all the time.