topic Re: Has anyone got git integration working? in Data Engineering

Has anyone got git integration working?

PhantomThread — Tue, 19 Nov 2024 17:09:28 GMT

So we're a year in to using Fabric, and we've got quite a lot of stuff in there now.

However, I'm still outside of proper source control management as deployment pipelines don't work and deploying our branch to a new workspace never works.

If anyone has got it working and is properly doing anything close to CI/CD can you share with the community how you did it?

Cheers!

Re: Has anyone got git integration working?

Anonymous — Wed, 20 Nov 2024 03:05:19 GMT

Hi PhantomThread

Microsoft provides some documentation and blogs about CI/CD options in Fabric. They introduce some best practices.

Exploring CI/CD Capabilities in Microsoft Fabric: A Focus on Data Pipelines | Microsoft Fabric Blog | Microsoft Fabric

CI/CD workflow options in Fabric - Microsoft Fabric | Microsoft Learn

Best practices for lifecycle management in Fabric - Microsoft Fabric | Microsoft Learn

You said the deployment pipeline and git integration don't work, did you receive any error message? Error message sometimes could indicate a possible cause.

At the moment, some Fabric artifacts are not supported in deployment pipeline or git integration. If you hit these limitations, you might experience a failure. Please check the following documentation for troubleshooting first.

Fabric lifecycle management tools Frequently asked questions - Microsoft Fabric | Microsoft Learn

Troubleshoot the Fabric lifecycle management tools. - Microsoft Fabric | Microsoft Learn

If you are unable to resolve the failure you met, please provide more details about what you have tried for CI/CD process and share the error messages (note: please remove sensitive information).

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

Re: Has anyone got git integration working?

jwinchell40 — Wed, 20 Nov 2024 12:11:36 GMT

PhantomThread - This is still a work in progress for us because of the solution we provide and host for clients in Fabric we are waiting on PBIR App Support; then we will go full tilt on figuring it out.

What we have done to date (sorry for the long post), but while we not in an automated CI/CD space yet; we've been able to prove out a lot of things so we can look at building deployment automations.

Happy Path Projects:

For our happy path projects, mainly internal, where the Datasets/Reports are similiar we've connected each one of those workspaces to a GIT Repo with a Main branch and sub-folders for each individual workspace. Each workspace is connected to their respective folder. That has given us the ability to edit the Datasets/Dashboards directly in Power BI web so we can track changes. It has been very reliable and given the TMDL support detecting conflicts between changes local vs web has been huge.

New Deployments:

While this is not automated yet; when we have a new deployment of this solution we setup the Workspace and connect it to the same Project & configure the Sub-Folder for that workspace. In VS Code (local or web) I open the project and I copy my template files into this new sub-folder. Then I update the file names, report parameters. I save and commit all changes. Then on the Workspace side we pull changes and my Dataset & Reports are deployed and they just work. I've eliminated opening the template in Desktop, change parameters then publishing. I do it all in 5-10m and I'm done. Documenting these steps will help build the automation when we get time to do it; but we've been able to at least prove out that it will actually work.

Copy/Paste Changes:

I've been able to open the project in VS Code; add a measure to the template then Copy/Paste that measure code into all of the other reports. Save & Committ and all of the Datasets get that new measure. No longer am I having to open up each Dataset to add the change manually.

One thing I'm really excited about that we just proved out is making changes to data source queries. A lot of our solutions use data from sources we do not host and require a gateway. As we all know in Power BI Desktop you cannot connect to data sources behind a gateway. So everytime we have a change we have to repoint the dataset at a dummy, local db, make changes publish. Then change connection parameters and refresh so that the right data gets pulled. Very time consuming. We've been able to change Power Query steps in VSCode Commit them and then it just works in Web. That is a huge timesaver for us. It cuts a good 10-15m out of the process; in some cases more.