Power BI Git Integration - Missing Pipeline Dependencies?

[dbeavon3]

We are trying to deploy a git branch to a PBI workspace.

Notebooks are pretty easy to update in PBI workspace.  The code comes over from git as expected.  I generally have no problems with notebook code.

But "pipelines" are exceedingly troublesome to work with.  If I ever need to update them from scratch on an empty PBI workspace., it is a huge struggle.  There are lots of errors about missing dependencies, many of which don't actually tell you what is missing.

Assuming I have a git branch, and some json-backed pipelines, how can I get these to be transferred over to my PBI workspace without these errors.  I'm spending tremendous time investigating missing dependencies, especially if the pipelines were given to me by a teammember who is not available to provide any hints about the underlying dependencies.

I guess what I'd like to have is some sort of "best effort" update, where the missing dependencies are skipped, but the general structure of the pipeline is transferred into the PBI workspace from git.  Does anyone have tips or tricks to this type of a "best effort" update, perhaps in a manual way.  I'm guessing there are guids that need to be cleared out or something like that?  I'm not a full-time Fabric developer, and only spend about 40 hours with pipelines in any given year.  Given the poor git experience in Fabric, this time may increase up to be 2x or 3x what it should be. 

Any help would be appreciated.

[v-shamiliv]

Hi @dbeavon3 
Could you let us know if you have submitted your idea in the Ideas Forum?
Thank you.

[v-shamiliv · Accepted solution]

Hi @dbeavon3 
Thank you for sharing your insights regarding the visibility and limitations with Git integration and ADF, as well as the challenges in identifying issues. Your perspective highlights a significant gap that warrants further attention. Could you please formalize this feedback by outlining the specific challenges and potential impacts you mentioned? Feedback submitted through the Microsoft Fabric Ideas portal is often reviewed by the product teams and can lead to meaningful improvements.

I kindly request you to submit your ideas and feedback here Microsoft Fabric Ideas feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.

 
Thank You.

[v-shamiliv]

Hi @dbeavon3 

For clarification I said that if there are any issues with GUID clear those issues.I meant to say about issues like duplication, I didn’t say about clearing out GUID’s.

We do not handle Git integration or deployment issues, nor any backend issues that cannot be reproduced.

If there are any deployment issues, please raise a support ticket so that our backend team can assist you.

The following link will guide you how to raise a support ticket, kindly refer to it.
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

Thank you.

[dbeavon3]

Hi @v-shamiliv 

>> We do not handle Git integration or deployment issues, nor any backend issues that cannot be reproduced

The problem with these ADF pipelines is the json is NOT source-control-friendly.  It doesn't even feel like it is my code.

I suppose the ADF team and the git team to just point fingers at each other for the next couple of years.  Meanwhile customers are struggling with recovering our saved work from source control.  It is a big "tease" to have the code so close at hand, and yet so far away from being usable again.

Do you really think it is reasonable for ADF team and the git integration team to point fingers at each other?  These teams are both working on the same Fabric, albeit probably on different continents.

I think you need to share tips with your customers on how to harvest our pipelines, after they have become "trapped" within our git repos.  Else we have to rebuild from scratch which is plain crazy. 

[v-shamiliv]

Hi @dbeavon3 
Thank you for reaching out microsoft fabric community forum.

when deploying pipelines from a git branch to a Power BI workspace. Here are a few tips that might help you :

• Before deploying, ensure that all dependencies required by the pipelines are available in the target environment. You can create a checklist of dependencies and verify them manually.
• Consider using a dependency management tool like pip for Python or npm for Node.js to handle dependencies automatically. These tools can help identify and install missing dependencies.
• Set up a baseline environment with all necessary dependencies installed. You can use this environment as a reference to ensure that the target environment has the same setup.
• Maintain a document that lists all dependencies required by your pipelines. This can be a reference for troubleshooting and ensuring that all dependencies are accounted for.
• Implement scripts that automatically check for missing dependencies and attempt to install them. This can save time and reduce manual effort.
• If you suspect that GUIDs might be causing issues, try clearing them out and re-deploying the pipelines. Sometimes, GUID conflicts can cause unexpected errors.
  Make sure  that your pipelines and dependencies are properly version-controlled. This can help in tracking changes and identifying issues more easily.
  
  For more information  please refer to the following link:
  Troubleshoot the Fabric lifecycle management tools. - Microsoft Fabric | Microsoft Learn
  

  If the issue persists, we recommend raising a support ticket for further assistance. To create a support ticket for Fabric and Power BI, please refer to the steps outlined in the following guide:

  How to create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn

  
  
  If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.
  
  

  Best regards,
   Shamili.v

[dbeavon3]

Hi @v-shamiliv 

A lot of your suggestions are assuming that preparation was done in advance (ie. putting dependencies in place, and maintaining documentation, and implementing custom scripts).

My question is specifically about recovering a pipeline from source control in non-ideal circumstances.  Ig. if the pipeline was built by a teammember who is not available and did NOT make those preparations in advance.  Assume for now that they had properly pushed the pipeline to git, at a minimum.  That should be enough to do some sort of "best effort" deployment, right?  How should we proceed with that deployment?

The path forward is unclear and troubesome.  We get meaningless error messages like the one shown below:

Troubleshoot the Fabric lifecycle management tools. - Microsoft Fabric | Microsoft Learn

... even if we wanted to ignore such messages, and retrieve a partial/incomplete copy of the original pipeline, it seems like there is no mechanism for that!

You said:

>> If you suspect that GUIDs might be causing issues, try clearing them out and re-deploying the pipelines.

Is there an approach for clearing out guids, and avoiding the meaningless error above.  Do we remove the json elements, do we set the GUID's to zeros?  to blanks?  Is this workaround documented?  Is there a blog about it?

Sorry, I just want to double check because lots of moderators are copy/pasting from chat GPT and giving answers that are either totally generic or totally unworkable.  Chat GPT may be able to form English sentences, but it has no-first hand experience with this git integration stuff in the ADF pipelines.

(PS. I don't know if you realize it but all your links are clearly being generated by chat GPT.  The URL parameters include this : utm_source=chatgpt.com).