Forum Discussion
Need recommendations for Fabric integration with Azure DevOps Repos
Hello,
We are in the process of setting up Azure DevOps Repos for a proposed Fabric implementation and are planning to manage the following components:
Lakehouse
Warehouse
Notebooks
Pipelines
Semantic Models
PBI Reports
Dashboards
Activators
Eventhouse
Eventstream
API (GraphQL)
I would like to understand the best way to organize and manage version control for all of these objects within Azure DevOps Repos. Specifically:
What are the recommended practices for maintaining version control across these components?
Are there any objects within this list that cannot be effectively version-controlled using Azure DevOps? If so, how should we handle them?
During our research, we found that Lakehouse and Warehouse changes don’t seem to reflect properly through DevOps Repos. Could you share any insights regarding limitations with version control in these areas, and any potential workarounds to address these challenges?
I appreciate any guidance or recommendations from those with experience managing similar setups.
Thank you!
Hi DineshKalyankar ,
Having worked on several Fabric deployments with Azure DevOps, I can share some recommendations for your setup:
For version control of Fabric assets in Azure DevOps, here's what I'd suggest:
General best practices:
- Use a Git-based repo structure with separate folders for different component types (notebooks, pipelines, etc.)
- Implement branch policies requiring pull requests and code reviews before merging to main
- Consider organizing by workspaces and then by component type within each workspace
Component-specific recommendations:
- Notebooks: These work well with git. Store as .ipynb files
- Semantic Models: Export as .bim files for versioning
- Pipelines/Dataflows: These can be exported to JSON/YAML format
- Power BI Reports/Dashboards: Use PBIX files (though you'll need to manually download/upload)
- Lakehouse/Warehouse: This is where you'll face the biggest challenges
For Lakehouse and Warehouse: You're right that these are problematic. The schema/definition can be version-controlled, but the actual data cannot be effectively managed through DevOps. This is a known limitation.
Workable approaches for Lakehouse/Warehouse:
- Version control the DDL scripts that create tables/views
- For Lakehouse, consider using Spark notebooks with code that defines the schema/tables and commit those
- Use database projects (.sqlproj) for more structured SQL components
- Implement CI/CD pipelines that deploy schema changes through automation
For objects that can't be effectively version-controlled:
- Document them thoroughly in a wiki or markdown files in your repo
- Consider implementing a change management process outside of DevOps
- Use feature flags where possible to control rollout of changes
One approach that's worked for my team is creating a deployment framework using PowerShell/Python scripts that interact with the Fabric REST APIs to automate deployments of components that don't work natively with git.
Hope this helps with your implementation! Feel free to ask if you need more specifics on any particular component.
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
3 Replies
- burakkaragoz
Super User
Hi DineshKalyankar ,
Having worked on several Fabric deployments with Azure DevOps, I can share some recommendations for your setup:
For version control of Fabric assets in Azure DevOps, here's what I'd suggest:
General best practices:
- Use a Git-based repo structure with separate folders for different component types (notebooks, pipelines, etc.)
- Implement branch policies requiring pull requests and code reviews before merging to main
- Consider organizing by workspaces and then by component type within each workspace
Component-specific recommendations:
- Notebooks: These work well with git. Store as .ipynb files
- Semantic Models: Export as .bim files for versioning
- Pipelines/Dataflows: These can be exported to JSON/YAML format
- Power BI Reports/Dashboards: Use PBIX files (though you'll need to manually download/upload)
- Lakehouse/Warehouse: This is where you'll face the biggest challenges
For Lakehouse and Warehouse: You're right that these are problematic. The schema/definition can be version-controlled, but the actual data cannot be effectively managed through DevOps. This is a known limitation.
Workable approaches for Lakehouse/Warehouse:
- Version control the DDL scripts that create tables/views
- For Lakehouse, consider using Spark notebooks with code that defines the schema/tables and commit those
- Use database projects (.sqlproj) for more structured SQL components
- Implement CI/CD pipelines that deploy schema changes through automation
For objects that can't be effectively version-controlled:
- Document them thoroughly in a wiki or markdown files in your repo
- Consider implementing a change management process outside of DevOps
- Use feature flags where possible to control rollout of changes
One approach that's worked for my team is creating a deployment framework using PowerShell/Python scripts that interact with the Fabric REST APIs to automate deployments of components that don't work natively with git.
Hope this helps with your implementation! Feel free to ask if you need more specifics on any particular component.
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
Thank you, burakkaragoz, for your quick response. This is really helpful. I’ll wait for a few more people to share their experiences.
Hi DineshKalyankar,
Thanks for reaching out to the Microsoft fabric community forum.
It looks like you are working on integrating Azure DevOps Repos into a Microsoft Fabric implementation to version control and manage various Fabric components. As burakkaragoz already responded to your query, kindly go through his response and if it answers your queries do mark it as solution.
I would also take a moment to thank burakkaragoz, for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.
If I misunderstand your needs or you still have problems on it, please feel free to let us know.
Best Regards,
Hammad.
Community Support TeamIf this post helps then please mark it as a solution, so that other members find it more quickly.
Thank you.
