Forum Discussion
Process dataset clear
I have a Power BI dataset in import mode.
I need to store it in the Git Repo, however with data it is too large. How can I process the dataset clear so I am storing just the structure and not the data?
8 Replies
Hi -Blair- ,
It sounds like what you're describing is a Power BI template. Take a look at the documentation linked below and see if this is what you're looking for.
https://docs.microsoft.com/en-us/power-bi/create-reports/desktop-templates#creating-report-templatesIt's not a template. It's the dataset . The datasets are used in active development.
Rather than having the datasets in current development stored on a drive on someones PC they need to be in source control. However source control has limitations with processing big files.
I need a way of storing the pbix files for traceability and reusability.
Create dataset, deploy to development workspace
Convert to template.
Template saved to source control.
template deployed to test?
template deployed to uat?
changes made to template.
template deployed to dev?
See what I'm getting at?, I need to pick it up and still develop off it.
I want to store the pbix file as that is what is developed off and that is what is deployed for the reports to use.
How do I carry out development changes on the template and how do I deploy the template as a dataset? Simly answer is you can't, you have to then convert it which means which you are storing in source control is different to what you are deploying.
Using the Power BI Dataflows, you can easily create reusable data integration pipelines that can be used in one or more Power BI reports. You can also share your dataflows with other users within the organization, and they can leverage the dataflow as well.
I'm not complety across data flows in Power BI.
My understanding is they are for deploying between environments.
The problems I have seen is that you ware limited to the number of environments you can deploy to and there is also no integration with Azure Devops or GIT.
Without GIT and DevOps how do you track changes, assoiciate stories (backlog items in the sprint), perform code reviews and pull requests?
It seems Power BI is being run very differently to the rest of Azure from the development integration standpoint. For everything else Azure you use AzureDevOps and GIT.
From experience GIT doesn't handle large files well, hence I'm looking for a way to "process clear" the datasets.
So if I can't store the datasets and reports in GIT due to size constraints then where are you meant to store the pbix?
Need a way fir multiple developers to be able to work on the files while following a development lifecycle like you would with any other code project. Changes should be tracked like they would in GIT with the ability to roll back easily.
How does everyone else manage the development lifecycle for Power BI?
Build, Commit, Test, Deploy, Repeat
