Forum Discussion
DataFlow and Dataset size
Hi
I started creating bunch of dataflows and subsequently created datasets using them and later paginated reports. It was working smoothly. Now we added more data years and each dataset size grown as 400MB+.
How do we know the Dataflow size? (Upon searching in this forum, it says not possible)
How do we modify the code and control the dataset size during our development? It is becoming hard to debug or add more tables to it. or any other suggestions for faster development?
TIA
10 Replies
400 MB seems small. What is your constraint?
To control development dataset size you can use a fake incremental refresh that uses RangeStart and RangeEnd to limit the dev size but then in Power BI you specify that the storage range is the same as the refresh range.
Or use Deployment Pipelines to point at different (sub) data sources.
Two part to my question:
a) I would like to know is there any way to know the dataflow - size, each table/query size inside the dataflow. I use deployment pipelines between environments. So, I like to compare between UAT and Prod enironments by sizes and code (.json file). FYI, UAT and Prod both points to same datasources.
b) Let me clarify further, Size is not an issue at PBI service. I am aware that it can go upto 10GB.
My Issue is more related to the dataset editing (.pbix file), it is taking too much time and being remote and VPN causes even slower. To do changes I am downloading and editing and then publishing.
Thanks for replying.
a) you can get that information from the dataflow refresh history. Click on that little download arrow and then look inside the downloaded CSV file to see the partition sizes.
b) My proposal to use the RangeStart and RangeEnd parameters seems to fit your requirement. As you likely know any structural updates will re-trigger the full refresh anyway.
