The ultimate Microsoft Fabric, Power BI, Azure AI, and SQL learning event! Join us in Stockholm, Sweden from September 24-27, 2024.
2-for-1 sale on June 20 only!
Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started
Hello Community,
I want to bring this as a fresh discussion and seek help on managing data flows schedule refresh. There is one data flow i have created and i knew that it holds only 30 records. However , it takes more 30 minutes to refresh the data flow. Am I doing something wrong in following processes ?
Some where i read that If I am under premium capacity, they suggested to run datasets/dataflows at different times. But i checked the schedules and not more than 5 datasets are running currently. In this way, I am not getting good use of dataflows and unable to convey pros of using data flows with clients .
I am under Premium capacity and sure that environment doesnt have any momory issues as well. Could some one tell how to get rid of this situation ?
Thanks,
G VEnkatesh
hi @Anonymous
You may try to restart the capacity to if it get better.
https://docs.microsoft.com/en-us/power-bi/service-admin-premium-restart
Regards,
Lin
Hi Lin,
Sure .Thanks for the suggestions. We would certainly restart the premium capacity as a measure to free up the memory.
Thanks,
G venkatesh
Hi Gilbert,
Thanks a lot for sharing those whitepapers. This information is very useful . I have reached out to Power BI Admin in my org. I have gone through some of those Statistics ( CPU Utilization and Memory usage) which we need to look at . I see that only 10 GB of Memory added for data flows . While my admin is yet to get back to me about the usage details, I would like to know if there is a way to increase this limits so that the dataflows will run faster .
At this time, I am just looking for one thing . I want my Dataflows to RUN FAST to make sure data is not old in my reports . We are not concerned about buying new hardware or clean up/flush out memory for dataflows to run fast.
Please assist !!!
Thanks,
G Venkatesh
Thanks,
G Venkatesh
Hi Gilbert/Community,
Also attaching the health statistics (metrics) currently we have our Premium capacity. While there are metrics for datasets and paginated reports as well, but i am particularly looking at dataflows only.
Based on this, What does you suggest for us to have the dataflows refresh quickly ? Also to avoid seeing stale data in reports, what is the frequency of refreshes i can do on each data flow ?
Thanks,
G Venkatesh
Hi Gilbert,
Thanks for your suggestions . I have discussed this with the Admin team and they are planning to increase data flows size to 10 GB.
Could you please show us how to enable Enhance compute Engine under Data flows ? We tried looking for this option ,but didnt find it in Admin portal.
Kindly assist !!
Thanks,
G venkatesh
Hi @Anonymous
You can find it under the Capacity Settings and then under Workloads
Thanks Mate(champ) . This helps me a lot . I will take this sugestions forward with Admin .
G Venkatesh
Hey Gilbert,
Hope you are doing good. After all the discussions we had last week , i requested my admin to increase the data flows capacity and enable the Enhance compute memory engine.
They tried their best and increased the size to 5 GB and also enabled the Compute memory engine option .
However, I still see that my data flows are running hours to complete refresh. I created a small report based out of SQL as data source in Power BI desktop and Data flow as a data source in Power BI desktop to compare data .
The updates are not proper for data flow record .
Please suggest what we have to do in this cases.
Hey Gilbert & R1k91,
Looks like i have some good news after increasing memory limit to 10 GB for data flows and container size to 1000 MB.
- The dataflows are running faster than before ( taking 5 mins today Vs 1 hour yesterday ) . However, i am running them in business off hours and resources might be available. I would like to trigger them after couple of hours and see how they looks like.
Mean while, I am thinking about the Enterprise gateway and the doubts you have raised about it.
where do i need to check if my gateway is working fine or not ? Also,when can we say that gateway is a potential bottleneck ? I would like to know the use cases to answer your questions .
Please assist
Hey Gilbert,
I have also attached few more screen shots of the metrics we have in our capacity as of now. Please see if all are good.
I'm not sure Enhanced Compute Engine could solve your issue in this way.
If you read the FAQ section of documentation you'll find that in some cases the refresh will be slower because much memory is needed with ECE enabled during load.
https://docs.microsoft.com/it-it/power-bi/service-dataflows-enhanced-compute-engine
You also have to consider that when loading data with ECE enabled, behind the scene it's loading data in both Azure Data Lake Storage and Azure SQL Database hidden instace to push future transformation that will be quicker using the query folding.
Hi R1K91,
Thanks for replying . I have actually followed a suggestion from one of the member in community. But looking at your post now, I think Data flows are not helping me to fetch data quicker as the refreshes arent working as expected. Before the refresh is Completed, my data in report is getting older .
Are my above statements are true with regards to Data flows or am i doing some thing wrong with those capacity settings ?
Please suggest !!!
Thanks
G VEnkatesh
Well it isn't any easy issue. You said your dataflows should fetch just 30 rows but I've some question:
Have you tried to move the same workload on a pbix in order to see which is the time needed to perform the transformation?
Hi R1k91,
Thanks for replying . I have answered your questions below. Please find them and provide your support .
Answer : Source is Power query(m- query) which is in fact data coming from SQL server .
Answer : I am using enterprise gateway
Answer : i do not have any transformations . Its a plain data coming from single table .
Have you tried to move the same workload on a pbix in order to see which is the time needed to perform the transformation?
Answer : yes i tried . it just took 30 seconds to come up with the data .
So.. no transformations at all and 1 single table with 30 records.
Have you already tried to publish the pbix with the same workload and to configure it to refresh the dataset through the same gateway dataflows are running?
This could help you to understand if gateway is the bottleneck.
Yes Buddy.. I tried that and i didnt find any issues . we have almost 150 reports which run on this gateway every day and they are even scheduled to refresh every one hour (datasets) . we didnt come across any major issues.
Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.
Check out the June 2024 Power BI update to learn about new features.
User | Count |
---|---|
57 | |
42 | |
30 | |
26 | |
23 |