Power BI Scheduled Refresh for Big Data

[Anonymous]

Hello, Seeking your advice or recommendations for scheduled refresh in the PBI Service for big data.

for context:

Our KPIs Dataset is connected to different Datasources (SharePoint, CRM, AWS/ERP), and we use VDI (Virtual Desktop) for the gateway since most of its critical data is being dumped (it is also scheduled) into a local folder (csv file).

what our workaround is to manually refresh the KPIs Dataset and publish it in to the workspace everyday and it takes around 1hr to 1hr and 30mins before to finish the manual refresh.

the reason why, we never set a scheduled refresh in PBI Service it will always fail (we're suspecting that this is because of its bid data?)

Question/s:
- Can we use Power Automate for this? to reduce the waiting time for this to be refreshed?

- or we can still configure it in the service?

- if the two mentioned above cannot be done, can you recommend other tools or ways?

Thank you!

[amitchandak]

@Anonymous , I think the reason could be timeout also.

The best is to share what Error you are getting.

Also is there incremental logic for data or all are full load?

In case you started Micorosft Fabric Trail. I Would recommend Using Data Pipeline, Data Flow Gen and Notebooks for data ingestion, choosing the best suited for the purpose. And then create a Schema in DW

Mastering Microsoft Fabric 35+ Videos in English:
https://www.youtube.com/watch?v=p-v0I5S-ybs&list=PLPaNVDMhUXGYU97pdqwoaociLdwyDRn39&index=1