Forum Discussion
Dataflow Scheduled Refresh Not Pulling All Data
It suggests to me that at 5:30 in the morning when you are doing the scheduled refresh, some process is happening on the server and all of the data isn't there.
To test this theory, go set the scheduled refresh to run 5min from now, or at the next half hour increment, then wait and see if that has the full data.
I can't really use that as a check at this point because the data that I am pulling is static throughout the day and comes from a data warehouse which is only updated over night. It is scheduled by our IT guys to run at midnight and I start my report refreshes at 4am to give everything time to finish. I've asked the IT folks to look into any recent logs or history to see if that nightly refresh has started to run long recently.
ok, let it run again at 4am. It it has garbage data again, then don't do a manual refresh, set the timer for 15min into the future and let that run.
I suspect the DW isn't up to date.
I pushed back my dataflow and dataset refresh by 30 minutes for this morning and it loaded a full version of my data for it looks like my issue was related to source data being incomplete.
Now I need to work with the IT folks to figure out why the source data refresh is taking so much longer than previously.
Glad you found the issue.
What I would recommend is you work with the team to see if they (the system) can send you an email alert when the data warehouse is fully updated.Then you can use Power Automate to monitor your emails for that alert, and when it comes in, then dataflows and datasets can be triggered.
So some days everything might start at 3am, but if there is an issue, and the email doesn't come in until 9am, well, yeah, reports are late, but at least they aren't loaded with half data.