Forum Discussion
Avoid throttling when loading from SharePoint folder
I have an environment with in excess of 18,000 individual files and multiple Dataflows running at the same time and yes it suffers from throttling but Microsoft support pointed me in the direction of buffering each Binary before you expand it and this significantly improved the performance.
So I land these CSV files into Dataflows and then pull that into the model.
Source = Csv.Document(Binary.Buffer([Content]),[Delimiter=",", ...............
Does not completely get rid of the issue so I also schedule a number of refreshes.
Hope that helps.
Regards
Andrew
Hi Anonymous ,
Sorry for misunderstanding. Please check whether the following documentations can help you.
Retrieving data from SharePoint causing throttling errors
Avoid getting throttled or blocked in SharePoint Online
Best Regards
Rena
Thanks a lot Anonymous
I believe my issue is related to the first post.
There is no solution to it as of today so I'll follow up in that thread in case somebody was able to solve it.
Kind regards,
Jon
I have an environment with in excess of 18,000 individual files and multiple Dataflows running at the same time and yes it suffers from throttling but Microsoft support pointed me in the direction of buffering each Binary before you expand it and this significantly improved the performance.
So I land these CSV files into Dataflows and then pull that into the model.
Source = Csv.Document(Binary.Buffer([Content]),[Delimiter=",", ...............
Does not completely get rid of the issue so I also schedule a number of refreshes.
Hope that helps.
Regards
Andrew
Thanks a lot Anonymous,
The load failed from the desktop version once (so I believe the issue can reappear sometimes), but for the first time, I was able to refresh the published version.