Forum Discussion
Need help replacing data source (16M text Records) from SharePoint to sem model as data source
We have hundreds of .txt files which ends up being about 16M records. Scheduled refresh fails due to XML timeout (even with premium capacity). Makes sense loading that many .txt files from SharePo...
If you have a date created for the text files you could use incremental refresh. You would create a policy and publish to the service. Connect with tabular editor to apply the policy to create the partitions. Then connect with SQL server management server and manually load each partition. On subsequent refreshes only new files would be loaded in the active partitions.
The best option would most likely load these files into a warehouse or lakehouse and load from there