Forum Discussion
Handling Large Data Sets
supton How I wish I was just starting my CRM project. You'll find pretty quickly that to build out a solution you will either spend a ton of time pulling in only the data you need, and the processing will still take awhile, or you look for other solutions.
I manually built out a solution using Kingswaysoft to import all the tables from our online instance to a local DB. Built a tabular model on that data set and created my Power BI Reports.
MSFT released Solution Templates 2 days before I launched the reports... I cried :)
Check out the Solution Template for Dynamics 365 here - this basically guides you through building a full scale solution in Azure.
I'm assuming you want to do more than just query a few things, the above is my experiance and recommendations, but other approaches on a smaller scale could work, I just got frustrated with all the processing time, searching for things etc. And we needed a full support solution to report on all aspects of our CRM.
Thank you for the reply! I think I am going to try to rid the data of unncessary columns at this point, but considering I am reporting on summation data of all the rows according to certain linking criteria, I dont think that filtering into specific rows would make too much sense! We are getting over 2000 new records in a single custom entity daily (right now, we have a single entity with 1.5 million records), and we need to form reports on all of them, including the older stuff.
Columns make more of a difference than rows. Try not to query any columns that you don't strictly need.