Forum Discussion
Handling Large Data Sets
supton How I wish I was just starting my CRM project. You'll find pretty quickly that to build out a solution you will either spend a ton of time pulling in only the data you need, and the processing will still take awhile, or you look for other solutions.
I manually built out a solution using Kingswaysoft to import all the tables from our online instance to a local DB. Built a tabular model on that data set and created my Power BI Reports.
MSFT released Solution Templates 2 days before I launched the reports... I cried :)
Check out the Solution Template for Dynamics 365 here - this basically guides you through building a full scale solution in Azure.
I'm assuming you want to do more than just query a few things, the above is my experiance and recommendations, but other approaches on a smaller scale could work, I just got frustrated with all the processing time, searching for things etc. And we needed a full support solution to report on all aspects of our CRM.
We use an Azure mirror for our CRM. But in any case you should try to write your queries to pull the minimum amount of data needed for the report. Don't pull every column if you don't need them, filter rows, etc.
Thank you for the reply! I think I am going to try to rid the data of unncessary columns at this point, but considering I am reporting on summation data of all the rows according to certain linking criteria, I dont think that filtering into specific rows would make too much sense! We are getting over 2000 new records in a single custom entity daily (right now, we have a single entity with 1.5 million records), and we need to form reports on all of them, including the older stuff.
Columns make more of a difference than rows. Try not to query any columns that you don't strictly need.
