Forum Discussion
Big Data Cleansing
I'm curious on how you all handle cleansing/massaging big amounts of data. By big data, talking about > 1,000,000 rows. My data source is a direct SQL query so I am not uploading any files. I just received this project and am currently doing very simple stuff but it takes quite a bit of time for PowerBI to finish the task. Like over 3 mins to remove delimiters from columns, etc...
From my previous research, I was under the assumption that PowerBI is very capable of handling data in the millions of rows. Is it going to be this time consuming to make minor updates?
Just asking.
Hi Anonymous,
Since it's a Direct Query mode, I would suggest you apply the data manipulations in the data source as more as possible. There is a limitation of the rows of Direct Query that is 1 million rows. Please refer to service-get-data#considerations-and-limitations. Mostly because of the performance.
Best Regards,
Dale
1 Reply
Hi Anonymous,
Since it's a Direct Query mode, I would suggest you apply the data manipulations in the data source as more as possible. There is a limitation of the rows of Direct Query that is 1 million rows. Please refer to service-get-data#considerations-and-limitations. Mostly because of the performance.
Best Regards,
Dale