Forum Discussion
Column profiling not using query folding on Databricks
Hi, I am using the databricks connector to connect to a Hive table that exposes a table with a two digit million number of rows. Users now would like to use the Column profiling functionality in Po...
It isn't that folding no longer works fmdus - it is that Power Query retrieves 100% of the data, then must do an analysis on it. It is not optimized in any way. Whatever granularity you have when you request the full dataset is what you get, and on a few million records, it can take forever, or simply time out.
I rarely recommend changing the 1000 sample size to full dataset unless your full dataset is 100,000 or fewer records. Even at that level, it gets really slow.
