Power query fuzzy clustering with large data sets

[Anonymous]

Hello,

Wondering if anyone has any suggestions or best practice regarding Power Query Clustering Values with large dataset(dataset has over 1.5million rows). I've tryied just about everything to reduce the lag and improve performance but with no luck.

it buffers or gets stock for hours without any results:

Any advice or recommandation is greatly appriacate, thank you!

[Anonymous · Accepted solution]

Hi @Anonymous ,

Before applying the fuzzy cluster, we can refer the following link to optimize the data model to improve the performance.

Optimizing the data model

As any fuzzy operation on the dataset will bring a huge performance impact to the data processing. The reason is that every text value has to be compared with every other text value in the table, the similarity threshold of the two has to be calculated (based on the algorithm above). This process is a very time-consuming process. Especially with more rows in the data table, you will feel it much more... Please find more details about Fuzzy Clustering...

Fuzzy Clustering in Power BI using Power Query: Finding similar values

Best Regards