Forum Discussion
Measuring similarity between strings
Anonymous wrote:
I still have no information on what measure is used to measure the string similarity.
As I mentioned above, the measure used is Jaccard similarity.
Calculating the similarity between all pairs of 2 million strings means 4 trillion comparisons. This will take a lot of computing time and Power Query likely isn't the best tool for this particular job. You'll probably need some optimization techniques to make this more tractable. See this approach for example,
https://hpccsystems.com/blog/edit-distances-and-optimized-prefix-trees-big-data
Generating a trillion pairs is challenging no matter what tool you use. What are you trying to do that requires all possible pairs?
Microsoft documentation says that "Power Query uses the Jaccard similarity algorithm to measure the similarity between pairs of instances." Wiki: https://en.wikipedia.org/wiki/Jaccard_index