Dataflow Gen 2 failure with fuzzy matching

Hi Teskay,

Thanks for reaching out to the Microsoft fabric community forum.

It looks like you're facing performance issues with fuzzy merging in Dataflow Gen2, especially when dealing with large datasets. Fuzzy matching is inherently resource-intensive, and DF Gen2 may not be the best tool for handling such operations at scale. You can use Fabric Notebooks with PySpark which is best suited for large datasets and is optimized for distributed processing, instead of performing fuzzy matching in DF Gen2. You can also use a SQL-based approach with Soundex or Levenshtein distance as it runs faster in Lakehouse compared to DF Gen2.

If you still want to go with fuzzy matching then, reduce the number of comparisons by preprocessing. You can apply Standardization like convert all to lowercase or remove spaces and special characters before matching or use blocking techniques for example: compare only rows starting with the same letter.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.