Matching script using python in Fabric - running using pandas vs park data frame

fo6168 Hey,
I will follow below steps to troubleshoot the issue.

for script 1:

1) Consider using multiprocessing to parallelize the computation, leveraging multiple cores on your machine.

Alternatively, use dask library for parallelized dataframe operations.

2) Simplify the inner loop comparison by using vectorized operations where possible with numpy and pandas, though this may require significant changes in how comparisons are performed.

3) Use similarity heuristics or initial filters to reduce the number of candidates to compare before performing detailed fuzzy matching.

Script 2: Spark with RapidFuzz

Suggestion: 

1) Instead of fuzzy matching every record, perform a join operation on comparable columns and then apply fuzzy matching only on resulting candidate pairs.

df_candidate_pairs = df1_clean.join(df2_clean, df1_clean['some_column'] == df2_clean['some_column'], 'inner')

2) UDF can slow down Spark as they often run slower than built-in functions. Consider trying Spark SQL functions available or a pandas_udf.

3) Convert the resulting Spark DataFrame to Pandas for final operations that require the session access, bypassing worker restriction issues.

4) I will use pandas_udf: Leverage Pandas UDF, which operates on Python objects directly and is executed at Python-level inside the JVM.

from pyspark.sql.functions import pandas_udf @pandas_udf(schema, SparkSession.sqlContext) def fuzzy_match_udf(df1_chunk: pd.DataFrame) -> pd.Series: matches = [] for row_i in df1_chunk.itertuples(): # Perform the same logic as defined in your function # Append results to matches return pd.Series(matches) df_matches = df1_clean.withColumn("Matches", fuzzy_match_udf(df1_clean))

Ensure df2_clean is as small and optimized for broadcasting operations, minimizing memory footprint.

Thanks

Harish M

Kindly accept it as solution if it solved your problem. Kindly give kudos.