Parallelization on Fabric using TF_on Spark

Your code is mostly correct, but there are a few issues that may prevent it from running properly:

Issues in Your Code

1. Syntax Error in map_fun Definition

   • The function definition contains a misplaced : after ctx😞, likely a copy-paste issue.
   • Correct syntax:
     python

     CopyEdit

     def map_fun(tf_args, ctx😞

2. Spark Context (sc) Initialization Issue

   • TFCluster.run expects sc (SparkContext), but in your code, you are passing spark.sparkContext. This should work, but if any errors arise, explicitly create a SparkContext:
     python

     CopyEdit

     from pyspark import SparkContext sc = SparkContext.getOrCreate()

3. Printing ctx Directly Doesn't Work as Expected

   • Instead of print('ctx'), you likely meant to print the actual context:
     python

     CopyEdit

     print(ctx)

4. Cluster Shutdown

   • The line # cluster.shutdown() is commented out. You might want to ensure that the cluster shuts down after execution to free up