Forum Discussion
Replicating TCP-DS Benchmark in Fabric
Finally generated the data using the TCP-DS Duck DB extension and saved them into parquets in my lakehouse
Hi v-menakakota ,
I've manually generated the spark.sql.perf.jar file on my local machine and uploaded as a custom library on the Fabric environment.
The current issue is I'm not able to import it into my notebook with the following error:
----> 1 import spark.sql.perf ModuleNotFoundError: No module named 'spark'
Hi VictorMed ,
Thanks for the update! Since you've manually uploaded the spark.sql.perf JAR, here are a few steps to ensure it is correctly imported in your Fabric Notebook:
- Try adding the JAR to your Spark session manually and please ensure that the path matches where you've uploaded the JAR in your OneLake Files or Lakehouse.
- Since you’re getting ModuleNotFoundError: No module named 'spark', try using this import pattern. If com.databricks.spark.sql.perf is not found, it likely means the JAR isn’t properly loaded into the Spark environment.
spark.sql.perf is primarily a Scala-based benchmarking library. Fabric’s Notebooks use PySpark, so it may not be compatible unless you run it in a Scala environment (which Fabric currently does not natively support in Notebooks).
If this post was helpful, please give us Kudos and mark it as Accepted Solution to assist other community members.
Thank you.
