Forum Discussion

NagaRK's avatar
NagaRK
Advocate I
1 year ago

Spark in Notebook taking more time to process the data.

Hi all,   I'm working on a diagnostic log ingestion engine built with PySpark and Delta Lake on Microsoft Fabric. My setup parses incoming ZIP logs from a server, transforms signal data per file in...
data engineering
notebook
v-sdhruv's avatar
v-sdhruv
Community Support
1 year ago

Hi NagaRK ,

You can try to optimize the synapse that reduces the number of small files and increases individual file sizes, improving write performance and reducing I/O overhead.

You can enable it using:

spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true")

Use V-Order is a write-time optimization for Parquet files that improves read performance and compression. While it adds ~15% overhead to write time, it can significantly reduce read latency and storage costs.Use this if you want to reduce total file size.


spark.conf.set("spark.sql.parquet.vorder.default", "true")
 

You can even Parallelize Writes with Spark Native Execution Engine


spark.conf.set("spark.native.enabled", "true")

Hope this helps!